Trends in open and distance learning research : 2005 vs 2015

Purpose – The purpose of this paper is to analyse the research trends in the field of open and distance learning (ODL) as reflected in journal articles. Design/methodology/approach – It compares research articles published in 2005 and 2015. Content analysis was conducted on a total of 288 research articles published in seven peer-reviewed journals on ODL. The study covers research areas and keywords, research methodology and participant types, and author collaboration. Findings – The results show that macro-level research on areas such as globalization and cultural aspects of ODL remain relatively scarce, and international collaboration appear uncommon in both 2005 and 2015. However, there was an increasing amount of international collaboration in developing macrolevel research. Empirical research studies, especially those using quantitative methods, have become the dominant methodology. The data sources have also been broadened. Several new keywords which did not exist or were rarely used (e.g. massive open online courses) have become common in 2015. Originality/value – The analysis offers insights for researchers into how they can develop their research effectively in the field and enhance the chances of their research outputs being accepted. Recommendations are also made for ODL researchers on the types of research that tend to be accepted for publication and will have a high potential impact in the future.


Introduction
Open and distance learning (ODL) practices have been evolving, in particular as educational technologies being applied to ODL are developing at an unprecedented pace.Research in the ODL field appears to have also been evolving.This paper attempts to investigate the changes that have taken place in the last decade by systematically comparing research publications.
For effective research in the field and enhancing the chances of their research outputs being accepted, it is important for researchers to be aware of trends in the field.
In order to facilitate healthy and sustainable development of ODL research, scholars in this area have suggested conducting empirical research to analyse, and monitor the changes over time (e.g.Bozkurt et al., 2015).To remain connected to the constant advances in technology that have been taking place at an increasingly fast pace, it is especially important to follow closely the trends in ODL research.

Coding procedures
For the variables of number of authors and research collaboration, two raters were involved.One rater counted the number of authors for each article and coded the variable at five levels, i.e. "one author", "two authors", "three authors", "four authors", and "five author or above".If there was more than one author, he then determined whether the authors were from the same institution (coded as "same institution" for the variable "research collaboration"); from different institutions, but the same region/ country (coded as "cross-institution only"); or from different regions/countries (coded as "cross-border").The other rater cross-checked the coding made by the first rater and ensured there were no mistakes in the coding.

AAOUJ 11,2
For the variable of target population and/or participant group, two raters made a joint effort to search for the relevant information in each article and coded the variable into one of 12 categories, namely, bachelor's/sub-degree students, postgraduate students, academician/teachers, students, document/data file, administrators, K-12 students, specialists, institutions, system/programme, multiple types, and other.
The two variables mentioned above were relatively straightforward to code with a little subjective judgement.The variables of research areas and research methodology involved much more subjective interpretation.Therefore, two raters made judgements on each article on these two variables independently to avoid bias in the coding results.For research areas, the classification scheme was based on the one developed in Zawacki-Richter et al.'s (2009, pp. 22-25) study.The two raters each assigned a category from a total of 15 research areas at three research levels to each article.
Research areas of ODL categorized by Zawacki-Richter and von Prümmer ( 2010): (1) Macro-level: distance education systems and theories: • access, equity, and ethics; • globalization of education and cross-cultural aspects; • distance teaching systems and institutions; • theories and models; and • research methods in distance education and knowledge transfer.
(2) Meso-level: management, organization, and technology: • management and organization; • costs and benefits; • educational technology; • innovation and change; • professional development and faculty support; • learner support services; and • quality assurance.
(3) Micro-level: teaching and learning in distance education: • instructional design; • interaction and communication in learning communities; and • learner characteristics.
The definitions and descriptions for each type of empirical research methodnamely, quantitative, qualitative, and mixed methodwere based on those specified in Creswell (2013).Two raters each judged whether the research article was empirical or nonempirical (i.e. a literature review or theoretical article in which the arguments were not supported by empirical data).For empirical research articles, each rater judged which type of methods was employed based on Creswell's (2013) criteria.An inter-rater reliability analysis using Cohen's κ statistic was performed to determine the consistency of the judgements of the two raters on the research area and the research methodology of each article.For the research area, the inte-rrater reliability was found to be κ ¼ 0.73, po0.01, 95 per cent CI (0.68, 0.78); and for research methodology, the

219
Trends in ODL research inter-rater reliability was κ ¼ 0.77, po0.01, 95 per cent CI (0.71, 0.83).According to Landis and Koch (1977), values of κ from 0.40 to 0.59 can be considered moderate, 0.60 to 0.79 substantial, and 0.80 outstanding.Therefore, the κ statistic showed that the two raters had reached a good level of agreement with each other on the research area and research methodology of the articles.For each article with inconsistent coding between the raters, a final consensus was reached through discussion.
For the keywords indicated in the research articles, 203 keywords from 42 articles in 2005, and 220 keywords from 45 articles in 2015 were extracted from all the articles.For accuracy and avoidance of duplication, a final list of 337 keywords was then employed.The frequency and proportion of articles containing each keyword in the list were counted for descriptive analysis.

Results
As noted earlier, the major part of the results was based on content analysis of 288 research articles sampled from all issues of seven ODL journals in 2005 and 2015, covering the aspects of research areas, authorship patterns, research collaboration, research methodology, target population, and/or participant groups.To analyse the trends reflected in the keywords in the research articles, we investigated a sample of 87 research articles from three ODL journals as specified above.the research articles published in both 2005 and 2015, in that the macro-level research was underrepresented.There was no significant difference between the percentage of each level of research area between 2005 and 2015, and macro-level research constituted less than 20 per cent of all articles in both years.Of the five categories in the macro-level research area, none had published over 10 per cent of all articles in both years.Such a pattern of results is consistent with previous findings (Bozkurt et al., 2015;Kanwar, 2014;Zawacki-Richter et al., 2009), which showed a persistent lack of attention to macro-level research in ODL research by the end of 2005.

Research areas
In contrast, micro-level research, although with only three categories of research area, constituted the largest proportions of the three levels in both 2005 (45.3 per cent) and 2015 (44.0 per cent).Among the three micro-level research areas, "instructional design" had the largest group of research articles published in both 2005 (25.5 per cent) and 2015 (21.4 per cent).With the continuous technological progress in the past decade, especially related to digital learning environments (Bozkurt et al., 2015), utilising the new technology to enable better delivery of course materials, innovative teaching and learning approaches, and more effective assessment practices has remained an important research issue that has attracted a lot research effort.In addition, a significant increase in research articles published can be observed in the research area "learner characteristics" in 2015 (14.3 per cent) compared to 2005 (5.7 per cent).This difference indicates that there may have been a shifting focus on learner-centred approaches and an increasing amount of research on individual differences, e.g.motivational and behavioural patterns among learners who participated in ODL.
ODL research in 2015 also showed an increase in the percentage of research at the meso-level, although the change was not statistically significant.Of the seven categories of meso-level research areas, a notable increase in percentage was observed for "quality assurance", from only 8.5 per cent in 2005 to 14.3 per cent in 2015, ranking the third highest among all 15 research areas.This trend may have been brought about by the emerging new technology and new modes of learning in the past decade (e.g.new massive open online courses (MOOCs) programmes), indicating that quality assurance has become an increasingly important issue for keeping up with the pace of technological innovation in ODL.

221
Trends in ODL research articles decreased in 2015 (26.9 per cent) compared with 2005 (47.2 per cent) at a statistically significant level.Multiple authorship had become the norm in 2015, with over 70 per cent of the sampled research articles having more than one author.
We further analysed how research collaboration was conducted in the multi-authored research.Table III presents the percentages of multi-authored research articles with different types of research collaboration, i.e. between authors from the same institution, authors from a different institution but from the same country or region, and authors from different countries or regions.A χ 2 test of independence was performed to examine the relation between types of research collaboration and year, and the test revealed that the distribution of different types of research collaboration was not significant difference between 2005 and 2015 ( χ 2 (2, n ¼ 189) ¼ 0.25, p ¼ 0.88).In the sampled research articles published in both 2005 and 2015, a majority of research collaboration was done between authors from the same institution.

Research collaborations by research area
To examine the trend in research collaboration at different levels of research area, we conducted a three-way cross-tabulation analysis on the relations between the level of research area, and the type of collaboration in the different years.Table IV presents the percentage of the three types of collaboration at each level of research area in 2005 and 2015.Because the breakdown of expected cell counts for the 2005 contingency table did not meet the assumptions for χ 2 tests (Yates et al., 1999), the χ 2 test was performed only for the 2015 data.The relation between the level of research area and the type of collaboration in 2015 was significant ( χ 2 (4, n ¼ 133) ¼ 11.17, p o 0.05), showing that the types of collaboration varied at different levels of research area.Specifically, although around 60 per cent of the research collaboration was done by authors from the same institutions at all three levels of research area, the breakdown of the research collaborations done by authors from different institutions differed across the three levels.Among studies from authors affiliated with different institutions, the collaborations for macro-level research were mainly cross-border, while those for meso-level and micro-level were mainly within border.
Compared with the data for 2005, the distribution of different types of collaboration at the meso and micro levels had not changed much statistically in 2015.However, for the research collaborations at the macro-level, a greater proportion of cross-border collaboration had been conducted in 2015 than in 2005.This results show that, although both macro-level research and cross-border collaboration remained underrepresented in ODL research in 2015, there was an encouraging trend of an increase in cross-border collaborations at the macro-level of research.12.9 per cent) compared with 2005 (34.0 per cent) at a statistically significant level.In contrast, the percentage of articles using quantitative methods increased significantly in 2015 (43.4 per cent) compared with 2005 (24.5 per cent).A higher percentage of empirical, especially quantitative, research had been done in 2015 than in 2005, revealing a shifting norm in ODL research that theoretical work had to be validated by empirical, especially quantitative, data in order to be published in good journals.

Research methodology by research areas
To examine whether the trend in research methodology found in Section 4.4 varied at different levels of research area, we conducted a three-way cross-tabulation analysis on the relations between research methodology, and type of collaboration in the different years.Table VI presents the percentage of the four types of research methodology at different levels of research area in 2005 and 2015.For all three levels of research, there were more non-empirical research articles published in 2005 than in 2015, although the difference was not significant at the meso-level.For all three levels of research, there were more quantitative research articles published in 2015 than in 2005.This trend in research methodology can be consistently found at all three levels of research area.

223
Trends in ODL research students at non-specific levels being recruited in 2015 than in 2005.These students included those who participated in online learning through various MOOCs platforms but did not register with higher education institutions as undergraduate or graduate students in a traditional way.There was a higher percentage of documents/data files, including the data collected from online platform or social network service providers, such as Twitter and Facebook, in 2015 than in 2005.Also, there was a lower percentage of studies recruiting multiple types of participant groups, typically students, and instructors involved in traditional face-to-face programmes, in 2015 than in 2005.These three findings jointly uncover a trend that more research effort has been devoted to studying the emerging, innovative ways of learning, such as MOOCs, in which students were from various levels, communication was made online, and traditional face-to-face interactions between students and instructors were absent.

Keywords
A descriptive analysis of the keywords indicated in the research articles sampled from three ODL journals was conducted on the assumption that keywords give a holistic reflection of the topics of the research papers.
Keywords from 42 journal articles in 2005 and 45 articles in 2015 were ranked in terms of their frequency of appearance in the articles in these two years.The results showed that "MOOCs", coined in 2008, was the keyword indicated most frequently in 2015, appearing in 18 per cent of the selected articles; and "blended learning" ranked third.A possible reason is that from mid-2000 onwards, there have been more attempts to define the field of blended learning (Chew et al., 2010;Graham, 2006), attracting more researchers to investigate the related topics, resulting in an increasing number of publications devoted to this area.
Whereas some new keywords appeared in 2015, some keywords found in 2005 fell significantly in terms of their frequency.For example, "e-learning" appeared in 17 per cent of the articles in 2005, but only once in 2015.However, "online learning" was indicated relatively more frequently in 2015 (9 per cent), suggesting that in recent ODL research the term "e-learning" may have been redefined as this old term no longer fitted the developments in educational technology in the last decade.More specific keywords such as "online learning" may have become more useful for indicating the related research topics.

Discussion
The findings above are summarized below and their implications are highlighted.They are also compared with relevant findings from other studies: (1) Micro-level research, especially instructional design, still dominated the ODL research sampled in both 2005 and 2015.This is consistent with what was observed from ODL research sampled until 2008 (Zawacki-Richter, 2009) and until 2013 (Bozkurt et al., 2015).An encouraging change noted in 2015 was that meso-level research accounted for a proportion of research articles similar to that of micro-level research.A number of important areas, such as quality assurance, attracted considerable attention.However, much work still remains to be done to facilitate macro-level research as it continued to be underrepresented in 2015.
(2) International collaboration appeared uncommon in both years, but seemed to have become more common in macro-level research in 2015 publication data.Since macro-level research frequently involves different perspectives and data sources from researchers around the world, rather than a localized setting, such international collaboration will be likely to foster high-quality macro-level research than other types of research collaboration. (

225
Trends in ODL research will benefit ODL research by reflecting multiple facets of students and teachers' learning and teaching processes, especially in the interaction and communication domain.It seems desirable that a greater effort should be made to explore the potential usage of various types of data in addressing research topics apart from learning and teaching, particularly in macro-level research, such as how open educational resources or MOOCs penetrate the learning activities for global users of the internet, and mobile networks.
There are a number of limitations in the present research.The sample size in this study is relatively small compared to prior research with the same purpose; and our analyses, especially the analysis of keywords, were based on a small number of journal articles.Also, we conducted a relatively small-scale analysis, comparing only research published in two years.Therefore, only the differences between the beginning and the end of a ten-year period (2005 to 2015) were observed, but the changes in the years in between have not yet been uncovered.Since the trends in ODL research aspects were not necessarily linear (Zawacki-Richter et al., 2009), we could have missed important information about such changes, which might have taken place under the influence of a variety of contextual factors, such as the introduction of new technology or new local or global policies in certain years.In addition, the variables coded in our study may not be comprehensive enough to cover the various aspects of ODL research we were interested in.Future research should focus on improving these aspects through a more thorough sampling of ODL research journals and research articles, covering data in more years, and exploring and measuring more variables.This would give a deeper level of understanding of the research trends, and higher convergence validity in the ways employed to operationalize ODL research trends.

Conclusion remarks
The present study uncovered the evolutionary trends of research in the ODL field as reflected in refereed research journal articles.We observed changes in ODL research in 2015 compared with 2005.These trends are worthy of researchers' attention and can facilitate their identification of study areas and approaches.
For conducting research and publishing a high-research output, this study points to the following recommendations for ODL scholars.Research publications on macro-level issues with the support of empirical, especially quantitative, results on innovative theoretical positions will be in greater demand.Research would be more promising if there is international collaboration or its scope is extended to incorporate multiple facets to cross-validate the initial conclusions, and conclusions made by others.In addition, the use of multiple types of sources for data collection should be explored, as such uses have been made possible largely by the exciting emerging technological advances.
Table I presents the percentage figures for each type of research article that was classified in one of the research areas categorized by Zawacki-Richter and von Prümmer (2010), as well as the total percentage figures for each level of research area.The three levels of research area (macro, meso, and micro-level) showed an imbalance in distribution among Table II presents the percentages of research articles with different numbers of authors in 2005 and 2015.A χ 2 test of independence was performed to examine the relation between the number of authors and the year of publication.The relation between the two categorical variables was significant ( χ 2 (4, n ¼ 288) ¼ 13.53, p o 0.01), showing changes in authorship patterns from 2005 to 2015.The percentage of single author Table V presents the percentages of research articles with different types of research methodology in 2005 and 2015.A χ 2 test of independence was performed to examine the relation between research methodology and year.The relation between the two categorical variables was significant ( χ 2 (3, n ¼ 288) ¼ 23.95, po0.01), showing changes in research methodology in 2015 compared with 2005.The percentage of non-empirical articles decreased in 2015 ( Table VII shows the percentages of research articles with different types of target population and/or participant group in 2005 and 2015.Cross-year comparison of the proportion shows three notable changes in 2015.There were a higher percentage of Notes: **,***Significant at p o 0.01; p o0.001, two-tailed levels, respectively Notes: a Students include non-specific-level students, online learners, adult learners, etc.; b document/ data files include university archives, research papers/dissertations, online platform archives, online activity log files, etc.; c multiple types means more than one type from items one to ten, e.g., both students and instructors; d other types means those categories not covered by items 1 to 10, e.g.community sample.*p o0.05, two-tailed