Quantitative Research, 14 pages (3500 words)

Doctoral education in quantitative research methods: some thoughts about preparing future scholars

Subject: Health & Medicine

Info

Published: December 27, 2021
Updated: October 8, 2022
University / College: The University of Melbourne
Language: English
Downloads: 37

Table of Contents

The Complexity of Social Science Research
Difficulties in Teaching Quantitative Methods
Possible Solutions
Author Contributions
Conflict of Interest Statement
References

Understanding social science domains is difficult because of the complex nature of the domains and the consequential challenges in capturing and analyzing data. To address such challenges, many analytic techniques have been and continue to be developed [ 1 ]. This means that social science fields like education, despite the perception that they are “ soft” sciences, may be the hardest fields to research [ 2 – 4 ]. Within social sciences data can be either numeric or text, requiring statistical and qualitative techniques of data collection and analysis [ 5 – 7 ]. Perhaps post-structuralist or post-modern approaches [ 8 ] to data cannot or should not be automated or statistically analyzed. However, while analysis of qualitative data is normally undertaken by humans, automation of the analytic process for these data is being developed (e. g., statistical discourse analysis; [ 9 ]). Such developments suggest that the future of social science research may bring greater synchronization between these approaches to data.

Consequently, this opinion piece focuses upon the teaching of quantitative and statistical methods in the doctoral degree. In this piece, I first provide evidence for the complexity of social science research. Then I consider the challenges and issues in our current arrangements in doctoral education in the field. I conclude with some tentative solutions for improving doctoral education in quantitative methods. Unsurprisingly, it is my opinion that we need to admit that it is not evident how to balance the quantitative research methods curriculum such that it both prepares a wide variety of doctoral students for their careers as analysts, researchers, or scholars and does justice to the complex nature of reality and scientific investigation.

To exemplify the complex nature of social science data and the types of techniques required, consider the impact evaluation of the Head Start program. Head Start began in 1965 with the aim of boosting the school readiness of low-income children in the United States [ 10 ]. The impact study examined nearly 5, 000 children (ages 3 and 4) in two-separate cohorts randomly assigned to Head Start or a control program and followed for 4 years. The cohorts were not equivalent in racial/ethnic characteristics and the control groups did not have equivalent and common alternative programs. The children were in 23 different states served by 84 randomly selected agencies, and 383 randomly selected Head Start centers were involved. The outcomes of children were examined at multiple times in multiple domains. Data were collected from multiple stakeholders using a variety of methods. Despite randomisation, not all children enrolled or remained in their assigned condition.

A number of advanced statistical techniques were used. To make adequate generalizations from the sample to the population, sampling weights were calculated for each child [ 11 ]. Hierarchical or multilevel analysis [ 12 ] was used to take into account that students were nested in centers in different states and such grouping influenced the nature of their experience. Unsurprisingly, about 20% of children were absent at various data collection points; hence, missing data analysis was needed [ 13 ]. Determining the effect of change in the various measures required techniques for repeated measures analysis [ 14 , 15 ].

Thus, answering a relatively straightforward question (i. e., How much impact does Head Start have?) required complex data design, data collection, and sophisticated data analyses to account for multiple confounding and interacting factors. This creates substantial challenges for researchers and those involved in preparing future researchers in doctoral programs. Doctoral education in the social sciences has yet to fully resolve what students need to be taught in quantitative research skills to be able to cope with life beyond their own degree. On the one hand, students need a deep, but probably relatively restricted, knowledge and skill set in order to make an “ original and substantive contribution.” On the other hand, once graduated and employed in the field of social science research, junior researchers need a broad repertoire of analytic skills [ 1 , 4 , 16 ] to resolve the challenges and opportunities arising from the availability of large scale data [ 17 – 19 ].

Most new graduates are unable to implement many techniques beyond those required for their own dissertations or thesis work [ 20 ]. While it is expected that new graduates will develop a more sophisticated suite of capabilities in their early work, this depends entirely on the work environment in which they are employed. It may be more common than not, that new doctorates have little opportunity to practice what they have learned or extend their competencies. Of course, this self-limiting reliance on already known methods applies equally to tenured and experienced researchers, who having successfully established a reputation within a field, may restrict themselves to what they already know how to do [ 21 ], even if the data might be better addressed with alternative methods. This means that doctoral education in the social sciences and which seeks to prepare graduates for competence in quantitative research methods faces problems in terms of both curriculum (what to teach) and pedagogy (how to teach).

Difficulties in Teaching Quantitative Methods

Social science studies that have to address large and/or complex data are different to the relatively simple world of small sample laboratory experiments involving measurements either side of an intervention. In that situation, relatively simple statistical measures are sufficient; the difference in mean can be tested for statistical significance using a t -test or ANOVA F -test; the practical size of the mean score difference can be evaluated using an effect size adjusted by the inter-correlation of the dependent variable scores. In true experiments, no sophisticated procedures are needed to estimate counter factual data, the control data exists. Hence, education and social science students have an immense mountain to climb, relative to perhaps the statistics needed in a robust psychology experimental study. While the experiment may be the default method, a further complicating factor that hinders the adoption of advanced, yet appropriate, analytic methods might lie in the preference of funders and sponsors of research for easy to understand results. Without necessarily corrupting the quality or independence of research, stakeholders who need to deal with the public (i. e., politicians) may not welcome complicated analyses and may discourage the use of appropriate techniques that are harder to explain [ 22 ].

Also complicating the field of quantitative research methods are issues to do with the nature of measurement and variables used to operationalize complex phenomena. For example, operational definitions of latent phenomena (e. g., intelligence) are implemented through psychometric statistics (e. g., factor analysis, item response theory) of manifest variables that measure latent constructs [ 23 , 24 ]. However, this approach is contested, especially by Michell [ 25 ] who argues that psychometric processes do not conform to scientific standards of measurement (e. g., there are no inherent units of measurement for latent constructs like intelligence and the tools used to measure do not have additive structure). While psychological constructs may not be inherently measurable, Borsboom and Mellenbergh [ 26 ] suggested item response theory modeling may overcome some of these criticisms. Humphry [ 27 ] suggested that the problem can be overcome in part by conceiving of responses to psychometric measures over a standardized time period as creating a constant scale or continuum on which individuals vary on a domain theoretically aligned to the data collection items. The debate about the properties of psychometric measurements and consequently how various measures ought to be statistically analyzed is not yet resolved.

However, it is worth remembering that not all variables in social sciences and education are so problematic because the relevant phenomenon is directly observable (e. g., attendance, truancy, tardiness, standardized test score performance). If test scores are used to describe abilities on specific tests (e. g., reading comprehension or mathematics) and no attempt is made to infer from the observed performance to a latent construct (e. g., intelligence), perhaps this debate becomes moot. Nonetheless, each researcher needs to develop a stance toward the properties of the tools used to create variables whose means and variations are used to make claims about social realities. Consequently, all researchers need a deep understanding of the philosophic underpinnings of scientific and statistical methods [ 28 – 30 ] and the need to make nuanced judgments concerning the methodological choices they make.

Further complicating this matter is the problem that the vast majority of doctoral students in education begin with relatively inadequate preparation to tackle the problems of large and complex data [ 31 ]. Although many research students have significant experience in professional practice, they often encounter substantial hurdles in coming to terms with the logic and practice of quantitative research design and analysis [ 32 ]. Indeed, even getting consensus among professors of research methods as to essential methods for postgraduate study is difficult [ 32 ]. One solution to the diversity of method is to expose students to many methods rapidly with little time or support for developing a deep understanding or competence; this creates the “ inch-deep, mile-wide” phenomenon in which real competence is not achieved.

Furthermore, the linkages between research design and analysis and real-world problems are not necessarily taught well within research methods courses. For too long, quantitative research methods have relied on lectures, textbooks, proofs, and a cook-book acquisition of computational skills. Current graduate coursework in research methods either focuses on introducing students to a multiplicity of methods with little time to develop deep understanding [ 33 ] or choosing a specific topic (e. g., regression analysis) and teaching it thoroughly to the exclusion of extensions or alternatives. These approaches can be off-putting and generally results in negative stance toward quantitative research [ 34 ]. Field’s [ 35 ] new textbook in statistics shows some sensitivity to this problem by using a comic-book style illustrated science fiction narrative to embed the teaching of statistical logic into an emotionally engaging story line of a young man whose partner has left him. Indeed, a case could be made that the real curricular goals of quantitative methods can only be attained when students master computation and interpretation, have a positive attitude toward using statistics, and develop a clear understanding as to how and why statistics are needed. This can be called thrill, skill, and will [ 36 ].

Possible Solutions

A number of solutions to the “ mile-wide, inch-deep” curriculum [ 37 ] seem feasible. Since research experts cannot be expert in all methods, it makes sense to share our expertise; though funding for this is difficult. A simple, but expensive, solution is the pooling of research methods instruction (e. g., the Bremen International Graduate School of Social Sciences, https://www. bigsss-bremen. de/) or the funding of research training by central government agencies (e. g., Leibniz Education Research Network, https://www. leibniz-gemeinschaft. de/en/research/leibniz-research-alliances/education-research/).

Much cheaper is the use of self-directed instruction, using online tools (e. g., Geoffrey Hubona’s suite of R programming courses on Udemy, https://www. udemy. com/comprcourse/#instructor-1 ; Daniel Soper’s free statistics calculators, http://www. danielsoper. com/statcalc/default. aspx ; the Campbell Collaboration’s online effect size calculators, https://www. campbellcollaboration. org/escalc/html/EffectSizeCalculator-Home. php ; and Bill Trochim’s Research Methods Knowledge Base, http://www. socialresearchmethods. net/kb/index. php ). These resources reduce the need to offer courses or teach students individually and exploits just-in-time and technology-based learning. Nonetheless, quality assurance and relevance of these resources needs to be established before students rush in. Indeed, independent students may invest considerable effort in self-directed learning which turns out to be both inefficient and inappropriate. Hence, the promise of technology to allow independent learning may be illusory and still requires interaction with peers or supervisors.

The automation of statistical analysis is currently feasible [ 38 , 39 ]. Unsurprisingly, these implementations are either commercial or exist only in Python language code form, which is inaccessible to the content-domain expert who may be considerably weaker in statistics or coding. The development of an automated system with a “ wizard”-like user-interface might be able to take advantage of machine learning algorithms and high speed computing to pre-analyse datasets in accordance with specified characteristics of the variables and analytic goals. Such a tool would reduce the burden of becoming expert in all analytic procedures and let users focus more on learning and applying appropriate discipline specific knowledge and theory.

An unsurprising critique of these suggestions is that they focus on the skill of quantitative research methods. What are the solutions for inspiring greater motivation (i. e., will) or excitement (i. e., thrill) among social science and education researchers to develop the skills our data require? The quality of the research environment and supervisory arrangements matter to ensuring that new researchers are “ switched on” [ 40 ]. For example, working with real data that are relevant to the content discipline of students can be expected to raise motivation and interest. More senior academics who work as a mentor for junior researcher on real problems with the goal of co-authorship in a submitted manuscript seems to inspire and motivate, as well as equip, new researchers [ 41 ]. Supportive research environments include ensuring research units have at least five research active staff or post doctorates and at least 10 research students; adequate library and IT facilities; preparation programs that cover the diversity of research skills and knowledge; and the provision of access to seminars, conferences, presentations, and teaching/demonstrating experiences [ 40 ]. This requires moving away from the solitary academic to a team model which is more commonly seen in the pure sciences. If we wish students to enter a community of research scholars, then there must be a community into which they can be inducted, rather than relying solely on the idiosyncrasy of an individual scholar.

Unfortunately, this opinion piece does not come to a definitive conclusion as to what should be done to revisit both the curriculum and pedagogy of quantitative methods so that our graduates will be future capable [ 42 ]. Instead multiple suggestions have been made with little evidence as to their efficacy. These speculations do not disguise the fact that there is little clarity and consensus about what methods and tools students should be taught, what kinds of environments are more effective and long-lasting, or which pedagogical activities and process are most effective. This is a global problem. Perhaps an international Delphi study [ 43 ] of research method instruction experts would be a first step toward agreeing on how skill, will, and thrill in the education of doctoral students in the quantitative social sciences can be achieved.

Author Contributions

The author confirms being the sole contributor of this work and approved it for publication.

Conflict of Interest Statement

The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

References

1. Hodis F, Hancock GR. Introduction to the special issue: advances in quantitative methods to further research in education and educational psychology. Educ Psychol.(2016) 51: 301–4. doi: 10. 1080/00461520. 2016. 1208750

The Complexity of Social Science Research

Difficulties in Teaching Quantitative Methods

Possible Solutions

Author Contributions

Conflict of Interest Statement

References