In: Statistics and Probability
Quantatives Causal-Comparative Research Sometimes researchers
are interested in exploring the reasons behind existing differences
between two or more groups. Such studies are known as
causal-comparative studies. In a sense, this type of research is
similar to correlational research in that it intends to study
conditions that have already occurred. Data are collected to try to
determine why one group is different from another (Johnson,
2008; Mertler, 2014). Causal-comparative research designs are also
referred to as ex post facto-or “after-the-fact”—designs. The
reason for this is that the study first observes a difference that
exists within a group of people, for example, and then looks back
in time
to determine possible conditions that might have resulted in this
observed difference.
The researcher is looking for a possible cause “after the fact,”
since both the precursory conditions and the resulting differences
have already occurred; that is, the study is taking place
retrospectively (Gay et al., 2009). Two or more groups are compared
to find a “cause” for—or consequences of—existing differences in
some sort of measurement or score (Fraenkel et al., 2012; Johnson,
2008). However, it is once again important to note that
causal-comparative research cannot establish true cause and
effect—as experimental
research can—because no variables are being manipulated.
The Causal-Comparative Research Process.
Similar to correlational research, the steps in conducting a
causal-comparative research study are fairly simple. One of the
substantive differences is that analysis of causal-comparative data
can involve a wider variety of statistical techniques (Gay et al.,
2009). The steps in conducting a causal-comparative
research study are as follows:
1. Identification of the topic/problem to be studied. Problem
identification in a causalcomparative study begins by identifying a
phenomenon of interest and then considering possible causes for, or
consequences of, that phenomenon (Fraenkel et al., 2012). Once
possible causes have been identified, they are typically
incorporated into a formal problem statement and research questions
or hypotheses. Similar to the process in correlational studies,
possible causes or consequences should be identified based on some
logical rationale. Again, research attempting to investigate a
large number of variables just to “see what turns up” should be
avoided entirely. Potential research questions are stated in terms
of group differences (e.g., How does teacher training—traditional
versus alternative—affect empathy in the classroom? or Does gender
have an effect on mathematical problem-solving skills?).
2. Review of related literature. Conducting a literature review in
a causal-comparative study can provide guidance in the
identification of possible causes or consequences of a particular
phenomenon. Related literature may also aid the researcher in
making methodological decisions, including those related to methods
and instrumentation for data collection and data analysis.
3. Identification and selection of participants. One of the most
important factors in selecting a sample in this type of study is to
carefully define the characteristic(s) that will serve as the
grouping variable(s), and then be sure to select groups that differ
specifically and measurably on the characteristic(s). Further, and
beyond consideration of the grouping variable, it is important to
select groups that are as homogeneous on other factors as possible.
Of course, this is impossible to accomplish with respect to all the
other factors that can influence human behavior, but measures
should be taken to try to control these other influences. Often,
the success of a causal-comparative study depends largely on how
carefully the comparison groups have been defined (Fraenkeletal.,
2012). Typically, the groups will differ in one of two ways: (1)
One group will possess a characteristic that the other group does
not, or (2) both groups will possess the same characteristic(s) but
to differing degrees or amounts (Gay et al., 2009). Since there is
only limited control within a causal-comparative design, it is best
to select
participants randomly from the two (or more) well-defined
populations, or groups. It is advisable to have a minimum of 30
participants in each group.
4. Specification of the design and procedures for data collection.
At this point, I will introduce some common notation used to depict
research designs. The symbols are as follows:
T = Treatment condition
O = Observation or measurement
EXP-GRP = Experimental group
CO-GRP = Control or comparison group
GRP = Nondescript group
GRP1, GRP2, . . . = Subscripts denote different groups
Using this notation, the basic causal-comparative design appears.
this design, two groups are determined based on the presence or
absence—or differing degree—of the characteristic of interest. Each
group is then measured on the dependent variable, and the
subsequent scores are statistically compared by group.
5. Collection of data. There are essentially no limits to what can
be used as instrumentation or sources for data collection in
causal-comparative studies, provided that the resulting data are
quantitative.
6. Analysis of data. The analysis of causal -comparative data
involves calculation of both descriptive and inferential
statistics, as well as the statistical comparison of two or more
groups on some quantitative variable. In Chapter 13, you will learn
that there are numerous methods for conducting statistical group
comparisons. They vary depending on the number of groups being
compared, the number of dependent variables being measured, and the
underlying purpose of the causal-comparative research study.
7. Answering research questions and drawing conclusions. The
results of the causal -comparative analyses should permit the
researcher to answer the guiding research questions, or address the
hypothesis, of the study. However, it is critical to remember that
interpreting the findings of a causal-comparative study requires a
good deal of caution
on the part of the researcher. Even when taking measures to ensure
that the groups being compared are relatively equivalent—with the
exception of the grouping variable—it is difficult to establish any
sort of cause-and-effect conclusions with any degree of
confidence.
It is examined how the impact of development on the overall college students population:
We study would randomly assign students to two treatment groups, one using the experimental materials and the other using a widely established comparative program. The students would be taught the entire curriculum, and a test administered at the end of instruction would provide unequivocal results that would permit one to identify the more effective treatment.
The truth is that conducting definitive comparative studies is not simple, and many factors make such an approach difficult. Student placement and curricular choice are decisions that involve multiple groups of decision makers, accrue over time, and are subject to day-to-day conditions of instability, including student mobility, parent preference, teacher assignment, administrator and school board decisions, and the impact of standardized testing. This complex set of institutional policies, school contexts, and individual personalities makes comparative studies, even quasi-experimental approaches, challenging, and thus demands an honest and feasible assessment of what can be expected of evaluation studies (Usiskin, 1997; Kilpatrick, 2002; Schoenfeld, 2002; Shafer, in press).
Comparative evaluation study is an evolving methodology, and our purpose in conducting this review was to evaluate and learn from the efforts undertaken so far and advise on future efforts. We stipulated the use of comparative studies as follows:
Suggested Citation:"5 Comparative Studies." National Research Council. 2004. On Evaluating Curricular Effectiveness: Judging the Quality of K-12 Mathematics Evaluations. Washington, DC: The National Academies Press. doi: 10.17226/11025. A comparative study was defined as a study in which two (or more) curricular treatments were investigated over a substantial period of time (at least one semester, and more typically an entire school year) and a comparison of various curricular outcomes was examined using statistical tests. A statistical test was required to ensure the robustness of the results relative to the study’s design.
We read and reviewed a set of 95 comparative studies. In this report we describe that database, analyze its results, and draw conclusions about the quality of the evaluation database both as a whole and separated into evaluations supported by the National Science Foundation and commercially generated evaluations. In addition to describing and analyzing this database, we also provide advice to those who might wish to fund or conduct future comparative evaluations of mathematics curricular effectiveness. We have concluded that the process of conducting such evaluations is in its adolescence and could benefit from careful synthesis and advice in order to increase its rigor, feasibility, and credibility. In addition, we took an interdisciplinary approach to the task, noting that various committee members brought different expertise and priorities to the consideration of what constitutes the most essential qualities of rigorous and valid experimental or quasi-experimental design in evaluation. This interdisciplinary approach has led to some interesting observations and innovations in our methodology of evaluation study review.
This chapter is organized as follows:
Comparative Study that count disaggregated by program and program type.
Seven critical decision points and identification of at least minimally methodologically adequate studies.
Definition and illustration of each decision point.
A summary of results by student achievement in relation to program types (NSF-supported, University of Chicago School Mathematics Project (UCSMP), and commercially generated) in relation to their reported outcome measures.
A list of alternative hypotheses on effectiveness.
Filters based on the critical decision points.
An analysis of results by subpopulations.
An analysis of results by content strand.
An analysis of interactions among content, equity, and grade levels.
Discussion and summary statements.
our methodology and its connection to the results stated and conclusions drawn. In the spirit of scientific, fair, and open investigation, we welcome others to undertake similar or contrasting approaches and compare and discuss the results. Our work was limited by the short timeline set by the funding agencies resulting from the urgency of the task. Although we made multiple efforts to collect comparative studies, we apologize to any curriculum evaluators if comparative studies were unintentionally omitted from our database.