Meta-Analysis Notes
Jamie DeCoster
Department of Psychology
University of Alabama
348 Gordon Palmer Hall
Box 870348
Tuscaloosa, AL 35487-0348
Phone: (205) 348-4431
Fax: (205) 348-8648
September 19, 2004
These were compiled by Jamie DeCoster, partially from a course in meta-analysis taught by Alice Eagly at
Northwestern University. Handbook references refer to Cooper & Hedges (eds.), The Handbook of Research
Synthesis. If you wish to cite the contents of this document, the APA reference for them would be
DeCoster, J. (2004). Meta-analysis Notes. Retrieved
from http://www.stat-help.com/notes.html
For future versions of these notes or for help with data analysis visit
http://www.stat-help.com
ALL RIGHTS TO THIS DOCUMENT ARE RESERVED.
Contents
1 Introduction and Overview 2
2 Formulating a Research Problem 5
3 Searching the Literature 7
4 Coding Studies 10
5 Calculating Mean Difference Effect Sizes 15
6 Calculating Correlation Effect Sizes 25
7 Issues in Calculating Effect Sizes 29
8 Describing Effect Size Distributions 32
9 Examining Moderating Variables 37
10 Writing Meta-Analytic Reports 42
11 Critically Evaluating a Meta-Analysis 46
1
Chapter 1
Introduction and Overview
1.1 Basics
• Definition of meta-analysis (from Glass, 1976): The statistical analysis of a large collection of analysis
results for the purpose of integrating the findings.
• The basic purpose of meta-analysis is to provide the same methodological rigor to a literature review
that we require from experimental research.
• We refer to the direct investigation of human or animal data as “primary research.” Providing a report
of primary research using statistical methodology and analysis is called “quantitative synthesis” or
“meta-analysis.” A report of primary research using traditional, literary methods is called a “narrative
review.”
• Meta-analyses are generally centered on the relationship between one explanatory and one response
variable. This relationship, “the effect of X on Y,” defines the analysis.
• Meta-analysis provides an opportunity for shared subjectivity in reviews, rather than true objectivity.
Authors of meta-analyses must sometimes make decisions based on their own judgment, such as when
defining the boundaries of the analysis or deciding exactly how to code moderator variables. However,
meta-analysis requires that these decisions are made public so they are open to criticism from other
scholars.
• Meta-analyses are most easily performed with the assistance of computer databases (Microsoft Access,
Paradox) and statistical software (DSTAT, SAS).
1.2 Criticisms of Narrative Reviews
• The sample of studies examined in a narrative review is based on the author’s whim, rather than on
publicly shared standards.
• Narrative reviews rely on statistical significance for evaluating and comparing studies. Significance
is dependent on sample size so a weak effect can be made to look stronger simply but adding more
participants.
• Narrative reviews lack acceptable rules of inference for going from the findings of studies to overall
generalizations about the research literature.
• Narrative reviews are not well-suited for analyzing the impact of moderating variables. Authors of
narrative reviews rarely reach clear conclusions regarding how methodological variations influence the
strength of an effect. They also typically fail to report the rules they use to classify studies when
looking for the effect of a moderating variable.
2
• Many research literatures have grown too large to for a human to accurately synthesize without the
aid of statistical inference.
1.3 Types of meta-analyses
• By far the most common use of meta-analysis has been in quantitative literature reviews. These are
review articles where the authors select a research finding or “effect” that has been investigated in
primary research under a large number of different circumstances. They then use meta-analysis to
help them describe the overall strength of the effect, and under what circumstances it is stronger and
weaker.
• Recently, as knowledge of meta-analytic techniques has become more widespread, researchers have
begun to use meta-analytic summaries within primary research papers. In this case, meta-analysis
is used to provide information supporting a specific theoretical statement, usually about the overall
strength or consistency of a relationship within the studies being conducted. As might be expected,
calculating a meta-analytic summary is typically a much simpler procedure than performing a full
quantitative literature review.
1.4 Steps to Perform a Meta-Analysis
1. Define the theoretical relationship of interest.
2. Collect the population of studies that provide data on the relationship.
3. Code the studies and compute effect sizes.
4. Examine the distribution of effect sizes and analyze the impact of moderating variables.
5. Interpret and report the results.
1.5 Criticisms of Meta-Analyses (and Responses)
• Meta-analysis adds together apples and oranges. The purpose of a literature review is to generalize over
the differences in primary research. Overgeneralization can occur just as easily in narrative reviews as
it can in meta-analysis.
• Meta-analysis ignores qualitative differences between studies. Meta-analysis does not ignore these dif-
ferences, but rather codes them as moderating variables. That way their influence can be empirically
tested.
• Meta-analysis is a garbage-in, garbage-out procedure. This is true. However, since the specific content
of meta-analyses is always presented, it should be easier to detect poor meta-analyses than it would
be to detect poor narrative reviews.
• Meta-analysis ignores study quality. The effect of study quality is typically coded as a moderator, so
we can see if there is any difference between good and bad studies. If a difference does exist, low quality
studies can be removed from analysis.
• Meta-analysis cannot draw valid conclusions because only significant findings are published. Meta-
analyses are actually less affected by this bias than narrative reviews, since a good meta-analysis
actively seeks unpublished findings. Narrative reviews are rarely based on an exhaustive search of the
literature.
• Meta-analysis only deals with main effects. The effect of interactions are examined through moderator
analyses.
3
• Meta-analysis is regarded as objective by its proponents but really is subjective. Meta-analysis relies on
shared subjectivity rather than objectivity. While every analysis requires certain subjective decisions,
these are always stated explicitly so that they are open to criticism.
4
Chapter 2
Formulating a Research Problem
2.1 Defining the Research Question
• There are several things you should consider when selecting a hypothesis for meta-analysis.
1. There should be a significant available literature, and it should be in a quantifiable form.
2. The hypothesis should not require the analysis of an overwhelming number of studies.
3. The topic should be interesting to others.
4. There should be some specific knowledge to be gained from the analysis. Some reasons to perform
meta-analyses are to
◦ Establish the presence of an effect.
◦ Determine the magnitude of an effect.
◦ Resolve differences in a literature.
◦ Determine important moderators of an effect.
• When performing a meta-analytic summary you often limit your interest to establishing the presence of
an effect and estimating its size. However, quantitative literature reviews should generally go beyond
this and determine the what study characteristics moderate the strength of the effect.
• The first step to defining your research question is to decide what theoretical constructs you will use
as your explanatory and response variables.
• You need to decide what type of effect size you will use. If the explanatory variable is typically
presented as a categorical variable, you should probably use d. If the explanatory variable is typically
presented as a continuous variable, you should probably use r.
• If you decide to use the effect size d, you then need to precisely define what contrast you will use to
calculate d. For a simple design, this will probably be (mean of experimental group - mean of control
group). Defining the contrast also specifies the directionality of your effect size (i.e., the meaning of
the sign). The directionality is automatically determined for the effect size r once you choose your
constructs.
2.2 Limiting the Phenomenon of Interest
• Once you have determined what effect you want to examine, you must determine the population in
which you want to examine it. If you are performing a meta-analytic summary you will often chose
very practical boundaries for your population, such as the experiments reported in a specific paper.
The populations for quantitative literature reviews, however, should be defined on a more abstract,
theoretical level. In the latter case you define a specific set of inclusion and exclusion criteria that
studies must meet to be included in the analysis.
5
• The goal of this stage is to define a population that is a reasonable target for synthesis. You want
your limits narrow enough so that the included studies are all examining the same basic phenomenon,
but broad enough so that there is something to be gained by the synthesis that could not easily be
obtained by looking at an individual study.
• The first criterion you must have is that the studies need to measure both the explanatory and response
variables defining your effect and provide an estimate of their relationship. Without this information
there is nothing you can do with a study meta-analytically.
• Each additional criterion that you use to define the population of your meta-analysis should be written
down. Where possible, you should provide examples of studies that are included or excluded by the
criterion to help clarify the rule.
• You should expect that your list of inclusion and exclusion criteria will change during the course of
your analysis. Your perception of the literature will be better informed as you become more involved
in the synthesis, and you may discover that your initial criteria either cut out parts of the literature
that you want to include, or else are not strict enough to exclude certain studies that you think are
fundamentally different from those you wish to analyze. You should feel free to revise your criteria
whenever you feel it is necessary, but if you do so after you’ve started coding you must remember
recheck studies you’ve already completed.
• It is a good practice to keep a list of the studies that turned up in your initial search but that you
later decided to exclude from your analysis. You should also record exactly what criterion they failed
to meet, so that if you later decide to relax a particular criterion you know exactly what studies you
will need to re-examine, saving you from having to perform an entirely new literature search.
6
Chapter 3
Searching the Literature
3.1 Basic Search Strategy
• Once you determine the boundaries of your meta-analysis, you need to locate all of the studies that
fit within those bounds. When performing a meta-analytic summary you will sometimes know at the
start exactly what studies you want to include. For other summaries, and for all quantitative literature
reviews, you will need to perform a detailed search to locate all the studies that have examined the
effect of interest within the population you defined.
• The steps to a comprehensive literature search are:
1. Search the literature to find possible candidates for the analysis using fairly open guidelines. You
should try to locate all of the studies that truly meet your criteria, even if your searches also
include a large number of irrelevant studies. More specific detail on this will be provided in
section 3.2.
2. Compile a master candidate list. Many studies will turn up in several of your searches, so you
need to combine the results into a list where each study only appears once.
3. Gain access to each of these studies for examination. Some of the studies will be available at your
library, while others will have to be obtained either through interlibrary loan or directly from the
authors.
4. Examine each of the studies on this list and determine whether they meet your criteria for inclusion
in the meta-analysis. You should start by reading the title and abstract and then continue to the
methods and results sections if you need more information to make your decision.
• You want to make sure that your master candidate list includes all of the studies you might be interested
in, even if this also means including many studies that you do not use. It is not uncommon to discard
over 90% of the studies from the initial list.
• You do not need to copy every study in the master candidate list. Many of these you will reject with
just a few minutes of reading. However, you will want to copy each article in your final meta-analytic
sample.
• Performing a comprehensive search of the literature involves working with a huge amount of informa-
tion. You would be well-advised to make use of a spreadsheet or a database program to assist you in
this task. For each study in the master candidate list you should record
1. A terse reference to the study (such as journal name, volume number, and starting page number)
2. The journal or book call number (if your library organizes its material by call number)
3. Where you can find the study or its current retrieval status (requested from author, requested
through interlibrary loan, etc.)
7
4. Whether the study was included or excluded from the analysis
5. What criterion was used for exclusion (if the study was excluded from the meta-analysis)
• If you want to provide an accurate estimate of an effect it is important to find unpublished articles for
your analysis. Many studies have shown that published articles typically favor significant findings over
nonsignificant findings, which biases the findings of analyses based solely on published studies.
• You should include foreign studies in your analysis unless you expect that cross-cultural differences
would affect the results and you lack enough foreign studies to test this difference. The AltaVista Trans-
lation website (http://babelfish.altavista.digital.com/cgi-bin/translate?) can be useful when trying to
read foreign documents.
• Sometimes the number of studies that fit inside your boundaries is too large for you to analyze them
all. In this case you should still perform an exhaustive search of the literature. Afterwards, you choose
a random sample of the studies you found for coding and analysis.
3.2 Specific Search Procedures
• Computerized Indices. A number of databases are available on CD-ROM or over the internet. These
will allow you to use keywords to locate articles relevant to your analysis.
◦ Selecting the keywords for your search is very important. First, you should determine the basic
structure of what you want in your search. For example, lets say you want to find studies that
pair the terms related to “priming” with terms related to “impression formation.”
◦ You should next determine the synonyms that would be used for these terms in the database.
For example, some researchers refer to priming effects as implicit memory effects. Similarly,
researchers sometimes refer to an impression formation task as a person judgment task. You
therefore may want your search to retrieve studies that use pair either “priming” or “impression
formation” with either “impression formation” or “person judgment.” Many indices, such as
PsycInfo, publish a thesaurus that should make finding synonyms easier. If the index has pre-
defined subject terms you should make sure that your list of synonyms includes all the relevant
subject words.
◦ Most indices support the use of wildcards, which you should use liberally. To locate research on
priming in PsycInfo we might use the search term PRIM*, which would find studies that use the
terms PRIMING, PRIMES, PRIMED, and other words beginning with PRIM.
◦ You should then enter your search into the database. Each construct will be represented by a
list of synonyms connected by ORs. The constructs themselves will be connected by ANDs. In
the example above we might try (prim* OR implicit memory) AND (impression formation OR
person judgment).
◦ Be sure to use parentheses to make sure that the computer is linking your terms the way you
want. For example, searching for (A OR B) AND C will give very different results from A OR (B
AND C).
◦ If your initial search produces a large number of irrelevant studies related to a single topic, you
might try to keep them out of further searches by introducing a NOT term to your search. This
will exclude all records that have the specified term in the document. For example, if our priming
search produced a large number of irrelevant studies related to advertising that we wanted to
exclude, we might revise our search to be (prim* OR implicit memory) AND (impression formation
OR person judgment) NOT (ads OR advertising)
◦ Whenever you conduct a computerized search you should record the name of the database, the
years covered by the database at the time of the search, and the search terms you used. You will
need to report all of this in your article.
◦ The databases most commonly used by psychologists are:
8
1. PsycLit/PsycInfo (PsycLit is the CD-ROM version, and is less complete)
2. ERIC (Educational Resources Information Center)
3. Dissertation Abstracts Online
4. ABI/Inform (a worldwide business management and finance database)
5. Sociological Abstracts (sociology literature)
6. MEDLINE (biomedical literature including health care, clinical psychology, gerontology, etc.)
7. Mental Health Abstracts
There are also a number of databases available within more specialized research areas.
◦ You should search every computerized index that might possibly have studies related to your topic.
Don’t be afraid to look outside your own field. However, you should keep in mind that different
indices use different terms, so you may have to define your search differently when working with
different databases.
• Descendant search. If you can locate a small number of important studies that were performed at
early dates, you can use the SSCI (Social Science Citation Index) or SCI (Science Citation Index)
to locate later articles that cite them in their references. This is a very nice complement to the
standard computerized search, and can now be performed fairly easily since both indices are available
on CD-ROM.
• Ancestor search. You should always examine the references of articles that you decide to include in
your analysis to see if they contain any relevant studies of which you are unaware.
• Research registers. Research registers are actively maintained lists of studies centered around a common
theme. Currently there are very few research registers available for psychological research, but this
may change with the spread of technology.
• Reference lists of review articles. Previous reviews, whether they included a meta-analysis or not, are
often a fruitful place to look for relevant studies.
• Hand search of important journals. If you find that many of your articles are coming from a specific
journal, then you should go back and read through the table of contents of that journal for all of the
years that there was active research on your topic. You might make use of Current Contents, a journal
containing a listing of the table of contents of other journals.
• Programs from professional meetings. This is a particularly good way to locate unpublished articles,
since papers presented at conferences are typically subject to a less restrictive review (and are there-
fore less biased towards significant findings) than journal arti
本文档为【Meta-Analysis Notes】，请使用软件OFFICE或WPS软件打开。作品中的文字与图均可以修改和编辑，
图片更改请在作品中右键图片并更换，文字修改请直接点击文字进行修改，也可以新增和删除文档中的内容。
该文档来自用户分享，如有侵权行为请发邮件ishare@vip.sina.com联系网站客服，我们会及时删除。
[版权声明] 本站所有资料为用户分享产生，若发现您的权利被侵害，请联系客服邮件isharekefu@iask.cn，我们尽快处理。
本作品所展示的图片、画像、字体、音乐的版权可能需版权方额外授权，请谨慎使用。
网站提供的党政主题相关内容(国旗、国徽、党徽..)目的在于配合国家政策宣传，仅限个人学习分享使用，禁止用于任何广告和商用目的。
浙公网安备 33021202002483