In the first scenario, there is considerable variation in student performance within each school at the start of the experiment, but average performance is quite similar across all the schools.In the second scenario, the situation is just the reverse: at the start of the experiment, students within a school perform vary similarly, but there is considerable variation in performance among the schools.The of the impact estimate attaches a numerical value to this uncertainty.
To come up with the right sample sizes, it is important to know something about the extent of variation that exists both among and within the units to be randomized.
Imagine, for example, two scenarios in which schools are the entities being randomly assigned to treatment or control conditions.
The presence of two sources of sampling error means that the impact estimates produced by a cluster randomized trial, although unbiased, will inevitably be less precise than the impact estimates produced by a randomized trial involving the same number of ungrouped individuals.
This is a key factor that researchers consider when they design cluster randomized trials, particularly when they determine the number of groups (e.g., schools, agencies) and the number of individuals in each group needed to produce impacts that are both statistically significant and large enough to be policy-relevant.
If a different mix of individuals had been randomly assigned to a study’s treatment and control groups, a somewhat different impact estimate would have been obtained.
This reality, which can be referred to as “randomization error,” creates uncertainty about whether the estimated impact is the true impact of the intervention.
In this methodology issue focus, the first in a series, we explain one such design, has been at the forefront of both the theoretical refinement and practical use of this methodology.
As the name suggests, cluster random assignment means the random assignment of whole groups, or clusters, of people.
Unless both kinds of sampling error are included in the standard error, investigators may wrongly decide that a program is making a significant difference when, in fact, it is not.
The strategy that Bloom and other leading social scientists employ in cluster randomized trials — referred to as “multilevel modeling” or as “hierarchical modeling” — takes account of both sources of sampling error in producing impact estimates.