Before-After-Control-Impact (BACI) designs are used to study ecological responses in large experimental units (e.g., lakes, forests and mesocosms) for which replication is difficult or impossible. Two units are monitored over time; one unit receives an intervention at some intermediate time, while the other is left as an undisturbed control. The pre-intervention differences in the response between units are compared to the post-intervention differences, with a large disparity interpreted as evidence of an effect of the intervention.
To compare the pre- and post-intervention differences, ecologists have typically used two-sample t-tests or randomization tests, both of which assume independence of successive observations. Furthermore, the interpretation of positive test results as evidence of intervention effects depends on the untestable assumption that the response trajectories in the two units would have been exactly parallel in the absence of the intervention. I develop a method that adjusts for serial correlation of successive observations and makes an ad hoc correction for unit-to-unit variability in response trajectories. Application of the old and new methods to simulated and real data sets shows that BACI analysis greatly overstates the evidence for intervention effects, and that the new method is extremely conservative. I conclude that it is probably best to avoid statistical testing on data from these unreplicated designs.