Background Early life stress (ELS) can compromise development, with higher amounts of adversity linked to behavioral problems. To understand this linkage, a growing body of research has examined two brain regions involved with socioemotional functioning—amygdala and hippocampus. Yet empirical studies have reported increases, decreases, and no differences within human and nonhuman animal samples exposed to different forms of ELS. This divergence in findings may stem from methodological factors, nonlinear effects of ELS, or both. Methods We completed rigorous hand-tracing of the amygdala and hippocampus in three samples of children who experienced different forms of ELS (i.e., physical abuse, early neglect, or low socioeconomic status). Interviews were also conducted with children and their parents or guardians to collect data about cumulative life stress. The same data were also collected in a fourth sample of comparison children who had not experienced any of these forms of ELS. Results Smaller amygdala volumes were found for children exposed to these different forms of ELS. Smaller hippocampal volumes were also noted for children who were physically abused or from low socioeconomic status households. Smaller amygdala and hippocampal volumes were also associated with greater cumulative stress exposure and behavioral problems. Hippocampal volumes partially mediated the relationship between ELS and greater behavioral problems. Conclusions This study suggests ELS may shape the development of brain areas involved with emotion processing and regulation in similar ways. Differences in the amygdala and hippocampus may be a shared diathesis for later negative outcomes related to ELS.