Study provides nuanced answer to whether KIPP students have higher college graduation rates

KIPP, the largest charter school network in the country, prides itself on its commitment to guiding students to and through college, so a recent study provided a compelling evaluation of its model.

In the study, researchers compared students who won a random lottery to attend a KIPP middle school with those who lost the lottery. The result: both groups attained college degrees at a similar rate of 22%. Mathematica, the company conducting the research, concluded that “KIPP middle schools had little to no impact on four-year degree completion rates.”

However, the researchers then analyzed a subset of students who also attended a KIPP high school, although this portion of the study was not based on a random lottery. The results from this subset were encouraging, showing that attending both a KIPP middle and high school significantly increased students’ chances of enrolling in and completing college.

These findings were highlighted by KIPP officials and the media when the study was released last month.

Combining the results, however, presents a nuanced assessment of KIPP charter schools. The combined outcomes for middle and high school are promising, aligning with KIPP’s track record of improving student learning. Nevertheless, the less encouraging results from middle school alone are based on a larger and more methodologically robust group of students due to the random lottery selection.

As a result, some experts argue that more research is necessary before definitive conclusions can be drawn about KIPP.

“This is suggestive evidence,” said Jon Baron, President for the Coalition for Evidence Based Policy, commenting on the middle and high school results. “This really needs to be tested in further study before being accepted.”

(Baron previously held a position at Arnold Ventures, the philanthropic organization that funded this research.)

The study examined over 2,000 students who applied to attend one of 21 KIPP middle schools in either 2008, 2009, or 2011. The researchers tracked and compared the outcomes of lottery winners and losers over the next decade and beyond.

Prior research has demonstrated that KIPP improves test scores, but this new study reveals limited long-term benefits from attending a KIPP middle school. While there was a suggestion that students were more likely to enroll in college, they persisted and graduated at similar rates as students who lost the KIPP lottery. This analysis forms the study’s “primary analysis.”

However, the Mathematica researchers also conducted an additional “exploratory analysis,” indicating a slightly lower level of confidence in these findings. In this secondary analysis, the study employed a rigorous approach that was not based on a random lottery. (Despite this, several news outlets and commentators inaccurately described these exploratory findings as lottery-based.)

This secondary analysis focused on a smaller group of KIPP middle school students who also attended a KIPP high school. These students demonstrated significant benefits from the extended KIPP education, with 39% of them earning a four-year college degree compared to 20% in a comparison group.

These results, shared with reporters prior to the study’s release, were praised by KIPP leaders.

“The Mathematica study shows that a continuous KIPP education, spanning middle school and high school, is transformative and has the potential to close the educational opportunity gaps faced by Black and Latinx students,” stated Shavar Jeffries, CEO of the KIPP Foundation.

The Mathematica researchers suggest that these gains may be attributed to the emphasis KIPP high schools place on guiding students toward college. The study proposes that “this large effect results from combining the well-established benefits of attending a KIPP middle school (a substantial boost to students’ academic achievement) with the strong emphasis on college-related support found in KIPP high schools.”

However, the two sets of findings present something of a puzzle: the group of KIPP middle and high school students who experienced significant gains in college completion were a small subset of the larger group of middle school students who did not observe any improvements. Nevertheless, the overall impact on college completion for middle school students was estimated to be close to zero. Ira Nichols-Barrer, one of the Mathematica researchers, suggests that this is because the middle and high school group constituted a small subset of the entire sample.

He affirms that further research would help clarify the discrepancy between the study’s two major findings. “Our hope is that this study is not the final phase,” he added.