The Charter Blog

Different by Design: A Review of Three Papers by Nat Malkus

Over the past six months, Nat Malkus from the American Enterprise Institute published a series of three papers that compare charter public schools with district public schools by looking at differences in their demographics, proficiency rates, and suspension rates: Differences on Balance, Unlike Their Neighbors, and Differences by Design. Malkus’ work is the first of its kind to study this issue at a national level and in a balanced and systematic way. Further, his analyses reveal “important patterns of differences” between charter and district public schools.

For all three of his papers, Malkus utilizes the same methodology. Because this is such an extensive exploration of how charter schools compare to their district school peers, it’s worth taking a look at the methodology and the data.

Methodology Overview:

In conducting his research, Malkus aggregated 2011-12 data from three primary sources, all collected by the US Department of Education: the Common Core of Data (CCD) for grade configuration, location, and demographic information; the Civil Rights Data Collection (CRDC) database for information on English language learners, students with disabilities, and suspension rates; and the EDFacts database for proficiency information.

Malkus then attempted to match each charter school in the country with five comparable district schools using their location, authorizer, and grade configuration. Charter schools authorized by school districts were only matched with other district schools in the same district and charter schools authorized by other entities were matched with district schools from any nearby district. In addition, charter schools were matched with district schools with similar grade configurations that were less than 30 miles away.

Of the approximately 5,700 charter schools with 2011-12 data, Malkus excluded nearly 900 schools because they were virtual schools, special-purpose schools, or too small to form an adequate comparison group. Of the remaining 4,800 charter schools, 4,300 were matched with five comparable district public schools. Thus, only 75 percent of the 5,700 charter schools with 2011-12 data are included in the study.

Malkus then assessed the composition of each charter school relative to the average composition of the district school comparison group across eight dimensions: percentage of Black students; percentage of Hispanic students; percentage of White students; percentage of low-income students; percentage of special education students; percentage of English language learners; proficiency rates; and suspension rates.

Malkus then categorized each charter school according to their similarity or dissimilarity (along each of the eight dimensions) with the district public school comparison group:

  • substantially less—negative differentials of 20 percentage points or more;
  • somewhat less—negative differentials of 6 to 19 percentage points;
  • similar—differentials that are within 5 percentage points on either side;
  • somewhat more—positive differentials of 6 to 19 percentage points; and
  • substantially more—positive differentials of 20 percentage points or more.

In other words, if 63 percent of the students in a given charter school were low-income students, compared to just 40 percent of the students in the five school comparison group, then the charter school would be considered to have “substantially more” low-income students.

In comparing charter schools to their district school peers, Malkus finds considerable differences among and across states. In some states, such as Ohio, the charter sector as a whole served far more Black (35 percent served “substantially more”) and low-income students (41 percent served “substantially more”).

In other states, such as North Carolina, the charter sector served much smaller proportions of Black (28 percent served “substantially less”), Hispanic (12 percent served “substantially less” and 61 percent served “somewhat less”) and low-income students (53 percent served “substantially less”). However, far more North Carolina charter schools had markedly higher proficiency rates than their neighboring district schools (52 percent produced “substantially more” proficient students).

Most states fell somewhere in between these two extremes. Further, the charter sector as a whole defies simple and sweeping descriptions—as more charter schools end up at the top and the bottom of most distributions. At the national level, the picture is again mixed—some states have charter schools that served far more disadvantaged students and other states have charter schools that served far less.

Important Limitations:

  • The data are five years old. In the world of charter schools, which have only been around for 25 years, five years is a long time. The demographic composition of the public school sector (and the charter sector more specially) can change significantly over a period of five years. In 2011-12, there were 2.1 million charter school students and in 2016-17 there are an estimated 3.1 million charter school students—which represents a nearly 50 percent increase over the past five years. Urbanization and gentrification have also been dramatically reshaping the demographic composition of many larger cities and urban school districts across the country.
  • The matching parameter of 30 miles or less is unrealistic. Given the context that charter schools operate under the auspices of parental choice—it’s unrealistic to assume that two schools that are 30 miles apart from each other are actually comparable. That is, virtually no parents or students would entertain the idea of attending a school that is 30 miles away—even under a robust school choice system.
  • Based on Malkus’ methodology, 25 percent of charter schools with available 2011-12 data are excluded from the study. Charter schools are different by design. Many charter schools are much smaller than district schools and many charter schools are “special-purpose schools” that fill a niche that is not well served by school districts. However, Malkus’ methodology excludes 25 percent of all charter schools with available data in 2011-12. That fact that 25 percent of charter schools are without a peer group under Malkus’ methodology drives home a central point—charter schools are often different by design.
  • There are known limitations with these datasets that disproportionally impact charter schools. According to a 2013 report by the U.S. Government Accountability Office (GAO), it was not possible to compare “ELL enrollment in charter schools to ELL enrollment in [district] public schools [because] the only available data on school-level ELL enrollment were unreliable and incomplete. Specifically, for over one-third of charter schools, the field for reporting the counts of ELLs enrolled in ELL programs was left blank. These blank fields cannot reliably be interpreted to mean that the charter schools did not have ELLs enrolled.” Further, it has been well documented that the number of low-income students is often underreported in charter schools because they often lack the funding for federally compliant kitchen facilities—and are thus less likely to participate in free and reduced price lunch programs (the primary educational proxy for socio-economic status).
  • Over the past few years, charter schools have made significant strides in addressing issues surrounding their suspension and expulsion rates. A report by the Center for Reinventing Public Education (CRPE), which studied outcomes and policies in Washington, D.C., and New Orleans, found that “in both cities, discipline numbers are declining (New Orleans saw big changes in expulsion, for example, but suspensions, which are far more common, remain outside the centralized system).” In addition, a recent report by the National Association of Charter School Authorizers (NACSA), found that charter school expulsion rates “declined nearly 63 percent” between 2011-12 and 2014-15.
Malkus summarizes his work as follows: “these comparisons dispel some myths, confirm some differences, and reveal new ones, showing a diversity in charter schools that defies simple characterization.” I couldn’t agree more. And it is important to remember that charter schools are indeed different by design.