Re: Those tricky charts
Chad Adelman, Education Sector’s new policy associate, digs into our high-achieving students study and thinks he’s found a smoking gun. In particular, he has a beef with us looking at National Assessment of Educational Progress scores since 2000, instead of 2003:
But No Child wasn’t signed into law until January 2002. The first NAEP tests measuring its true impacts could not have been until 2003, represented by the dotted line.
When we make this correction, the claims in the report do not seem to stand up as well. The lower tenth of performers made gains throughout the chart, but especially from 2000 to 2003, where they gained 13 points to their high achieving peers’ six. Notably, this accounts for almost all the gain claimed in the Fordham report.
Chad, you’re right, though using 2003 as the starting point—more than a year after the law’s enactment—isn’t perfect either. Tom Loveless, author of our NAEP study, discusses this issue at length (see pages 18-20):
Another important consideration concerning time intervals should also now be apparent from examining the NAEP data. Three grade-subject combinations exhibit a consistent pattern, a straightforward story of narrowing gaps during the NCLB era—mostly the result of sharp gains by low-achieving students from 2000 to 2002 or from 2000 to 2003. But whether these years belong in the NCLB era is debatable. The starting point matters. Using the NAEP test immediately before NCLB’s passage as a baseline, as this study does, includes growth that may have nothing to do with NCLB. Selecting a later date—2003, for example—and arguing that the act’s accountability provisions could not have been implemented before then would lead to the conclusion that growth was much less during the NCLB era (although still statistically significant, as shown in appendix A), and that the gaps between low and high achievers were essentially unchanged. But it would also omit influence that NCLB may have had on NAEP scores during the debate and early implementation of the legislation.
Neal and Schanzenbach provide an example. In the fall of 2001, “with the passage of NCLB looming on the horizon,” the state of Illinois placed hundreds of schools on a watch list and declared that future state testing would be high stakes. If such actions influenced educators’ behavior and students’ test scores, an “NCLB effect” may have been registered in 2002. The bottom line is that there is no clear boundary between pre- and post-NCLB periods and no perfect way to delineate the NCLB era using the NAEP test years. Critics and defenders of NCLB alike can (and do) exploit this ambiguity to their advantage. The fairest approach is to point out the large gains in NAEP scores in the period around 1998-2003 and acknowledge that NCLB’s association with these gains is unknown.
P.S. Readers can see the long-term trends for high-achieving students by viewing the foreword to the report, where they are presented since the 1990s. As we wrote, “Looking at long-term NAEP trends for the top 10 percent, one spots a steady line inching ever-so-slowly upward from the early 1990s to today. Enter NCLB, and nothing changes. It’s ‘benign neglect’ in pictures.”
You can skip to the end and leave a response. Pinging is currently not allowed.
June 20th, 2008 at 7:06 pm
I couldn’t care less about whether the methodolgy made the top 10 percentile made the gap look better or worse in comparison to the bottom 10 percentile.
The problem is the same old distortion of the effects of NCLB on the bottom.
Focus on the lower performers and the text of your study provides more ammunition against NCLB. Firstly, you should focus on 8th grade, especially on Reading, where students were subjected to five years of NCLB accountability and resources. If you focus on 2003 to 2007, then the bottom 10 percentile was flat. Focus on 2002 to 2007 and you have an actual decline.
I’d be glad to go into detail why the Reading scores and older scores are more important, but it should be obvious. Math instruction had been more primitive. Fifteen years ago, how many people thought much about teaching elementary teachers to teach math? Improvements were needed but it wasn’t on many peoples’ radar. More importantly, Reading comprehension is virtually life and death in being educated for the 21st century. Even with NAEP scores, we won’t know if 4th grade increases mean much until they are reflected in higher middle school scores, and higher science scores, and higher high school scores, and so on.
This is another example of the difference between the text of main reports and the Executive Summary and the spin. Take a look at the press releases of politicized liberal organizations like the Ed Trust, If you don’t read the actual methodology and evidence, and if you get hypnotized by their pretty graphs, you can conclude that NCLB is working. But I don’t understand why you bought into these tricks.
In fairness though, Eduwonk has a post on the subject that you should read. It illustrates what went wrong with NCLB. I was stunned in 2003 when our new CEO said that we need to accelerate the learning process for low achieving students. He didn’t say we should remove the roadblocks to effective instruction or that we should create efficiencies in schooling. Instead he was instructing us to do something that, for the most part is cognitively impossible. Then I listened to his consultant. Like so many consultants across the nation, the argument was pure math. Since lower performing student were mathematically lower, it was easier for the numbers to go up.
They, and the poster on Eduwonk, didn’t know what they didn’t know.
They didn’t consider that there often are factors in the lives of lower performing students, physical health, mental health, poverty, etc., that impede learning. Often, their academic performance can’t be improved until those realities are addressed.
You may not agree, but you ought to debate based on reality and numbers based on reality, not abstract numbers devoid of any connections with reality.