The Education Gadfly
A Weekly Bulletin of News and Analysis from the Thomas B. Fordham Institute
February 19, 2009, Volume 9, Number 7
New from Fordham: The Accountability Illusion
Today the Thomas B. Fordham Institute and the Kingsbury Center at Northwest Evaluation Association released a groundbreaking new study, The Accountability Illusion. It peels back layers of the No Child Left Behind Act as implemented and reveals an enormously uneven and misleading system of school accountability. Analysts took 36 real schools (18 elementary, 18 middle) and "moved" them from state to state (28 states in all) to see how many would make "Adequate Yearly Progress" (AYP) under each state's NCLB rules. The alarming results? In some states, nearly all of the elementary schools would make AYP while in others practically none of them would. Mind you, these are the exact same schools. This tells us that the present system isn't working. A school's AYP status depends at least as much on what state it's in as on the performance of its students.
Click here to find the study and its related materials, including the "Fix that Failing School" video game.
This week on the Education Gadfly Show Podcast: Sweet chili Petrilli
Contents
From Checker's, Mike's, and Amber's Desks
Recommended Reading
Flypaper's Finest (The best from Flypaper, Fordham's blog)
The Education Gadfly Show Podcast
Short Reviews
- Closing the Expectations Gap: Fourth Annual 50-State Progress Report on the Alignment of High School Policies with the Demands of College and Careers
- Impact of For-Profit and Nonprofit Management on Student Achievement:
- An Evaluation of Teachers Trained Through Different Routes to Certification: Final Report
From Our Readers
Announcements
From Checker's, Mike's, and Amber's Desks
The accountability illusion
Take away all the jargon, emotion, envy, confusion, and embarrassment and much of the No Child Left Behind debate comes down to this: Which schools are good, which are bad, and does NCLB do a decent job of telling the difference?
The short answer, provided by a major new study from Fordham and the Kingsbury Center at the Northwest Evaluation Association, is no, not by a mile.
The analysis is complex and the report is long but its premise is simple: Take a set of real schools, pretend that we can drag them across the map and drop them down in various states, and see how many would make "Adequate Yearly Progress" (AYP) in each place. If the U.S. had something akin to a shared notion of what it means to be a good or bad school, we wouldn't see a whole lot of variation.
Yet we found nearly the opposite. In a few of the 28 states we studied (e.g, Wisconsin, Arizona), almost all of the elementary schools in our sample made AYP, while in other jurisdictions (e.g., Massachusetts, Nevada), almost none did. Putting it bluntly, most of the schools in our sample would be considered failures in some states but perfectly okay, even praiseworthy, in others. These are the same exact schools, mind you. Same students. Same teachers. Same achievement. What's different--sometimes drastically different--are the arcane AYP rules that vary from state to state.
Such variation surely existed before NCLB. Does it matter that it exists now?
Yes, for three reasons. First, it surely demoralizes educators (and probably students, parents, taxpayers, etc.) to know that their very own schools, deemed "in need of improvement" under NCLB, would be considered acceptable, even praiseworthy, if located elsewhere. Such capriciousness not only breeds cynicism toward NCLB; it undermines the idea of accountability itself--and certainly retards efforts to revitalize horrific schools and boost low-performing pupils.
Second, what drives the state-to-state variation in AYP results isn't principled differences about what it means to be a good school. Instead, obscure, deep-in-the-bureaucracy decisions around "cut scores," "annual measurable objectives," "minimum n sizes," and "confidence intervals" create these discrepant and discordant outcomes. We'd mind less if the variation were based on things that truly matter: whether schools are judged on their progress over time, for example, rather than each year's absolute performance; whether they're helping all students make gains or just those below a fixed bar; whether determinations hinge solely on reading and math or include other core subjects like science and history; and so forth. Those would be legitimate reasons for discrepancies, issues worth arguing over. But that's not what we're seeing here. Rather, we witness state education departments going through subtle machinations to create outcomes that they judge sensible, or at least politically saleable.
Third, NCLB created the trappings of a national accountability system. Variation around school ratings was okay when states also set the penalties (if any) for schools that didn't make the grade. But now every state operates under a federal mandate to offer "public school choice" and "supplemental services" (tutoring) to children stuck in "failing schools," and under further mandates to overhaul change-resistant schools. They're told in which year to intervene in which way. This cascade of sanctions is the same everywhere in America. The man in the street surely believes that it's a uniform accountability system. Yet we see that it's not. All those sanctions and interventions, uniform as they are, are triggered by AYP systems that couldn't be more different. At best, this is a disconnect. At worst, it's chaos.
What to do? Some imply that NCLB might be "repealed." Not likely. Another option would be to nationalize and standardize everything. Perhaps that's not as unthinkable as it might once have been, given the astounding growth of federal authority these past few months. We could move toward a national school system. Arne Duncan would determine each year which of America's 100,000 public schools makes the grade and what to do about those that don't.
But that isn't what we'd recommend. That approach would push Uncle Sam ever deeper into the impossible task of running schools and turning around those that fail.
We picture a very different approach to NCLB 2.0. Uncle Sam should create incentives for states to sign on to common standards (such as those now under development by the National Governors Association and other groups) and tests. Ensure that the tests are rigorous and comprehensive. Publish their results for every school in the country, with data sliced every which way--by race, income, disability status, progress over time, etc. And then stop.
That's right, stop. That's the end of Uncle Sam's role under NCLB 2.0.
Return to a regimen where each state then decides what to deduce from those test data and what to do about schools whose results don't satisfy the state's definition of success. Civil rights groups and others (e.g. Education Trust) that don't like the states' judgments can create their own school ratings, using the same uniform national data, accessible and transparent to all. So could private organizations such as GreatSchools.net. We could reopen the debate about what it means to be a good or bad school. And then it would be up to states and communities to do something about schools that aren't making the grade.
Such "Reform Realism" foregoes the utopian rhetoric of the NCLB era. It admits that Uncle Sam cannot actually ensure that every American child will get a world-class education. But what this strategy would do is ensure greater transparency around student results--something our new study shows is hard to come by--based on assessments that are rigorous, credible and uniform throughout the land. And it would reinforce the idea that states remain responsible for K-12 education and must make decisions that their own citizens will accept. Best of all, it would end the gamesmanship that has characterized the federal-state relationship since Congress passed NCLB.
"Study: Academic standards vary across states," by Libby Quaid, Associated Press, February 19, 2009
Email to a friend | Comments (2)
Recommended Reading
A dandy from Randi
Gadfly tends to give AFT President Randi Weingarten a (deserved) hard time, so when she does something praiseworthy his antennae perk up. So it was this week, when she took a strong stand for national standards in the Washington Post. "[E]very child attending U.S. public schools should be taught to high standards, regardless of where he or she lives," she opined. We agree. But how to get from here to there? An idea: Start with the big cities. Let them secede from their respective state accountability systems and band together under a national umbrella. Big cities already tend to be hot spots for reform and many of their superintendents already favor national standards. Furthermore, Randi's AFT tends to be the union in their systems. With all these players on board, these metropolises could pave the way. So, Randi, what say you to that?
"The Case for National Standards," by Randi Weingarten, Washington Post, February 16, 2009
Email to a friend | Comments (0)
Aversion to conversion
Just last week, we learned that Michael "Noah" Bloomberg would pack his ark with four charter-converted Brooklyn Catholic schools. Unfortunately, though not unexpectedly, the seas for this journey are already proving stormy. "There are real concerns about whether this is parochial school education by another name," yelps Donna Lieberman of NY's Civil Liberties Union. Concerns over creaming, lingering religious influence, and curricular issues (specifically teaching sex ed, which is banned in Catholic schools but required by the city in public ones) abound. Even more problematic is a pesky state law banning the conversion of private schools (including those of parochial persuasion) to charter schools. Mayor Bloomberg will need to take his case to Albany to see the law changed. If we could cease the hand-wringing for one moment, we'd encourage New Yorkers to look south to D.C., which fretted its way through many of these problems when it converted 7 Catholic schools to charter last year. The legal challenges are certainly real, but let's give Bloomberg a chance to right the ship and take his case--and D.C.'s precedent--upstate before drowning the plan altogether.
"Hurdles for a Plan to Turn Catholic Classrooms Into Charter Schools," by Javier C. Hernandez, New York Times, February 15, 2009
Email to a friend | Comments (0)
But did he score proficient?
At last, veritable proof that test prep pays real-life dividends. Seventeen-year old Geoffrey Stanford approached the Kansas state test just like his teachers told him to: "Every sentence. Every word. Slow down." Unfortunately, the test-makers would do well to heed such advice, for young Geoffrey's fastidiousness uncovered a gaffe on their part: "emission" (of greenhouse gases) was spelled "omission." His reward: 100 Facebook friend requests, emails and texts from strangers, and a cable news appearance. Has this young American everyman--linebacker, IB student, aspiring mechanical engineer--become the Joe the Plumber of the assessment world? Time shall tell. As to the state's error, DOE spokeswoman Karla Denny explained: "I think it's one of those things where the people writing the test were so close to it, they probably just read over it." Gadzooks, does the phrase "a fresh pair of eyes" mean nothing on the prairie?
"Error on state test slips past everyone--except East High student," by Suzanne Perez Tobias, The Wichita Eagle, February 12, 2009
"Blizzard of buzz for student who found error on test," by Suzanne Perez Tobias, The Wichita Eagle, February 14, 2009
Email to a friend | Comments (0)
A-level tan
The Brits have taken voc ed to new lows. In an effort to encourage teens to sign up for courses more suited to their abilities, education officials have expanded the number of courses that count toward "league tables," i.e. how schools are evaluated and compared. But it's not just shop and home ec that made it onto the expanded list; now pottery, cake decorating, flower arranging, and fake tanning make the cut, too. A diploma in "tanning treatments," for which students are taught how to operate sun beds and apply a fake tan without streaks, is now worth 45 points--the same as an "A" in an advanced math course. That's like creating Advanced Placement basket weaving and counting the scads of 5s scored on its exam towards the school's overall performance. But more than inflated school attainment measures is the worry that students will neglect traditional qualifications--core subjects--in favor of these easy "A" classes. Luckily, exam watchdog Ofqual has brought the matter to light. Enjoy the fake rays while they last, young Brits; tomorrow, perhaps, you'll again have to head to the Costa del Sol to obtain that grade-A tan.
"Now tanning courses are 'equal' to maths A-level," by Laura Clark, Daily Mail, February 17, 2009
"Courses in tanning 'worth the same as A-level maths' in school league tables," Telegraph.co.uk, February 17, 2009
Email to a friend | Comments (0)
Flypaper's Finest (The best from Flypaper, Fordham's blog)
Dear Saint Nick
Mike Petrilli
I've been running a bit behind all week...but this weekend's New York Times column by Nicholas Kristof deserves a comment. It reminded me of that old story of a homeless man who sat all day, every day on a wooden box and begged strangers for money. One day someone asked him if he'd ever thought to open the box and look inside. "No, why would I do that?" he responded. But later he decided to take a peek. And sure enough, the box was filled with gold...Read it here.
Email to a friend | Comments (0)
Strickland?s throw away response
Terry Ryan
In the news business, reporters have a saying for a boiler plate quote an editor can remove to tighten a story. It's "throw-away" and that's exactly what the governor's response to the Fordham/Paul Hill study deserves. Strickland's spokeswoman talked of the governor's plan having components that have been shown to help students succeed. We should hope so. But, again, there's no applicable evidence that they will for all children across an entire school district, let alone across an entire state...Read it here.
Email to a friend | Comments (0)
The Education Gadfly Show Podcast
Sweet chili Petrilli
This week, Mike and Rick discuss Fordham and Northwest Evaluation Association's latest report, The Accountability Illusion (after John Cronin, one of the authors and studio guest, gives us a Research Half Minute), what to make of Arne's $650 million, and Milwaukee's flirtation with mayoral control. Then Amber tell us about a new Mathematica and IES report that's not really about alternative certification, and Rate that Reform frowns on facial ornaments in Central Florida. Click here to listen through our website and peruse past editions. To download the show as an mp3 to your computer, click here (no iPod required--this link will play through any music software on your computer, including Windows Media Player or RealPlayer).
Email to a friend | Comments (0)
Short Reviews
Closing the Expectations Gap: Fourth Annual 50-State Progress Report on the Alignment of High School Policies with the Demands of College and Careers
Achieve, Inc.
American Diploma Project Network
February 2009
While many in the education reform community focus on closing the achievement gap, this compendium marks Achieve's fourth annual summary of states' efforts to close the expectations gap--the gulf between high school exit requirements, and colleges' and employers' entrance expectations. This report gauges whether states possess five main things: standards, graduation requirements, assessments, P-20 longitudinal data systems, and accountability measures. Since the report is annual, we can look at previous editions (2008, 2007, and 2006) to judge actual progress. For instance, in 2008, 13 states anticipated aligning their standards with college readiness requirements by 2009--yet only 4 actually did so. Still, since the 2005 National Education Summit on High Schools there has been a sea change in states aligning their K-12 academic expectations with those of colleges and employers. Four years ago, just 2 states had college- and career-ready graduation requirements; today, 20 (plus D.C.) have them. Three years ago, only three states had P-20 longitudinal data systems; now 12 do. The report also has a nifty state-by-state chart detailing in which categories states excelled and which they fell short. Texas, for example, has the most effective policies in place, while Georgia, Tennessee, and Washington have taken significant steps in that direction since 2006. To see for yourself, click here.
Email to a friend | Comments (0)
Impact of For-Profit and Nonprofit Management on Student Achievement:
Paul E. Peterson and Matthew M. Chingos
John F. Kennedy School of Government at Harvard University
Spring 2009
When the RAND Corporation released its lukewarm evaluation of Philadelphia's school management experiment in February 2007, Paul Peterson responded with his own study. Now two years later, Peterson is joined by Matthew Chingos in a report that builds on those initial findings and compares student achievement in reading and math at for-profit, nonprofit, and district-managed schools (whose achievement remains below the district median) from 2001-2008. The new study yields similar findings: Philadelphia's for-profit providers are beating the competition by miles when it comes to student achievement. (For those who may not recall the background, seven years ago, the Philadelphia School Reform Commission (SRC) arranged for 30 of the city's lowest performing schools to be taken over by for-profits and 16 by nonprofits.) RAND found that only district-managed schools made significant progress. Peterson and Chingos say otherwise. Accordingly to their analysis, not only do students learn "substantially more in reading and math if they attended a school under for-profit rather than one under nonprofit management," but they learned almost 60 percent more each year of the six years studied in a for-profit school than one under district management. The district, however, has been backing away from the whole outsourcing approach. Perhaps they should reconsider? You can find the Peterson-Chingos study here.
Email to a friend | Comments (0)
An Evaluation of Teachers Trained Through Different Routes to Certification: Final Report
Jill Constantine, Daniel Player, Tim Silva, Kristin Hallgren, Mary Grider, and John Deke
Mathematica Policy Research
Institute of Education Sciences
February 2009
What are the effects of different routes to teacher certification on student achievement? And what aspects of certification programs (e.g. amount and type of coursework) are associated with teacher effectiveness? These are the key questions posed in this study. Looking at 2,600 students in 20 districts and 63 schools across 7 states, what makes this analysis different is its strong research design: students were randomly assigned within each school to either the class of an alternatively certified teacher (AC) or a traditionally certified teacher (TC). Interestingly, the researchers also excluded the most selective AC providers like Teach For America since the vast majority of AC and TC programs aren't selective and don't produce most of our teachers anyway. It found that there were no statistically significant differences in student performance between AC and TC teachers (all were novices) nor was there evidence that the content or amount of coursework is correlated with teacher effectiveness. Very important, though the definitions and terminology make it a bit confusing. TC teachers were simply defined as those who had completed their training while AC teachers were those who had not yet completed their training. So the majority of the AC providers--like the TC providers--were education schools and not truly "alternative" as it's commonly understood. Thus, this isn't exactly a study of AC vs. TC so much as a comparison of the type and amount of teacher training and whether it makes a difference. But the bottom line is clear: it doesn't. In other words, there's no evidence here that more coursework, or even coursework that deals with specific topics like pedagogy or includes fieldwork, strengthens teacher effectiveness. And that's very important to know. You can find it here.
Email to a friend | Comments (0)
From Our Readers
?Hurricane Wagner? is no hurricane
To the editor:
I continue to be both surprised and disappointed by the response ("Hurricane Wagner," February 5, 2009) by some conservatives to my book, The Global Achievement Gap, and their claims that I don't believe in testing, don't value academic content knowledge, and lack skills as a researcher.
Anyone who has read the book knows that I am a strong advocate for accountability. I spend most of one chapter analyzing the difference between good and bad tests. My critique of NCLB is precisely that the standards for accountability, which rely mainly on multiple choice factual recall tests, are too low and are putting our country at a serious competitive disadvantage, compared to many countries that use open-ended, constructed response assessments to test critical thinking skills. I advocate for a national writing test and use of the Collegiate Learning Assessment and PISA tests for accountability purposes. And I spend an entire chapter discussing the importance of holding schools of education and teachers to a much higher standard of demonstrated excellence for certification and re-certification and suggest ways of phasing out the tenure system. Finally, I describe two public charter schools that succeed in teaching their predominant minority students both rigorous academic content and 21st century skills, while holding themselves to the standard of graduating 100 percent of their students, and sending nearly all to four-year colleges.
As to the charge that I don't care about academic content, interested readers might want to peruse pp. 261-263 of my book where I make very clear my views on the importance of academic content knowledge and cultural literacy. Besides, as anyone who has spent time in classrooms knows (I have 16 years of teaching experience), you can't teach critical thinking without strong academic content.
Finally, while my skills as a qualitative researcher may be subject to debate, one cannot so easily dismiss the words of scores of senior executives from some of the country's leading companies whom I interviewed--as well as nearly a dozen studies that I cite--all of which substantiate my point that we are not graduating sufficient numbers of students with the skills needed for careers, college, and citizenship in the 21st century.
We need to move beyond polarizing, and often polemical, debates and turn our attention to discussions of what to do about this growing crisis that threatens the future of both our students and our nation.
Tony Wagner
Co-Director
Change Leadership Group
Harvard Graduate School of Education
Email to a friend | Comments (0)
No hurricane? It?s CAT 5
To the editor:
Tony Wagner responds ("'Hurricane Wagner' is no hurricane," February 19, 2009) to critics like myself by saying he really is "a strong advocate for accountability" and fully understands "the importance of academic content knowledge and cultural literacy."
But his new vision of accountability involves replacing useful and measurable knowledge with ill-defined and un-measurable skills. To say that you want to hold schools accountable for teaching students creativity and collaboration means not being able to hold them accountable at all.
Nor is his embrace of "strong academic content" convincing. It is true that on p. 261 of his book Wagner makes a nod toward "core academic subject knowledge." But a more telling passage about Wagner's thoughts on academic content can be found on p. 92. There he doubts the need to require algebra for college-bound high school students. He writes: "So-called advanced math is perhaps the clearest example of the mismatch between what is taught and tested in high school versus what's needed for college and in life. It turns out that knowledge of algebra is required to pass state tests... But why is that?"
To support his case for dropping algebra requirements he cites a "survey" of MIT graduates in which "the overwhelming majority reported using nothing more than arithmetic, statistics, and probability." That survey (see p. 9-10 in particular) consists of an interview with 17 graduates. Twelve of them explicitly mention using algebra, linear algebra, or calculus in their adult lives; another four mention using statistics or probability (which would seem to require knowledge of algebra). Even those with simple math skills could see that this study, such as it is, supports the opposite of Wagner's claims. In fact, this passage reveals that Wagner actually would like to reduce academic content, specifically in algebra.
All of this "21st century skills" talk appeals to our desire to prepare children for the future, but producing critical thinking, creativity, etc. takes more than just slogans. We need evidence on methods of teacher preparation, curriculum, and pedagogy that actually get us to those goals. Unfortunately, Wagner fails to provide that.
Jay P. Greene
Endowed Chair and Head of the Department of Education Reform
University of Arkansas
Email to a friend | Comments (1)
Announcements
Understanding the illusion of accountability
Don't forget. Next Monday, February 23, from 4:30 to 5:45 pm we'll be hosting a "team of rivals"--i.e., four Fordham board members--to discuss our new report, The Accountability Illusion. The topic of discussion, specifically, is whether NCLB damaged school accountability and if President Obama can mend it. While the event is currently sold out, you can be added to the waiting list by emailing Christina Hentges at rsvp@edexcellence.net!
Email to a friend | Comments (0)
The 21st century skills debate continues
Please join Common Core in welcoming Diane Ravitch (Research Professor of Education, New York University), E.D. Hirsch, Jr. (Founder and Chairman, Core Knowledge Foundation), Daniel Willingham (Professor of Psychology, University of Virginia), and Ken Kay (President, Partnership for 21st Century Skills) as they discuss "What is the Proper Role of Skills in the Curriculum? A critique of the idea of 21st century skills." It's all happening on February 24 from 1:30 to 3:00 pm. RSVP to info@commoncore.org.
Email to a friend | Comments (0)
Job opening
Are you or someone you know trying to launch a career into policy? Education Sector may be looking for you to be their new Research Assistant! You'd get to work with some of Ed Sector's great minds, while conducting research and analysis on leading education issues. You can find more and how to apply here.
Email to a friend | Comments (0)
About Us
The Education Gadfly is published weekly (ordinarily on Thursdays), with occasional breaks, by the Thomas B. Fordham Institute. Regular contributors include Jack Byers, Amy Fagan, Daniela Fairchild, Chester E. Finn, Jr., Mickey Muldoon, Jamie Davies O'Leary, Eric Osberg, Stafford Palmieri, Emmy Partin, Michael J. Petrilli, Laura Elizabeth Pohl, Terry Ryan, Janie Scull, and Amber Winkler. Have something to say? Email us at thegadfly@edexcellence.net. If you would like to subscribe, you may either email thegadfly@edexcellence.net with "subscribe gadfly" in the text of the message or sign up online here.
The Thomas B. Fordham Institute is a nonprofit organization that conducts research, issues publications, and directs action projects in elementary/secondary education reform at the national level and in Ohio, with a special emphasis on our hometown of Dayton. (For Ohio news, check out our Ohio Education Gadfly, published bi-weekly, ordinarily on Wednesdays.) The Institute is neither connected with nor sponsored by Fordham University.