We’ve just returned from the School Finance Redesign Project Panel (and were caught in the rain, no less!) Here are some final thoughts:

Great ideas... not so practical. I’ll give you two examples.

In the ideas area, I was struck by Guthrie’s thoughts on the transformation of American education. 50 years ago, he asserts, you could drop out of high school and still get a job and have a productive, comfortable life. Today, those jobs that don’t require education have either disappeared or moved overseas. As a result, we, as a nation, are facing a momentous task: educating everyone and educating them well. He proposes that we’re the first modern, democratic, industrialized nation to confront this challenge. NCLB may be an “awkward instrument” but we are venturing into new territory.

Criticizing NCLB is the new “it” thing to do (presidential election, anyone?). I can almost hear someone saying, “It’s an axe when we need a scalpel”...[cue laughter] The point is that Guthrie is largely right. Most countries don’t even attempt to educate everyone or consider that doing so is a laudable goal. Take Germany, for example. It fits all of Guthrie’s criteria (or those that I managed to scribble down during the event—there may have been more): modern, democratic, and industrialized. Yet children in Germany are tracked from age ten into college-bound, vocational, and remedial schools. There is no assumption in the German education system that every child deserves the chance to go to college. As The Economist so candidly puts it, “The cleverest go to Gymnasien, the main route to university; the ordinary are sent to Realschulen; and the dullards attend Hauptschulen, often breeding-grounds for disaffection.” (Angela Merkel, Chancellor of Germany, is attempting to reform the system, but facing incredible resistance.) “The dullards?” That’s an American lawsuit waiting to happen.

Now, I’m not saying that every student should actually go to college, but children in this country at least exist under the assumption that they could go to college if they wanted to—that they can learn if we figure out how to teach them most effectively. (Some might disagree that we teach students to believe in themselves—the teaching for social justice crowd comes to mind—but I stand by my assertion that this country is a fountain of opportunities spilling over.) We talk about NCLB as if it were the antichrist—dumbing down education, teaching to the test, ruining children’s lives left and right. But what was our education system like before NCLB? Despite its problems of structure and implementation, hasn’t NCLB taught us something, at least? Would we know where to go next if we hadn’t been where we were?

It’s great to contemplate the ideological mindsets behind our schools but when Amber asked how we would couple teacher salaries with weighted student funding, none of the panelists could answer the question. As you can see below, I noted Paul Hill’s response, but Jim Guthrie took a stab at the question first—and didn’t answer it satisfactorily at all (actually if there was an answer in his response, somebody please let me know. I did not hear one). The problem is that we really do need an answer. Why? Under most school funding schemes currently in place, dollars are allocated based on the number of teachers, administrators, and programs (or that’s how I understand it). If we have dollars following children as WSF calls for, how do we a) determine teacher salaries, b) determine teacher pay scales, and c) create collective bargaining agreements? While I appreciated and understood the “real world” aspect that would be injected into schools trying to make a bottom line, I also understand the “real world” aspect of teachers’ unions and their guaranteed objections (read: outrage) to this scheme. I merely wish that Paul Hill and company had taken on this question in their report—and moved beyond the niceties that keep this conversation, again, in the land of policy—instead of practice.

Martin West and Ludger Woessmann have published a fascinating study in the winter edition of Education Next. Its conclusion—that there is a positive correlation between the prevalence of private schools and high test scores—is something that is widely argued but not so widely proven. West and Woessmann compared PISA scores from 2003 with the number of private schools in 29 countries to find that,

a 10 percent increase in enrollment in private schools improves PISA math test scores by more than 9 percent of a standard deviation, nearly equal to a half of a year’s worth of learning. For science and reading, a 10 percent increase in private school enrollment generates an improvement of more than 5 percent of a standard deviation—more than one-fifth of a grade-level. And in educational spending, a 10 percent increase in the private school enrollment leads to a $3,209 reduction in spending per student—on average, more than 5 percent of the total education spending per student through age 15 for OECD countries.

The part, however, that caught my eye was how they controlled for the causes of private school proliferation. They recognized that a country with a high average income or commitment to education would result in more private schools. On the other hand, a school with very poor quality public schools might increase the demand for private alternatives. So, to eliminate these potential effects, they used a totally unrelated number: the Catholic population of the country in the year 1900. Perhaps our readers will disagree but I thought it was a rather innovative solution. Since the Catholic Church founded many Catholic schools during the turn of the century upswing of public education for reasons totally unrelated to current student achievement, West and Woessmann could isolate the effects of private school competition on student achievement and on per-pupil expenditure. You can read the whole study, here.

We’ve also written about Catholic schools—and their crucial role in American urban education.

Last Thursday, the Brookings Institution’s Metropolitan Policy Program unveiled a new paper by Sara Mead and Andrew Rotherham, Changing the Game: The Federal Role in Supporting 21st Century Educational Innovation. I was asked to respond to it, surely because of my role helping to create the Department of Education’s Office of Innovation and Improvement, which Mead and Rotherham want to rename the Office of Educational Entrepreneurship and Innovation. (Now that’s progress!)

As I said at the release, the paper is at once both underwhelming and incredibly audacious. First the underwhelming part: strip away the lofty rhetoric, the (annoying) “game-changing” language, and the Brookings panache, and what Mead and Rotherham are proposing is to steer federal funds to organizations they like. (This through a new program, the “Grow What Works” fund, which would allocate dollars for the “scaling up” of reform organizations such as KIPP or Teach For America.) Or, put less generously, it’s pork for their friends. At least that’s how the media and critics will depict it, I would guess. That’s what happened to those of us in the Bush Administration when we used the “Secretary’s Discretionary Fund” to support worthwhile school reform organizations. (The reaction was fierce.) I have nothing against giving TFA or KIPP dollars to help them grow, but it’s surely not a new idea, nor is it without complications. Which brings us to my second point.

Part of the proposal’s audacity is its desire for federal officials to pick winners and losers (those “entrepreneurs” that deserve federal largesse, and those that don’t) and to expect everything to just go swimmingly. I know Sara and Andy haven’t slept through the past seven years, but have they learned any of the key lessons of the No Child Left Behind era? Did they notice what happened to Reading First officials who dared to suggest that some of the reading programs available on the open market weren’t any good and didn’t deserve federal funding? Did they notice how “losers” in the process were able to push their claims through the media and the political process? If Edison gets “Grow What Works” funds and the National Heritage Academy does not, what exactly do Sara and Andy think will happen? They propose various ways to protect the new program from political interference, but I’m still skeptical.

But even more audacious is their claim that the federal government will magically be able to sweep aside the hurdles that are keeping the KIPPs of the world from growing faster: the charter school caps in so many states; the restrictive teacher certification requirements; the uneven funding for charters, etc. I’m sorry, but what gives anyone the hope that the federal government has or could have these abilities? If there’s anything we can glean from the NCLB years, it’s this: While it’s possible for the federal government to coerce states and school districts to do things they don’t want to do, it can’t force them to do those things well. And in the complex world of school reform, doing things well is an imperative.

Want another example? See what happened to NCLB’s “supplemental services” provision (a.k.a. free tutoring), which was energetically implemented by us Bushies but which failed miserably in most big cities. That’s because it required school districts to do a number of things well—informing parents, cooperating with tutoring companies, pay contractors, etc. And lots of districts decided to just go through the motions. They followed the letter of the law, but not the spirit. And there was nothing the feds could do about that.

It’s looking very likely that Barack Obama will be our next president, and I’d suspect that both Mead and Rotherham will serve in his Administration. That’s good news—they are both smart and dedicated policy wonks. But if they think that merely getting a smarter group of people working in the Department of Education will turn that agency into an effective change agent, I am guessing that they will be sorely disappointed. We don’t need another “game-changer” in education at the federal level—we’re still coping with the current game-changer, NCLB. What we need is more humility and some realistic expectations. But it looks like we’ll have to wait at least another four years for that.

*Update 10/21/08: Education Week article about the Brookings Institution report here.

Sam Dillon has a great article in today’s New York Times which illustrates the wide variation in the number of schools making “adequate yearly progress” under No Child Left Behind. He writes,

A state-by-state analysis by The New York Times found that in the 40 states reporting on their compliance so far this year, on average, 4 in 10 schools fell short of the law’s testing targets, up from about 3 in 10 last year. Few schools missed targets in states with easy exams, like Wisconsin and Mississippi, but states with tough tests had a harder time. In Hawaii, Massachusetts and New Mexico, which have stringent exams, 60 to 70 percent of schools missed testing goals. And in South Carolina, which has what may be the nation’s most rigorous tests, 83 percent of schools missed targets.

In December, Fordham and the Northwest Evaluation Association will release a new study—a follow-up to last year’s Proficiency Illusion—which will dig into this state-by-state variation and show that the tests alone are not to blame.

UCLA professor Bill Ouchi argues in today’s New York Post for giving principals autonomy—a point about which we surely agree—based on his forthcoming research that, when given control, principals can get great results by manipulating the school variable that (he finds) matters most: “Total Student Loads,” roughly described as “the number of students [teachers] must get to know each term.”

Count Stafford and Jay Mathews as skeptical that, in teaching, quantity might matter more than quality.

Economist Roland Fryer’s Educational Innovation Laboratory is off to the races, thanks to the Broad Foundation, experimenting with new ways of incentivizing kids to learn in three big cities (New York, Chicago, Washington). In D.C., the plan involves paying students in fifteen middle schools up to $1500 a year if they (a) attend, (b) behave and (c) get good grades.

I’m a longtime believer in giving young people real-world incentives to study hard and do well in school, though I’ve long supposed that means doing a better job of hinging promotion, graduation, college admission and jobs on school success. I don’t have any big problem with more immediate and kid-like rewards, either, such as taking students with perfect attendance records to a theme park at the end of the year or giving pizzas to those who read more books.

Paying them cold cash to do the right thing gives me pause, however. It’s fundamentally amoral. It creates weird and perverse incentives for pupils and teachers alike. It could get very expensive, using serious money that might otherwise go into better teachers, better textbooks, longer times, more instructional technology, etc. (Chicago has about 125,000 students in grades 5-8. At $1500 apiece, a totally successful program for them would cost $187.5 million per annum.)

Besides all that, I really want to know if it’s going to work—and I really want that question to be addressed by analysts completely independent of Roland Fryer. Is there a freestanding evaluation built into this? Will the necessary data be available to one and all to analyze? Fryer is a fine economist and presumably an honest man. But nobody should be his own evaluator. That’s an even more dubious proposition than paying kids to do what’s in their own interest in the first place.

Kudos to Jay Mathews for writing this:

When fixing schools, beware of miracle cures. Every week people send me ideas they say will change the future of education and lead all humanity to enlightenment. So, when management expert William G. Ouchi let me look at his new work on the surprising power of total student loads per teacher, or TSL, I was skeptical. 

As you should be, Jay, as you should be. But on to the meat of the article: TSL, Ouchi’s newest addition to the alphabet soup of acronyms. Yes, that’s William Ouchi of the widely read Making Schools Work, which rightly espouses the idea of weighted student funding, or (alphabet soup!) WSF. We’ve written on this topic too and we think it a darn good idea—but, like its reform-minded brethren, not a panacea. At first read, TSL sounds like an interesting concept. Ouchi does his homework, too, which makes me much more likely to read this forthcoming book (when it is published—unclear when that will be) with an open mind. 

Here’s the problem, though. Ouchi (according to Mathews, who it must be noted is the only one, it seems, who has read this elusive “chapter” of the forthcoming book he alludes to) seems to be advocating hiring teachers across the board. Teacher quality? Nah.

Could other factors, such as increasing teacher quality, explain the test score gains?  Ouchi and his researchers analyzed three years of student performance. They looked at the effect of class size, teacher experience, teacher credentials, professional development, time devoted to math and reading instruction, and a few dozen other factors. “Among these, only TSL had a noticeable effect on student performance in every district, and that effect was large,” Ouchi says. 

Hmmm. Not buying it. After introducing his column so promisingly (standing up against “miracle cures” and the like), I’m surprised Mathews doesn’t call Ouchi out for this. You can’t tell me that hiring lousy teachers just to reduce TSL is going to improve student achievement. Having 5-7 classes a day and student loads of 170 (New York City) to 225 (Los Angeles) students is stressful, sure, but you still have to teach them something while you’ve got them in the desks! More promising? Ouchi (and Mathews) see the connection between WSF and TSL: 

“One school, for example, may not need or want security guards or professional-development staff, while another may not want attendance clerks or registrars,” [Ouchi] says. Principals, unlike central office managers, know which jobs have been rendered obsolete by new technology and which jobs exist simply because they have always existed. 

Absolutely, as long as they’re hiring good teachers instead.

Today in Education Week: “NCLB Testing Said to Give ‘Illusions of Progress’”

Last October from Fordham: “No Child Left Behind’s ‘Proficiency Illusion’“

Fraud! Misleading information! A huge price tag for America! I’m not talking about the mortgage-backed securities meltdown. I’m referring to the new TOM LOVELESS ALGEBRA STUDY.

AP story here. The main point: The number of kids taking “Algebra” doubled from 1990 to 2007—but test scores for these 8th graders have actually declined. Call it the name game: the prestige of the course labels goes up, the amount of learning goes down. Just wait till reformers tackle “Calculus.”

Did you routinely win the estimate-the-weight-of-a-pumpkin contests at the state fair? Always know how to sneak on an already too crowded train? You may be stupendous at math! Or so a new study from Johns Hopkins, which as found a link between number sense—the ability for humans to estimate numbers—and math ability, concludes. Don’t run out and spend all your money on those jelly bean jar raffle tickets to practice, though, since researchers have not yet figured out if number sense can be learned.

Once in awhile, I take the time to sniff around and find an education study worth talking about in this blog. I wish I had the time to do it more often, but judging from my quick look-see this afternoon, the research terrain isn’t overflowing with milk and honey these days anyway.

First, there’s this Education Week news story about a technology study conducted by Central Connecticut State researchers. We’re told that, when college students respond to instant messages while they are reading, they take longer to read. Alrighty then. The supposed shocker of the research is that students still understand what they read... probably because they re-read. Now, I’m all for learning more about how new technologies affect learning. It’s one of the reasons I’m pretty excited that we have a new, federally-funded research center on education technology. To be sure, we need to better understand how to harness new technologies and learning forums for such media. But I’m also of the opinion that some research questions can be answered by common sense.  And whether kids take more time to read and understand while they are instant messaging falls into that category. (Granted, I didn’t read the study and cringe at not doing so, but Ed Week didn’t provide the link—which also makes me cringe.)

Onto the next study out this week. It’s a high school graduation report based on Common Core data from 2005-2006. It’s always good to keep up with these types of data since they’ve been largely ignored in the past and energy is now rightly going into making them more reliable. Anyway, analysts here use what they call an “averaged freshman graduation rate” or AFGR to calculate students receiving high school diplomas. (This is basically an estimate of the number of kids who come in as freshman and graduate four years later.) Across 48 states in 2005-2006, our AFGR is 73.4 percent (part of the reason we do poorly in the “high school graduation” event in the Education Olympics). Rates were particularly low in Alabama, Alaska,  and California, among other states. They were high in Connecticut, New Jersey, and Iowa, among others. Then there are the states that have markedly increased their graduation rates in recent years, like Hawaii, Tennessee, and Kentucky. Kudos to them.

Finally, there’s this report out from the Center on Education Policy on a similar topic—it tallies state progress on high school exit exams. We learn that, during 2007-2008, 23 states “withheld diplomas based on students’ performance on state-mandated high school exit exams.” Hmmm... that withholding diplomas part sounds like it might actually mean something—exactly what, no one can be sure. Between “alternative paths to graduation,” allowing kids to re-take exams multiple times, and questions about test quality, among other areas, we have very little handle on the true impact of these tests. Moreover, we have no national data regarding success and failure on the exams—and the state data, as usual, are all over the map. As with any accountability policy, we often like the “A” word more in theory than in reality. Accountable consequences are fine when it’s not your kid, your school, or your job on the line. If you were to ask 10 people why we should have high school exit exams, you’d likely get very different answers. One of them, if we’re truthful, is to point fingers at someone or something falling down on the job. Apparently we’re not ready to hear that. And until we get real about reporting high school exit data fully, accurately, and meaningfully, what they say will remain a mystery.

I’m always on the lookout for interesting education research, and Natascha (Fordham intern and fellow Wahoo) does a nice job helping me track down studies. She found this one carried out by researchers from our favorite university. Basically, they conducted experimental research with children ranging in age from 6 to 11 and found that the “style of information processing triggered by happiness could be a liability.” They “induced” (their word) happy or sad moods in children by playing certain types of music and video clips (unclear from the summary whether Mozart was the happy or sad music), then asked them to perform tasks which required attention to detail. Children induced to feel sad repeatedly did better on the task than those induced to feel happy. Researchers concluded:

Happiness indicates that things are going well, which leads to a global, top-down style of information processing. Sadness indicates that something is amiss, triggering detail-orientated, analytical processing. However, it is important to emphasize that existing research shows there are contexts in which a positive mood is beneficial for a child, such as when a task calls for creative thinking. But this particular research demonstrates that when attention to detail is required, it may do more harm than good.

Hmmmm....it’s been my experience that a very detailed task actually induces sadness in and of itself, so I guess I can be happy about that (uh, no I can’t).

Mike opened the door for my response to the Washington Opportunity Scholarship Program external evaluation, and I’ve just completed a fairly quick read of it. First, in the spirit of full disclosure, I’ll note that my former employer, Westat, was the prime contractor for the evaluation. Though I never personally worked with the Westat staff who conducted the evaluation, I do know their reputations for quality work. This is not the only reason, of course, that I found the evaluation to be of high-quality, but it’s worth mentioning. Disclosure aside, I have a couple takeaways from the evaluation.

First, the impact findings for the program are simply not that compelling (sorry Mike), and even the subgroup analyses—which do provide a ray of hope—are presented with important caveats. The design comprised a randomized controlled trial where eligible applicants were randomly assigned to receive or not receive the scholarship. By all accounts, the sample was drawn appropriately and is of sufficient size (n=2,308 which is, we’re told, larger than impact samples in previous, similar evaluations); furthermore, the analyses appear thoughtfully and meticulously conducted.

So, while I have few qualms with the evaluation design itself, I do think something that occurred naturally within the impact sample—namely, lots of student mobility—is worth keeping in mind. Over the course of two years in the treatment group, only 4 percent remained in the same school they were in when they applied to the program; 71 percent switched schools once, and 25 percent switched schools twice. Among the control group, 22 percent remained in the same school they were in when they applied to the program; 57 percent switched schools once; and 21 percent switched schools twice. That’s a majority of kids (even more so in the treatment group) not attending any one participating school for very long. The authors report that “both groups experienced higher rates of school mobility than the typical annual rate for urban students (22 to 28 percent).” It’s not surprising, then, to see unimpressive findings in an evaluation that covers such a short duration (2 years) and examines achievement data from students who are extremely transient (not to mention that students were tested on Saturdays!).

Second, I’m struck by the number of times that the phrase “adjustments for multiple comparisons suggest that this finding may be a false discovery” (or similar nomenclature) appears in the report. Researchers concern themselves with multiple comparisons because they are in a position of simultaneously evaluating multiple questions and hypotheses. Simply put, when you consider the results of multiple, separate statistical tests together, there is more room for error. The issue has gotten more attention of late, in part because of this recent report from IES which presents methods for dealing with the multiple comparisons problem. Like most people involved with education, I’m interested in the best research possible given the time and resources available to conduct it. Many statisticians believe that ignoring the multiplicity problem leads to misinterpretation of findings, so these researchers covered their bases.

But with all of those “false discovery” caveats in the report, I found myself harkening back to Judith Gueron’s comments in this book. Ms. Gueron (of Manpower Demonstration Research Corporation or MDRC) writes:

Finally, rigor has its drawbacks. Peter Rossi once formulated several laws about policy research, one of which was: the better the study, the smaller the likely impact. High quality policy research must continuously compete with the claims of greater success based on weaker evidence.

Ahh, so true. Sooner or later, we must come to terms with the fact that the bar we set for rigor may unintentionally and preemptively knock out of the running a program that may, in fact, make some improvement in American education.  Mind you, I’m not calling for a return to the age of education anecdote equals research. Here’s Gueron again on a lesson she learned about running successful social experiments:

You do not need dramatic results to have an impact on policy. Many people have said that the 1988 welfare reform law, the Family Support Act, was based and passed on the strength of research—and the research was about modest changes. When we have reliable results, it usually suggests that social programs (at least the relatively modest ones tested in this country) are not panaceas but that they nonetheless can make improvements. One of the lessons I draw from our experience is that modest changes have often been enough to make a program cost-effective and can also be enough to persuade policymakers to act. However, while this was true in the mid 1980’s, it was certainly not true in the mid 1990’s. In the last round of federal welfare reform, modest improvements were often cast as failures.

The question is: Will the OSP ultimately pass the “modest improvement” test? At two years—a time period that’s too short to capture impacts that may evolve over time—we don’t know. What I do know is that parents believe the OSP is making improvements, that improvement for certain groups of students may exist, and that school choice in and of itself may prove a laudable goal even without raise-the-roof achievement gains. Also, as an educational community, we’d be wise to continue the dialogue around the financial, political, methodological, and common-sensical (I think that’s a word) tradeoffs involved in rigorous research.

Kevin Carey expounds upon the reasons that research doesn’t always or even often make it to policymakers and into their policies. His suggested remedies are fine, especially the appeal for better writing. And yet, conspicuously absent from his piece is that research—at least education research—is rarely conclusive, and sometimes mere weeks pass between the publication of two different studies of the same topic that unearth about that one topic two utterly different and opposed findings.

Rarely addressed is the mutability of education research; certainly, reports can be tweaked in one way or another to reveal the data the authors desire. Furthermore, how many of such reports end with the limp, depressing words, “More research on this topic is needed”? (The practical reader wonders: “Well, why didn’t you do it, then?”) Policymakers generally have ideas about education that they’ve formed from their own experiences, listening to their constituents, or considering political ramifications. They use studies not to form their opinions but to bolster those they already harbor—and maybe, in rare instances, to develop an area in which their opinions are not yet fully formed. Who can blame them, though? Were they to predicate every decision on the conclusions of the extant research, they’d have no clarity on anything. In education, as in most policy topics, policymakers’ instincts and first principles matter—and few are the research studies that will change them.

I was reviewing a federal evaluation report that came out last week on small schools (also known as schools within schools or small learning communities). The idea is that large high schools are made impersonal, in part, by sheer magnitude; thus, efforts should be made to cut down on class sizes as to render a more individualized and personal education to students. As most folks who follow ed policy know, the Gates Foundation has done the most in recent years to bring attention (and money) to this issue. So I was interested in what the researchers at Abt Associates had found.

Turns out that reading about the key study finding (i.e., most schools are creating freshmen academies and career academies) wasn’t as interesting as another thing I noticed. And that is that most teachers received little more than three days of professional development per year related to teaching in small learning communities—these would be things like tailoring instruction to individual student needs. Talk to most any teacher and she will tell you that differentiating instruction based on student ability is one of the hardest things to do in a classroom; my former professor in graduate school, Dr. Carol Tomlinson, has written much about how to do this well. So I was struck that teachers participating in SLCs had received such paltry training in how to do what their school had presumably received a nice chunk of change to do.

It’s unfortunately typical of ed reform programs. Schools receive funding to do Reform X but little in the way of training to do it well, or in helping get teachers on board to ensure that the reform is implemented with fidelity and good faith.

I started my career teaching British, American, and world literature to high school kids. So I’m not thrilled to see the steady decline in the number of books read by middle and high school students. We’re told that last year, on average, 2nd graders read roughly 46.2 books compared to 4.5 books for 12th graders. That has me depressed. But before I cry in my beer (read: Starbucks Chai Latte Nonfat Extra Hot), I decided to download the study.

Yes, as a former program evaluator (another post-teaching vocation), I actually like to review the methodology of studies as opposed to relying upon the “bottom line” message often reported in the news media. As alluded in the Toledo news report, the study’s data are collected from a database at Renaissance Learning, a company that markets Accelerated Reader (AR)—a popular reading program in schools. Turns out, though, that the number of books students read is calculated by the number of quizzes that any particular student completes (each AR book title has an accompanying quiz). A caveat explaining such is included in the introduction to the report, which reads:

Please note: Renaissance Learning recognizes, of course, that not all book reading that happens in or outside of the classroom is captured through the Accelerated Reader software. However, it is reasonable to assume that for users of Accelerated Reader much book reading is captured in this way. AR quizzes number more than 115,000, which allows students a wide range of book selection; nearly every book found in a school, classroom, or local library has a quiz available.

They go on to explain that the study sample is one of convenience (duh), not a representative one. They say that they have records for more than 3 million students at more than 9,800 schools. That is all well and good. But let’s not make the mistake of concluding that the data in this report speak for school-age children in general. I’m left wondering about the conditions under which students completed these quizzes. Were they graded? Were there incentives tied to completing the quiz—a book-of-the-month gold star perhaps? These are critical data left out that would help us understand how we should interpret the findings.

So, then, what we should keep in mind about the study is this: The dismal number of books students are reported to have read pertain only to a large bunch of kids enrolled in schools that pay to use Accelerated Reading and then are asked (more likely required) by their teachers to complete a quiz on what they read.

Sure, I agree, kids are probably reading fewer hard copy books these days, but we need to know more about how and how much students read online. Emerging technologies are changing how academic reading is handled in schools, and innovative thinkers are reinventing teaching in exciting ways. So for now, I choose to remain optimistic about kids and reading—and especially what we’ll be doing in the future to enhance how they read.

Photo by Flickr user judybaxter.

Education Sector just released a new survey, Waiting to be Won Over, by Farkas Duffett Research—a top-notch policy research firm that’s done great work for Fordham in the past (and is working on a teacher survey of our own, due out later this year). It looks at teachers’ views about various reform ideas and includes some interesting (and generally depressing) trend data. The top-line findings are that unions are ascendant (54 percent of teachers view them as “absolutely essential” vs. 46 percent in 2003) and that merit pay (at least via test scores) has taken a bit of a beating (support is down four points to 34 percent).

Still, there are plenty of findings to hearten reformers, including strong support for “hardship pay” for individuals willing to serve in tough schools (8 in ten teachers support it) and, at least among newcomers, extra pay for shortage subjects like math and science (almost two-thirds of newbies support that).

These data are illuminating, and no doubt the survey’s authors are correct that “independent public opinion research that investigates what teachers think about various issues is a necessary contribution to the national conversation on education policy and reform.”

Still, as Rick Hess would say if he had a blog (Rick, no one reads books anymore), the views of current teachers (even new ones) shouldn’t be the last word on how tomorrow’s teachers might react to various workplace reforms. The point of merit pay, for instance, isn’t primarily to motivate teachers who are already in the classroom (who understandably gripe about being “incentivized” by the carrot of more money) but to motivate potential teachers, currently in college or another field, who might give public education a try if they viewed it as less bureaucratic and more performance-minded than most do today. And at a time when close to fifty percent of our teachers are nearing retirement age, it’s those potential teachers we should worry most about.

An article in yesterday’s Washington Post reports on Grover Whitehurst’s efforts as founding director of the Institute of Education Sciences to improve the quality and impact of education research.

The No Child Left Behind Act, in which the phrase “scientifically based research” appears 111 times, according to Whitehurst, has undoubtedly upped the demand for more and better education data. But the whole enterprise has proved too politically sensitive for Congress to be able to do it well:

Whitehurst, who in late 2002 became the founding director of the department’s Institute of Education Sciences, has discovered that his vision for the role of research sometimes conflicts with the turbulent forces of politics, policy and public opinion.

... [One] proposal called for recruiting double the number of students that Upward Bound is able to serve. Half would participate in the program, and half would become a control group. Researchers would track the progress of both groups.

Scientifically, it was sound. Politically, it was a non-starter.

Critics said it was unethical to introduce at-risk kids to Upward Bound’s opportunities if officials knew they couldn’t participate. At a February hearing on Capitol Hill, Rep. Barbara Lee (D-Calif.) called the evaluation design “discriminatory.”

After lawmakers proposed legislation to halt the study, Education Secretary Margaret Spellings agreed to scrap it.

That’s just one example of how lawmakers-turned-program evaluators have mucked things up. For a gorier picture, see Fordham’s recent report on the Reading First scandal. That program was designed to channel tax dollars to primary-reading programs based on scientific research. What happened? The pot of public money attracted a swarm of vultures who pecked and clawed each other mercilessly, bringing down the program with them.

What’s surprising is that so many people continue to believe that these embarrassments stem from a failure of political will, rather than the inherent obstacles posed by, as the Post puts it, the “turbulent forces of politics, policy and public opinion.” We always think we’ll do better next time around, when our guys or gals are in office.

But lawmakers have proven again and again (and it’s only natural, given the dynamics of representative government) that the voices of constituents and interest groups are louder than the voice of science. For another great example of this, see Michael Pollan’s wildly-popular New York Times essay on how the Senate Select Committee on Nutrition was hijacked by the meat and dairy lobbies and consequently released “scientifically-based” nutrition guidelines that have proven remarkably wrong-headed and disastrously influential on our eating habits.

Per my earlier post, here’s yet another example, from economist Steven Levitt, of statistics being incorrectly interpreted. One could unearth scads of such instances. But Levitt’s story involves medicine, and we seem to hear evermore frequently (from writers such as Jerome Groopman and Atul Gawande) that the medical field, long steeped in data, nonetheless still struggles to correctly use the stats it has. The construction of education policies atop data-based foundations is, comparatively, a new idea; ed reformers would be wise to learn from the experience of confused doctors and approach studies and reports with greater humility and skepticism.

Regarding Mike’s post, isn’t it odd that a school embraces healthy food alternatives only after a two-year research study? It reminds one of the humorous dig at think tanks: that they study reality to see if it conforms to theory. In Philadelphia’s schools, it seems, common sense has truly been vindicated. It is, in fact, correct that replacing soda and potato chips with healthful alternatives will make students healthier!

What happens, though, when this study is replicated in Memphis or Honolulu or Boise and yields no significant results? More studies, no doubt.

Here arises a problem with education reform overall: Common sense often dies at the hands of reports and statistics that obscure or even contradict it. (This occurs in lots of other fields, too. Michael Pollan, for example, makes a persuasive case that America’s national eating disorder is, in large part, a product of lousy scientific studies.) It’s counterproductive, of course, to toss out the baby with the bathwater and eschew all studies in favor of tradition, but one wonders just how enthralled by statisticians ed reformers wish to be.