To further illustrate the point that contamination may have occurred among Reading First and presumably “non” Reading First schools, a point I made in my piece in today’s Gadfly, Connie Choate, the director of Arkansas Reading First, writes:

I believe the design of the Impact Study is flawed.  The study compared funded Reading First schools with non-funded RF schools within the same district.  However in their RF proposals districts were required to include a plan for spreading the RF methodology to non-funded schools.  States were also required to do the same.  For example, all teachers across the state were invited to participate in ELLA, Effective Literacy, Summer Reading Camp, and several other professional development opportunities that are part of Reading First.  We aligned all of this professional development to SBRR.  So, even non-funded schools have benefited from RF.  One example is the revision of the State English Language Arts Frameworks. The knowledge gained from the National Reading Panel Report and Reading First enabled the state to revise the English Language Arts Framework to align with SBRR.  All professional development offered by the state is now aligned to SBRR.  This should align curriculum and instruction in all schools to SBRR, not just our RF funded schools.  We have created many materials in Reading First and have made them available to all schools. 

Ms. Choate got me thinking that it would be a good idea to take a look at the feds’ application for state RF grants. And sure enough, what she says rings true. Consider this from page 1:

Each SEA may reserve up to 20 percent of the Reading First funds it receives for State use. These funds will assist States in building and maintaining statewide capacity to effectively teach all children to read by third grade. States may expend up to 65 percent of these reserved funds for activities related to professional development... This unprecedented and significant funding will provide States with the resources and opportunity to extend this reading initiative and to improve reading instruction beyond the specific schools and districts that receive Reading First subgrants (emphasis added).

And should there be any confusion, page five includes the selection criteria for awarding grants. Potential state grantees, in a section called the State Professional Development Plan, are to answer this question: “How will teachers statewide receive professional development in the essential components of reading instruction, using scientifically based instructional strategies, programs and materials, and using screening, diagnostic, and classroom based instructional assessments?”

Again, it’s a great idea to spread the instructional reading wealth among state schools, but it sure makes it all the more difficult to assess what is really happening in this evaluation, which sought to draw a line in the sand between treatment and comparison schools.

Amber and Christina discuss the good and bad of the Reading First interim evaluation report:

Ohio AG Marc Dann isn’t the only one coming in for a beating. Take a look at this analysis of the recent Reading First interim evaluation study from Dr. James Salzman, the co-director of the Ohio Reading First Center.

To paraphrase Mark Twain: There are lies, damned lies, and the latest Reading First report. The report is methodologically flawed, statistically glamorous, and ultimately meaningless in terms of its conclusions. It’s caused the usual sharks to roil the waters as if chum were being served. And in the end, it says nothing about the positive impact of Reading First in Ohio.

Makes Fordham’s critique of the evaluation and defense of the program seem dispassionate and reserved. The key Ohio points:

- Students in Ohio have gained more than a year’s reading achievement for each year that they are in the program....If students stay within the program, they are able to catch up to benchmark scores in fluency, even though they start significantly behind.
- Students have closed the gap on state performance on the third grade Ohio Achievement Test (OAT) over the past four years.
- Teachers have helped students close the achievement gap for students of color.
- Equally importantly, Westat’s (2008) independent evaluation of Reading First Ohio has documented that the more time that students spend in Reading First schools the more they outperformed their peers in comparison schools across the state.

A message to my friend Russ Whitehurst (Institute for Educational Sciences director): Whenever you’d like to post a response to these critiques, this blog’s all yours.

Reid Lyon, former Reading Czar and one of the creators of Reading First, posted a comment about Shep Barbash’s Education Next article that’s so crucial to the current debate that it’s worth excerpting at length:

The recent Reading First Impact Study interim report did some thing s correctly (employed a strong design for the questions they asked), but appeared to miss some very important confounds, leading me to have difficulties interpreting the results. First, the evaluation did not address all of the evaluation targets established in the law, thus narrowing the scope and comprehensiveness of the evaluation Congress intended. Second, and most importantly, the sample of states selected for inclusion in the study was not sufficient to test a number of variables that are critical to interpreting the data. As hard as I try, I cannot see how the sample would be considered representative. Third, the evaluation examined the effect of resources (Reading First funding) on a single measure of reading comprehension. As Steve Raudenbush has argued convincingly, an evaluation study comparing a group that received the resources versus another group that did not answers very little about the programs actual effectiveness or the ability of a study to inform improvements in the program or guide policy.

There are many factors at the implementation and instructional level that have to be examined and studied to refine any interpretation of the main effect of no significant difference. As Mr. Stearns probably knows, many school districts that implemented Reading First in some schools implemented the same programs in non-Reading First schools. Professional development activities funded through Reading First were available to all schools in a district, not just Reading First schools. Most of the states that Mr. Barbash highlighted in his article where not included in the sample drawn for the impact study. Approximately 60 percent of Reading First and non-Reading First schools were implementing the same programs by the third year of implementation according to Tim Shanahan. There was a significant degree of contamination from the “bleeding” of programs and resources across Reading First and non-Reading First schools. Believe me, if the final impact study report clarifies all of these issues, that would be very helpful.

According to our own Checker Finn, just yesterday the Institute for Educational Sciences director Russ Whitehurst admitted in front of 200 people that the “bleeding” of Reading First methods to schools in the control group is a probable explanation for the findings of his evaluation. In other words, non Reading First schools were using Reading First’s (proven and effective) methods too. That would have been a nice point to get across to the press before they wrote stories declaring the initiative a failure.

Let it be clear: even the government’s chief education researcher is saying that the findings of his evaluation don’t mean that Reading First is “ineffective” and should be scrapped. Congress, are you listening?

While Americans feel no particular love for the U.S. Department of Education (see this graphic from Sunday’s New York Times Magazine), I have found that, in education circles at least, particular scorn is heaped upon state departments of education and their civil service employees. Colonized (in Paul Hill’s term) from federal programs above, and distant from the real action of schools and districts below, they are the consummate middle-men (and women) of America’s education system. Conventional wisdom says they are capable of little more than pushing paper: performing audits, writing regulations, and filing reports.

What sweet relief it is, then, to read Shepard Barbash’s Education Next piece about the implementation of the Reading First program, and the heroic role played by state departments of education.

The most enduring achievement of Reading First may be that it has nurtured a group of state leaders who have developed deep expertise in the science of reading instruction and have been able to get steadily better at helping the districts teach more children how to read. In states where Reading First is working, districts look not to their long-standing networks of consultants and colleges for expertise, but to their state administrators. This is a bureaucratic revolution.

Imagine that: state bureaucrats turned instructional leaders. Regardless of what you read about the program’s effectiveness (and if you must read something about that, read this or this), its implementation marks a milestone in the annals of federal-state relations. It’s a prescriptive, top-down, micro-managy program that states and districts love. Wonders never cease.

Reid Lyon, the Godfather of Scientifically Based Reading Instruction (so says Eduflack), provides a ton of important insights into RF’s interim evaluation study in this EdNews.org interview.

The media gleefully reported the news that a big interim Reading First study from the U.S. Department of Education’s Institute for Educational Sciences (IES) found the program to have no impact on reading comprehension.

And it’s hard to blame the media, for three reasons. First, it loves to pile on the increasingly-unpopular Bush administration. (Contemplate this AP headline: “Bush administration’s reading program hasn’t helped.”) Second, IES head Russ Whitehurst—who has earned a great deal of respect and credibility for moving the federal research and evaluation function toward a new level of rigor and professionalism—stands firmly behind the report. And third, Secretary of Education Margaret Spellings’s press office totally bungled the response, coming up with nothing better than the “popularity” of the program. (This is hardly the first time Spellings dropped the Reading First ball.) See this, from Amanda Farris, deputy assistant secretary, and printed in the AP story:

Secretary Spellings has traveled to 20 states since January. One of the consistent messages she hears from educators, principals and state administrators is about the effectiveness of the Reading First program in their schools and their disappointment with Congress for slashing Reading First funds

Here’s what Spellings’s team should have said:

This study provides important insights into the Reading First program, but readers should be cautioned that it’s not nationally representative. Because IES launched the study after the program was up and running, the evaluators had to settle for a very imperfect design. The schools selected for study might have similar demographics to Reading First schools in general, but they were different in important ways.

First, none of the states that won the first Reading First grants could participate in the study because their programs got started in advance of the evaluation. These states were the ones most enthusiastic about the program—and most prepared to implement it well. It’s quite likely that Reading First schools in these states are having a major impact.

Second, the schools selected for study were the ones that just barely won grants under the program, which were compared to schools that just barely missed funding. (Schools are ranked according to various criteria, such as poverty, need, etc. Let’s say there was enough money in a given district to fund 10 schools; then the study compared the 10^th-ranked school, which got money under the program, to the 11^th-ranked school, which did not.) But here’s the rub: the schools where you would expect the greatest impacts from Reading First are the poorest ones, enrolling students who are further behind in reading—schools that would have been ranked at the top of the priority list. Simply put, these schools weren’t included in the study.

The bottom line is that evaluators looked for schools that met their study design conditions, not schools that were nationally representative of the program. So we can’t say anything definitive about the effectiveness of Reading First—all we know is about the effectiveness (or lack thereof) of a handful of Reading First schools.

Yes, explaining this stuff to the media is difficult. But Spellings and her team should have tried.

Regarding the just-released study of Reading First’s effectiveness, Mike tells USA Today that

...the study was poorly designed and “certainly not the last word on Reading First’s effectiveness.” For one thing, he says, researchers looked at “lackluster” Reading First schools that just barely qualified for grants, comparing them to schools that just barely missed getting grants.

George Will explains that Clinton strategist Mark Penn was caught doing something sensible, surreptitiously. (In Penn’s case, it was promoting free trade.)

The same could be said about Chris Doherty, who was also caught doing something sensible, surreptisiously. (In Doherty’s case, it was promoting scientifically-based reading instruction.)

In today’s Washington, both faced the same fate.

Who’s ready for a new approach to politics?

Over at the “ELL Advocates” blog, whole language apologist Stephen Krashen makes a lame attempt to poke holes in Sol Stern’s recent Fordham report, Too Good to Last: The True Story of Reading First. In particular, he takes issue with Stern’s claim that the Golden State’s adoption of whole language reading in 1987 led to California’s disastrous, bottom of the barrel NAEP performance in 1992. Krashen is right about one point: The ‘92 NAEP was the first to break out results state-by-state, so it’s impossible to know whether California’s scores “plummeted,” as Stern argues. But then Krashen goes on to make a fool of himself. First, he offers this stunning piece of revisionist history:

Whole language, according to (urban) legend, was introduced by the 1987 Framework committee, which I was a member of. The 1987 Framework committee never mentioned whole language. We recommended that language arts be literature-based, hardly a revolutionary idea. Phonics was never forbidden.

This is ridiculous; of course California adopted whole language reading in 1987. For the definitive history of this episode, see here. Then Krashen goes on to argue:

Of great interest, and rarely noted, is that fact that California still ranks at the bottom of the US. NAEP scores released 2007 show that California is still in the basement, in a virtual tie for last place with Mississippi and Louisiana. Dumping whole language did not improve things.

But if dumping whole language did not improve things, why have the state’s African-American fourth graders gained 19 scale-score points on the NAEP since 1992 (equivalent to about a grade and a half) while its Hispanic fourth graders have gained 15 points (more than a grade level)—even while the state received a wave of new Hispanic immigrants during this period? Something has improved things dramatically for these groups of children—and the most likely explanation is the introduction of scientifically-based reading instruction.

Someone should tell Krashen: except among fringe elements, the reading wars are over—and scientifically-based reading instruction won.

On March 10, the Thomas B. Fordham Institute demanded an inquiry into scandalous efforts by the executive and legislative branches to sabotage the Reading First program.

Margaret Spellings addressed the Reading First state directors on Thursday and complained about Congress’s “devastating” budget cut of the program. It’s about time. If she had shown even an iota of courage 18 months ago, when the so-called scandal first broke, the program might have remained in-tact. But as Sol Stern shows in painful detail, she and the rest of the Administration headed for cover instead. Such decisions have consequences, Madame Secretary, consequences that are all too real for the 4,000-odd schools likely to see their Reading First funds disappear.