On Reading First, read the report first
The media gleefully reported the news that a big interim Reading First study from the U.S. Department of Education’s Institute for Educational Sciences (IES) found the program to have no impact on reading comprehension.
And it’s hard to blame the media, for three reasons. First, it loves to pile on the increasingly-unpopular Bush administration. (Contemplate this AP headline: “Bush administration’s reading program hasn’t helped.”) Second, IES head Russ Whitehurst–who has earned a great deal of respect and credibility for moving the federal research and evaluation function toward a new level of rigor and professionalism–stands firmly behind the report. And third, Secretary of Education Margaret Spellings’s press office totally bungled the response, coming up with nothing better than the “popularity” of the program. (This is hardly the first time Spellings dropped the Reading First ball.) See this, from Amanda Farris, deputy assistant secretary, and printed in the AP story:
Secretary Spellings has traveled to 20 states since January. One of the consistent messages she hears from educators, principals and state administrators is about the effectiveness of the Reading First program in their schools and their disappointment with Congress for slashing Reading First funds
Here’s what Spellings’s team should have said:
This study provides important insights into the Reading First program, but readers should be cautioned that it’s not nationally representative. Because IES launched the study after the program was up and running, the evaluators had to settle for a very imperfect design. The schools selected for study might have similar demographics to Reading First schools in general, but they were different in important ways.
First, none of the states that won the first Reading First grants could participate in the study because their programs got started in advance of the evaluation. These states were the ones most enthusiastic about the program–and most prepared to implement it well. It’s quite likely that Reading First schools in these states are having a major impact.
Second, the schools selected for study were the ones that just barely won grants under the program, which were compared to schools that just barely missed funding. (Schools are ranked according to various criteria, such as poverty, need, etc. Let’s say there was enough money in a given district to fund 10 schools; then the study compared the 10th-ranked school, which got money under the program, to the 11th-ranked school, which did not.) But here’s the rub: the schools where you would expect the greatest impacts from Reading First are the poorest ones, enrolling students who are further behind in reading–schools that would have been ranked at the top of the priority list. Simply put, these schools weren’t included in the study.
The bottom line is that evaluators looked for schools that met their study design conditions, not schools that were nationally representative of the program. So we can’t say anything definitive about the effectiveness of Reading First–all we know is about the effectiveness (or lack thereof) of a handful of Reading First schools.
Yes, explaining this stuff to the media is difficult. But Spellings and her team should have tried.
Related posts:
You can skip to the end and leave a response. Pinging is currently not allowed.
May 2nd, 2008 at 1:21 pm
Mike is correct in saying that the schools examined were not representative of all Reading First schools. But then again the first adopters or schools rated as being most in need of the program are also not representative. If the program only works for the most enthusiastic or those who most desperately need it, then it would have limited usefulness. What’s more, we don’t know that Reading First in fact helped the first adopters or those most in need. Mike is just speculating that if it did help those schools, this IES study would have missed it.
The thing that strikes me most about this study is that the time spent on phonics increased only marginally when schools adopted Reading First (see p. 44, where the increase is described as an extra 2-5 minutes per day). Perhaps the increase was so small because the implementation was poor, which would be consistent with Mike’s explanation. Or perhaps it was because Reading First is not a well-designed phonics program — so this would be an evaluation of Reading First, not the concept of phonics. Or perhaps the increase in time spent on phonics was small because the emphasis on phonics has already become pretty wide-spread, even in non-Reading First schools. If this last option is the case, then whetever benefit we can get from shifting to more phonics has already largely been achieved.
May 7th, 2008 at 3:17 pm
Mike Petrilli was correct in his concern that the delay in designing and implementing the evaluation made it impossible for the states that won the first grants to participate in the study given that there programs had started before the evaluation was initiated. One could hypothesize that these states began their Reading First implementations with solid implementation plans in place. The delay in designing and implementing the impact evaluation intended by Congress is not a trivial matter and goes far beyond the inability include schools in the sample who had already implemented programs by the time the evaluation started. This is the case for several reasons.
First, Bob Sweet and I in drafting the legislation felt that a comprehensive and rigorous evaluation of the Reading First program was essential. In fact, we felt that it was imperative that the evaluation must have the necessary scope, transparency, and funding to provide the fine grained details essential for program improvement and to also establish a model for future federal impact studies. For this reason, we crafted 10 primary analyses and measurements that an independent evaluator was required to carry out (again see Section 1205 of the law). Specifically the law required:
1) An analysis of the relationship between each of the essential components of reading instruction and overall reading proficiency.
(2) An analysis of whether assessment tools used by State educational agencies and local educational agencies measure the essential components of reading.
(3) An analysis of how State reading standards correlate with the essential components of reading instruction.
(4) An analysis of whether the receipt of a targeted assistance grant under section 1204 results in an increase in the number of children who read proficiently.
(5) A measurement of the extent to which specific instructional materials improve reading proficiency.
(6) A measurement of the extent to which specific screening, diagnostic, and classroom-based instructional reading assessments assist teachers in identifying specific reading deficiencies.
(7) A measurement of the extent to which professional development programs implemented by State educational agencies using funds received under this subpart improve reading instruction.
(8) A measurement of how well students preparing to enter the teaching profession are prepared to teach the essential components of reading instruction.
(9) An analysis of changes in students’ interest in reading and time spent reading outside of school
(10) Any other analysis or measurement pertinent to this subpart that is determined to be appropriate by the Secretary.
Because these are very complex analyses requiring research designs and methods (including sampling strategies) appropriate addressing each evaluation target, the law provided $25 million dollars PER YEAR over a six year period (total = $150 million) to ensure that the evaluation research tasks could be accomplished. The amount of funds set aside for the external evaluation was arrived at following a survey of evaluation researchers who were asked to identify the cost of the most rigorous evaluation possible. Twenty-five million dollars per year was the arrived at figure.
Second, the delay led to an inability to acquire the data essential to addressing each of the required analyses and measurements required in the legislation. As it stands now, it is difficult, if not impossible to identify which reading components were most highly related to comprehension outcomes, which instructional programs and materials had the most significant impact on every reading component including comprehension, whether students engaged in more outside reading as a function of the reading instruction they were receiving, whether professional development activities and content were significantly related to reading outcomes in both Reading First and non-Reading First schools, and whether critical implementation factors accounted for variation in reading outcomes. Had the evaluation been ready to be implemented in the early stages of Reading First the amount of critical information that we would have learned would have increased dramatically.
Third, unless the final Impact Study report addresses the required analyses and measurements articulated in the legislation and listed above, it is not the study that the Congress intended. A significant amount of information will have been lost. More importantly, it will not be possible to answer a fundamental question: For which students are which instructional materials (programs) most beneficial under well defined conditions (professional development, implementation fidelity, and so on).
It is unclear why the evaluation was delayed or not carried out to meet the requirements in the legislation. The legislation, with the evaluation requirements included in the law, was passed prior to the time the IES director came on board. It could be that the critical need to design and implement the evaluation rapidly was lost between the cracks somewhere, but there is documented correspondence from the Education and Workforce Committee inquiring about delays. Whatever the reason(s), the results from the interim evaluation are very difficult to interpret, particularly if one is trying to inform improvements in both the policy and the program. I have addressed a number of concerns about the interim evaluation report which can be found at:
http://www.EdNews.org
Interview with Reid Lyon: Reading First is the largest concerted reading intervention program in the history of the civilized world