Michael Martin, our Arizona colleague, continues to monitor the literature and to apply his considerable analytical talents to what he finds.  Below is his summary of the National Research Council report on their plan to evaluate the Washington, D.C. public schools.

——————————-

The National Academies of Science, America’s most prestigious organization in science, engineering and medicine, assigned their National Research Council to evaluate the District of Columbia’s public schools at the request of the Washington, D.C., city council. A prepublication version of their first preliminary report was released on March 4, 2011, through the National Academies Press.

Titled “A Plan for Evaluating the District of Columbia’s Public Schools: From Impressions to Evidence” this preliminary report laid out the parameters of how their study would be conducted. The importance of this “Plan” is that it also set out clear standards for the scientific basis of conducting that evaluation that provide a framework for considering the validity of other evaluations of public schools.

Most importantly, the report focused on the caution that should be exercised in evaluating public schools. In contrast to the many unofficial reports that crop up about public schools, the fourth paragraph of their executive summary clarified:

• First, although many U.S. cities have undertaken significant reforms to change their schools and researchers have examined what they have done, there is no established model for evaluating a district involved in reform-or, for that matter, any district.

Using test scores for evaluating schools or teachers or districts was repeatedly singled out as nefarious. In their second chapter summary of education reform efforts around the United States, they note:

• Researchers have raised questions about the inferences to be drawn from test scores, the most easily available measures of progress (see Chapter 5).

In chapter 5 they devote considerable effort starting on the first page to discredit the use of test scores for evaluating public schools. After noting:

• There is a long history of relying on student test data as a measure of the effectiveness of public education, and it is tempting to simply rely on those readily available data for judgments about student achievement and about causes and effects. However, student test scores alone provide useful but limited information about the causes of improvements or variability in student performance.

The report ends the next paragraph with “therefore, it is important to remember that the consensus of measurement and testing experts has long been to use test scores cautiously.” The very next paragraph reads:

• For this discussion, it is perhaps most important to underscore that most tests are not designed to support inferences about related questions, such as how well students were taught, what effects their teachers had on their learning, why students in some schools or classrooms succeed while those in similar schools and classrooms do not, whether conditions in the schools have improved as a result of a policy change, or what policy makers should do to solidify gains or reverse declines. Answering those sorts of questions requires more and different kinds of evidence than test scores. Looking at test scores should be only a first step-not an end point-in considering questions about student achievement, or even more broadly, about student learning.

The rest of chapter 5 mostly catalogs and explains some of the many pitfalls in using data without caution. For example, “The validity of data on dropout rates is, in itself, an issue of serious concern in interpreting achievement data.” The report explains the problematic state of existing data and suggests that future definitions of data and the disaggregation of data be given careful consideration. They then caution:

• In the meantime, naïve aggregate comparison of test scores among race-ethnic groups in the District should be interpreted critically and cautiously. Thus, analysts need to carefully consider student backgrounds when comparing average scores, for example, by disaggregating by socioeconomic background.

In chapter six, the report outlines some of the areas of school operations they would study for “performance management.” The report quickly noted:

• The trend in the private sector has been away from treating the financial bottom line as the primary performance measure-a trend that could be seen as analogous to the trend in education away from treating test scores as the primary performance measure.

In the next paragraph the report referred to a “National Performance Review” of best practices in performance measurement and listed 8 important “elements” that should be included in any performance measurement initiative, including “no punishments: learning systems with tools, no ‘gotcha’;”. The next paragraph began:

• Likierman (2009), in contrast, pointed to a number of “traps” in performance management. Among the common mistakes were making comparisons only against prior performance within an organization, focusing on the past, focusing on the existence of data and not its quality, and “gaming” or otherwise distorting measures.

Later in chapter 6, after discussing some often unappreciated influences on student achievement the report states:

• Identifying valid and reliable measures of how well a school district is doing with respect to its fundamental mission is a challenging task. Test scores and enrollment numbers are often used because they are readily available and because many people believe they are very important (as discussed above).

But they had previously cautioned:

• It is important to note that measuring teacher effectiveness is a complex endeavor for which there is no established consensus in the education research community.

In noting “Many factors influence classroom instruction” they reported:

• A review of qualitative and quantitative research on school leadership found that principals’ influence is nearly as important as that of teachers.

Towards the end of chapter 6 they even acknowledge that “The capacity of central office staff is also important.” The report explains:

• School districts are highly complex systems that require effective management of school buildings, vehicles, and many noninstructional business operations, including food and nutrition services, safety and security, information technology, and procurement. These underlying systems make it possible for school systems to function, and when they do not work smoothly, it provides immediate and powerful signals of an ineffective system.

And:

• It is critical to note that researchers have documented correlations between the attributes of facilities and student outcomes, finding that both students and teachers benefit from having clean air, good light, and quiet, comfortable, and safe learning environments.

And in contrast to many efforts to bring in Teach For America employees, the report stated:

• For example, teacher credentials-such as scores on licensure tests or academic degrees-have not been useful in predicting which teachers will be more effective with students; in contrast, a teacher’s years of experience do appear to have some predictive power.

The report also clarified:

• Knowledge of the subject they teach-that is, a body of conceptual and factual knowledge in a particular field-has been identified as a necessary, but not sufficient, foundation for teachers. To foster learning, teachers also draw on understanding of how knowledge develops in a particular field, which means understanding the sorts of difficulties students typically have as their learning progresses and how to build on students’ gradually accumulating knowledge and understanding (for summaries of this research, see National Research Council, 2000, 2005a, 2010b). Other knowledge and skills, such as classroom management and the capacity to plan effective lessons, also play a role.

In summary, the value of the NRC report to districts outside of Washington, D.C., is that this highly respected research organization set forth some very fundamental cautions about utilizing data in school reform. These experts cited other respected experts in describing raw test data as having very “limited information” about public schools, and cautioned that definitions of data and the disaggregation of data are important considerations when interpreting data.

Michael T. Martin