How do you evaluate a school district?

Michael Martin, our Arizona colleague, continues to monitor the literature and to apply his considerable analytical talents to what he finds.  Below is his summary of the National Research Council report on their plan to evaluate the Washington, D.C. public schools.


The National Academies of Science, America's most prestigious organization in science, engineering and medicine, assigned their National Research Council to evaluate the District of Columbia's public schools at the request of the Washington, D.C., city council. A prepublication version of their first preliminary report was released on March 4, 2011, through the National Academies Press.

Titled "A Plan for Evaluating the District of Columbia's Public Schools: From Impressions to Evidence" this preliminary report laid out the parameters of how their study would be conducted. The importance of this "Plan" is that it also set out clear standards for the scientific basis of conducting that evaluation that provide a framework for considering the validity of other evaluations of public schools.

Most importantly, the report focused on the caution that should be exercised in evaluating public schools. In contrast to the many unofficial reports that crop up about public schools, the fourth paragraph of their executive summary clarified:

  • First, although many U.S. cities have undertaken significant reforms to change their schools and researchers have examined what they have done, there is no established model for evaluating a district involved in reform-or, for that matter, any district.

Using test scores for evaluating schools or teachers or districts was repeatedly singled out as nefarious. In their second chapter summary of education reform efforts around the United States, they note:

  • Researchers have raised questions about the inferences to be drawn from test scores, the most easily available measures of progress (see Chapter 5).

In chapter 5 they devote considerable effort starting on the first page to discredit the use of test scores for evaluating public schools. After noting:

  • There is a long history of relying on student test data as a measure of the effectiveness of public education, and it is tempting to simply rely on those readily available data for judgments about student achievement and about causes and effects. However, student test scores alone provide useful but limited information about the causes of improvements or variability in student performance.

The report ends the next paragraph with "therefore, it is important to remember that the consensus of measurement and testing experts has long been to use test scores cautiously." The very next paragraph reads:

  • For this discussion, it is perhaps most important to underscore that most tests are not designed to support inferences about related questions, such as how well students were taught, what effects their teachers had on their learning, why students in some schools or classrooms succeed while those in similar schools and classrooms do not, whether conditions in the schools have improved as a result of a policy change, or what policy makers should do to solidify gains or reverse declines. Answering those sorts of questions requires more and different kinds of evidence than test scores. Looking at test scores should be only a first step-not an end point-in considering questions about student achievement, or even more broadly, about student learning.

The rest of chapter 5 mostly catalogs and explains some of the many pitfalls in using data without caution. For example, "The validity of data on dropout rates is, in itself, an issue of serious concern in interpreting achievement data." The report explains the problematic state of existing data and suggests that future definitions of data and the disaggregation of data be given careful consideration. They then caution:

  • In the meantime, naïve aggregate comparison of test scores among race-ethnic groups in the District should be interpreted critically and cautiously. Thus, analysts need to carefully consider student backgrounds when comparing average scores, for example, by disaggregating by socioeconomic background.

In chapter six, the report outlines some of the areas of school operations they would study for "performance management." The report quickly noted:

  • The trend in the private sector has been away from treating the financial bottom line as the primary performance measure-a trend that could be seen as analogous to the trend in education away from treating test scores as the primary performance measure.

In the next paragraph the report referred to a "National Performance Review" of best practices in performance measurement and listed 8 important "elements" that should be included in any performance measurement initiative, including "no punishments: learning systems with tools, no 'gotcha';". The next paragraph began:

  • Likierman (2009), in contrast, pointed to a number of "traps" in performance management. Among the common mistakes were making comparisons only against prior performance within an organization, focusing on the past, focusing on the existence of data and not its quality, and "gaming" or otherwise distorting measures.

Later in chapter 6, after discussing some often unappreciated influences on student achievement the report states:

  • Identifying valid and reliable measures of how well a school district is doing with respect to its fundamental mission is a challenging task. Test scores and enrollment numbers are often used because they are readily available and because many people believe they are very important (as discussed above).

But they had previously cautioned:

  • It is important to note that measuring teacher effectiveness is a complex endeavor for which there is no established consensus in the education research community.

In noting "Many factors influence classroom instruction" they reported:

  • A review of qualitative and quantitative research on school leadership found that principals' influence is nearly as important as that of teachers.

Towards the end of chapter 6 they even acknowledge that "The capacity of central office staff is also important." The report explains:

  • School districts are highly complex systems that require effective management of school buildings, vehicles, and many noninstructional business operations, including food and nutrition services, safety and security, information technology, and procurement. These underlying systems make it possible for school systems to function, and when they do not work smoothly, it provides immediate and powerful signals of an ineffective system.


  • It is critical to note that researchers have documented correlations between the attributes of facilities and student outcomes, finding that both students and teachers benefit from having clean air, good light, and quiet, comfortable, and safe learning environments.

And in contrast to many efforts to bring in Teach For America employees, the report stated:

  • For example, teacher credentials-such as scores on licensure tests or academic degrees-have not been useful in predicting which teachers will be more effective with students; in contrast, a teacher's years of experience do appear to have some predictive power.

The report also clarified:

  • Knowledge of the subject they teach-that is, a body of conceptual and factual knowledge in a particular field-has been identified as a necessary, but not sufficient, foundation for teachers. To foster learning, teachers also draw on understanding of how knowledge develops in a particular field, which means understanding the sorts of difficulties students typically have as their learning progresses and how to build on students' gradually accumulating knowledge and understanding (for summaries of this research, see National Research Council, 2000, 2005a, 2010b). Other knowledge and skills, such as classroom management and the capacity to plan effective lessons, also play a role.

In summary, the value of the NRC report to districts outside of Washington, D.C., is that this highly respected research organization set forth some very fundamental cautions about utilizing data in school reform. These experts cited other respected experts in describing raw test data as having very "limited information" about public schools, and cautioned that definitions of data and the disaggregation of data are important considerations when interpreting data.

Michael T. Martin

Opinion: 7 reasons why teacher evaluations won't work

We received this from Monty Neill at FairTest

It was initially published in The Record

Sunday, March 13, 2011
Last updated: Sunday March 13, 2011, 7:46 AM
The Record

Bruce Baker is an associate professor in the Department of Educational Theory, Policy and Administration at Rutgers University Graduate School of Education in New Brunswick.

THE Teacher Effectiveness Task Force report issued March 3 by a panel appointed by Governor Christie recommended basing teacher evaluation significantly on student test scores. A few weeks earlier, acting Education Commissioner Christopher Cerf recommended that teacher tenure and dismissal, as well as compensation decisions, should be based largely on student assessment data.

Implicit in these recommendations is that the state and local districts would design a system for linking student assessment data to teachers for purposes of estimating teacher effectiveness. The goal of statistical “teacher effectiveness” measurement systems, including the most common approach called value-added modeling, is to estimate the extent to which a specific teacher contributes to the learning gains of a group of students assigned to that teacher in a given year.

Unfortunately, while this all sounds good, it just doesn’t work, at least not well enough to even begin considering using it for making high-stakes decisions about teacher tenure, dismissal or compensation.

Here’s why:

1) It is not possible to equate the difficulty of moving a group of children 5 points (or rank and percentile positions) at one end of a test scale to moving children 5 points at the other end.

Yet that is precisely what the proposed evaluations endeavor to accomplish. In such a system, the only fair way to compare one teacher to another would be to ensure that each has a randomly assigned group of children whose initial achievement is spread similarly across the testing scale.

Real schools and districts don’t work that way. It is also not possible to compare a 5 point gain in reading to a 5 point gain in math. These limitations undermine the entire proposed system.

2) Even with the best models and data, teacher ratings are highly inconsistent from year to year, and have very high rates of misclassification.

According to one recent major study, there is a 35 percent chance of identifying an average teacher as poor, given one year of data, and a 25 percent chance given three years. Getting a good rating is a statistical crap shoot.

3) If we rate the same teacher with the same students, but with two different tests in the same subject, we get very different results. University of California at Berkeley economist Jesse Rothstein, re-evaluating the findings of a much-touted Gates Foundation study, noted that more than 40 percent of teachers who placed in the bottom quarter on one test were in the top half when using an alternative test.

That is, teacher ratings based on the state assessment were only slightly better than a coin toss at identifying which teachers did well using the alternative assessment.

4) No matter how hard statisticians try, and no matter how good the data and statistical model, it is very difficult to separate a teacher’s effect on student learning gains from other classroom effects, like peer effect (race and poverty of peer group).

New Jersey schools are highly segregated, hampering our ability to make valid comparisons across teachers who work in vastly different settings. Statistical models attempt to adjust away these differences, but usually come up short.

5) Kids learn over the summer too, and higher-income kids learn more than their lower-income peers over the summer. As a result, annual testing data aren’t very useful for measuring teacher effectiveness. Annual (rather than fall-spring) testing data poses a significant disadvantage for teachers serving children whose summer learning lags. Setting aside all of the unresolvable problems above, this one can be fixed with fall-spring assessments.

But it cannot be resolved in any fast-tracked plan involving current New Jersey assessments, which are annual. The task force report irresponsibly ignores this huge concern, recommending fast-tracked use of current assessment data.

6) As noted by the task force, only those teachers responsible for reading and math in Grades 3 to 8 could readily be assigned ratings (less than 20 percent of teachers). Testing everything else is a foolish and expensive endeavor. This means school districts will need separate contracts for separate classes of teachers and will have limited ability to move teachers from one contract type to another.

Further, pundits have been arguing that we should be using effectiveness measures instead of experience to implement layoffs due to budget cuts and that we shouldn’t be laying off core classroom teachers in Grades 3 to 8. But those are the only teachers for whom “effectiveness” measures would be available.

7) Basing teacher evaluations, tenure decisions and dismissal decisions on scores that may be influenced by which students a teacher serves provides a substantial disincentive for teachers to serve kids with the greatest needs, disruptive kids or kids with disruptive family lives.

Many of these factors are not, and cannot, be captured by variables in the best models. Some have argued that including value-added metrics in teacher evaluation reduces the ability of school administrators to arbitrarily dismiss a teacher.

Opportunities for sabotage

Rather, use of these metrics provides new opportunities to sabotage a teacher’s career through creative student assignment practices.

In short, we may be able to estimate a statistical model that suggests that teacher effects vary widely across the education system – that teachers matter. But we would be hard-pressed to use that model to identify with any degree of certainty which individual teachers are good teachers and which are bad.

Contrary to education reform wisdom, adopting such problematic measures will not make the teaching profession a more desirable career option for America’s best and brightest college graduates.

In fact, it will likely make things much worse. Establishing a system where achieving tenure or getting a raise becomes a roll of the dice and where a teacher’s career can be ended by a roll of the dice is no way to improve the teacher work force.

Using these metrics as a basis for dismissing teachers will not reduce the legal hassles associated with removal of tenured teachers. As the first rounds of teachers are dismissed by random error of statistical models alone, by manipulation of student assignments or when larger shares of minority teachers are dismissed largely as a function of the students they serve, there will likely be a new flood of lawsuits like none ever previously experienced.

Employment lawyers, sharpen your pencils and round up your statistics experts.

Authors of the task force report might argue that they are putting only 45 percent of the weight of evaluations on these measures. The rest will include a mix of other objective and subjective measures. The reality of an evaluation that includes a single large, or even significant, weight placed on a single quantified factor is that that specific factor necessarily becomes the tipping point, or trigger mechanism.

It may be 45 percent of the evaluation weight, but it becomes 100 percent of the decision, because it’s a fixed, clearly defined (though poorly estimated) metric.

Self-proclaimed “reformers” make the argument that the present system of teacher evaluation is so bad as to be non-existent. Reformers argue that the current system has a 100 percent error rate (assuming current evaluations label all teachers as good, when all, they suggest, are actually bad).

From the “reformer” viewpoint, something is always better than nothing.

Value-added is something.

We must do something.

Therefore, we must do value-added.

Self-fulfilling prophecy

Reformers also point to studies showing that teachers’ value-added scores are the best predictor (albeit a weak and error-prone one) of a teacher’s future value-added scores – a self-fulfilling prophecy.

These arguments are incredibly flimsy.

In response, I often explain that if we lived in a society where people walked everywhere, and a new automotive invention came along, but had the tendency to burst into a ball of flames on every third start, I think I’d walk.

Now is a time to walk!

Some innovations just aren’t ready for broad public adoption — and some may never be. Some, like this one, may not be a very good idea to begin with.

That said, improving teacher evaluation is not a simple either/or and now may be a good time to step outside the false dichotomy and discuss more productive alternatives.

You Pays Your Money and You Takes Your Choice

This was originally posted on the Kennewick School District Citizens web page and is reprinted here in the hope of reaching a different audience.

The following pairs of choices represent contrasting views of public education.  Please read each pair, then circle the view you would like to see in place for the schools of your children or grandchildren.  Following the pairings we will tell you how to tally up your selections and lay out some options.


1.  Academics

a.     Elementary schools are places where playing and learning are combined with alternative pedagogic approaches to academics.

b.     Elementary schools are places where the focus is on specific, narrow areas of the curriculum selected at the national level and the pedagogy favors teaching to the mandated tests.

2. Learning Environment

a.     The focus is on learning in a fear-free environment where creativity and risk-taking are encouraged.

b.     Creativity and risk-taking are avoided in an environment where testing failure looms at every turn.


1.  Respect

a.     Teachers are highly respected and appreciated.

b.     Teachers are regularly ridiculed in the media and by national leaders.

2.  Training and Selection

a.     All teachers have a master’s degree and selection into the profession is difficult.

b.     A master’s degree is not seen as beneficial by national leaders.  The current trend is to hire beginners with as little as 5 weeks of training in pedagogy.

3.  Autonomy

a.     Teachers are given considerable independence to select methods and materials appropriate for their students.

b.     Curriculum is set at national and state levels.  Even the pacing for classes may be mandated at the district level.  Materials may be selected at the local level but only from a prescribed list.

4.  Unions

a.     The labor union works cooperatively with the schools and government to improve educational opportunities.

b.     Teacher unions are seen primarily as protectors of incompetent teachers and are regularly bashed by the media and by highly-placed Federal officials.


1.  Delivery of Services

a.     Schools have full autonomy in developing daily delivery of education services.

b.     The district determines the schedule and content of the services.

2.  Curriculum Decisions

a.     Schools plan their own curriculum to reflect local concerns and needs of children.

b.     Curriculum is determined by state and national standards and mandated assessments.


1.  High-Stakes Exams

a.     No mandatory state or national high-stakes exams.

b.     Mandatory state standardized tests for students determine student progress, Federal funding, and even teacher retention.

2.  Test Development

a.     Teachers at each school develop their own tests and use descriptive feedback rather than numeric grades.

b.     Mandated testing is developed at the state or national level based on pre-determined standards for each grade level, is usually machine scored, and provides little useful feedback to the parent.

3. Cost in Time and Resources

a.     Little extra cost since the tests are teacher developed as part of the regular instructional process.

b.     State-level costs are in the hundreds of millions of dollars annually and time lost to test-prep and actual testing is measured in weeks of lost classroom time.


Step 1: Add up all of the “a’s”.

Step 2: If you had 3 or less, do nothing.  You are getting what you want.

Step 3: If you had 4 to 6, start calling or writing your school board and/or legislators to get more of what you want.

Step 4:  If you got 7 or more, move to Finland.  It is your best chance to get what you want for your children, because all of the 11 “a” choices are currently in place.  Ironic, isn’t it, that a country that does not do mandatory standardized testing produces the highest average scores on international tests in science and math.

As we said, you pays your money and you takes your choice.  Our choice is being made for us by corporate leaders, legislators, and Federal officials.

Is it your choice?  When are you going to speak up for the children?

Creating a New School of Thought-Updated


Thomas Friedman, in his book The World is Flat, writes that if you wish to be unaffected by the sea-change of events coming from the flattening of the global economy, you need to be special (Tom Brady), specialized (work that cannot be digitized, ie. brain surgeons), or anchored (barber). Even the specialized must constantly grow their skills or be pushed out. The primary skill in this arena is the ability to learn how to learn.

Michael Fullan in Leadership and Sustainability says he is particularly concerned that current trends in curriculum lack the “deep learning” needed to meet the demands of tomorrow’s complex global society.

A number of futurists have taken a look at the emerging world of work and projected the type of skills workers will need to be successful in the coming economy. The North Central Educational Laboratory developed a list of skills based on eight nationally recognized skill sets as well as literature reviews and input from a wide variety of practitioners. The result is called the enGuage 21st Century Skills and is provided here.

Digital-Age Literacy includes basic, scientific, mathematical, and technological literacies. Visual, information, and cultural literacies are also implied along with global awareness.

Inventive Thinking is the name given to adaptability and the ability to manage complexity. It includes curiosity, creativity, and risk taking as well as higher order thinking and sound reasoning.

Effective Communication includes but goes beyond interactive communication to include teaming, collaboration, and interpersonal skills. Personal and social responsibility are also part of this skill set.

High Productivity refers to the ability to prioritize, plan, and manage for results. It includes the use of real-world tools and the production of relevant, high-quality products.

In his book, The New Basics, Education and the Future of Work in the Telematic Age, David Thornburg describes his analysis of the skills listed in more than 500 current job advertisements. The list he developed from this research is similar to that provided by enGuage and included technological fluency, communication skills, teamwork, problem solving, and creativity. Other groups and authors have weighed in on the subject by producing lists of skill requirements that contain many of these same elements.

Although these lists have been developed primarily to serve the needs of the work force, close examination reveals the utility of such a skill set in the broader world as well. A person competent in the identified skills would be in good shape to face most of life’s challenges.


Even though thinking skills may be of great importance in the work place and in life, we are left with the problems of adding all of this new work to the curriculum and of managing the development of the skills. The first of these is not as difficult as might be perceived. As the general curriculum is delivered the thinking skills are embedded in the instruction, primarily through questioning strategies of the teacher. Many examples are provided in this paper and on the website. We believe that the only effective way to solve the second problem is to teach the students to manage their own thinking. This would be consistent with the study of human development and with research findings in the cognitive sciences.

Piaget first presented the idea that that learning progresses through developmental stages. Kohlberg and others have shown that many human characteristics such as moral growth also proceed through stages as an individual matures. Kurt Fischer and others have more recently refined the ladder-like models of development with one of recurring growth cycles for both behavior and brain development. These cycles are individual and based on the life experience of each person making it nearly impossible to manage the process in a lock-step curriculum. A way to do this is to teach each person to be aware of their own thinking (metacognition) and to use this knowledge strategically. The teacher’s role is to introduce new thinking skills where developmentally appropriate, make the use of the skills explicit, use the curriculum to practice the skills at an optimal level, and help the student develop a “Thinking Journal” that will follow them throughout their school years. The journal will include a thinking map and periodic inventories of learning style, multiple intelligence profile, and other behavioral data as appropriate.

Personal Intelligence Management

As we develop our ability to manage our own intelligence or to assist others in doing the same, we must begin by considering what is known about learning and by organizing the universe of what we are to manage. One way of doing this is to map three domains that play a significant role in becoming a more effective learner, thinker, and problem solver. These three domains are self knowledge, the nature of what is to be learned, and the skill sets that we can use to achieve our purposes.

Self Knowledge

Over the past couple of decades information about how the human brain works has exploded. Since the brain is our tool for learning it is important that each person have a basic understanding of how the tool works. Discussion should begin in the early grades with additional details added as students become older. For those who are already in high school or beyond, instruction should be provided through workshops and courses or, at least, through self-learning modules available on the Internet or elsewhere.

A second dimension of self-knowledge includes an array of personal attributes that affect learning. These include, but are not limited to, learning style, multiple intelligence profile, attitudes, emotional intelligence profile, and the current state of knowledge (both tacit and explicit) in the domain of interest. Many of these can be appraised with inventories that are readily available. Those for which inventories have not been developed must at least be raised to the awareness level in the mind of the learner because such things as attitudes toward learning have a profound affect on being willing to attempt a learning task or sticking to it long enough to accomplish significant learning. These self-knowledge characteristics can be displayed on an attribute map and modified as new experiences affect the learner.

The Nature of What is to be Learned

Learning strategies vary with the nature of what is to be learned. Declarative knowledge, facts such as who, what when, where and so on require far different methods than procedural knowledge, learning how to do something such as riding a bike. A third category, contextual knowledge, knowing the conditions that help us decide what procedures to use in a given situation, is learned primarily through the experiences one encounters in similar situations. As an individual is faced with a new learning situation it is useful to think about which of the categories are included and to plan appropriate learning strategies for each.

Skill Sets

  • Metacognition

Executive Control includes assessing our current knowledge state, knowing what we know and being able to plan goals and procedures, and monitoring progress along the way.

Knowledge of Self, in this case, refers to knowing how we feel about the situation and our own mental state as it relates to our ability to be successful. Being aware of our commitment to the completion of our effort and our attitudes about our skill play an important role in accepting a challenge and moving forward.

  • Higher-Level Thinking Skills

A variety of thinking operations are available to the learner as they attempt to navigate their way through the maze of problems one encounters in life. Some of them are depicted in the chart above A complete description can be found at:


Our purpose in writing this article is to set the stage for a series of activities that you can use personally or with your students to practice the strategies of the skilled thinker. There are many ways to conceptualize the organization of these skills. We have chosen to group them into the categories of gathering, assessing or considering, and applying.

Gathering skills can be thought of as discrete skills that are used to recall or collect bits of data that are thought to be useful in the pursuit of the solution to some sort of problematic situation. As the data begins to accumulate, the skilled thinker considers and assesses what has been gathered and begins to sketch out potential solutions. Often, additional gathering steps are required to fill in gaps. Application skills blend all of the pieces into a strategic attack on the problem at hand.

It is our contention that acquisition of these skills is often left to chance until students reach upper level courses. Individuals who do not take such course work are at a serious disadvantage. We would further argue that these skills can be learned beginning at a young age and should be part of the planned curriculum of every school. Assuming the reader wishes to improve their own thinking or to teach the skills to others, what follows is an outline of some of the critical components with a brief explanation.

The human brain specializes in noticing discrepancies and trying to resolve them. When a problem is perceived and gains our attention, a search through the files of our experiences commences and does not stop until a solution emerges or we give up in frustration. This often turns out to be a rather messy process and is seldom linear if the problem has any significance. It might take minutes to months or years and may never be resolved. The solution of one problem often leads to new questions and, in the brains of the most skilled and artistic among us, can lead to a lifetime quest for knowledge. What follows is not meant to be a sequential list of the skills to be applied, but is rather a map with a partial listing of stops where we can gain sustenance, repack our gear, or check out the vistas . Some may be revisited many times while others may not be touched on a particular journey. There are many additional unnamed stops that can be added to our maps as we progress through life’s journeys. If one were to embark on a major investigation or study, what are some of the skills that would be useful?

Gathering Information

Among the most basic of the thinking skills are those that help us gather information. These include recalling what we already have learned, observing, collecting, organizing, and classifying. We shall examine each in turn and provide the reader with examples and some practice exercises to begin the process of making the skill your own. These activities can, of course, be used as teachers of models of the skills that can be taught to others, beginning at very early ages. Additional exercises will be added in a new section of our web site as time allows.

Remembering information involves both putting it in and retrieving it later. When we try to memorize something we can use what is called rehearsal to encode the information by repeating the information over and over using words or mental pictures. Many of us have learned poems or lines to a play in this manner. Mnemonics is an encoding strategy that is useful for information such as the order of colors in the visual light spectrum (Roy G. Biv, red, orange, yellow, green, blue, indigo, violet) or items on a grocery list. Both rehearsal and mnemonics are used to encode and then later to retrieve the same information. When we do not initially recall needed information we can sometimes activate the memory by remembering where we were, whom we were with or when it was learned. If material is clearly understood when it is first learned, that is the information is connected to prior knowledge, events, or emotional content, it will most often be easier to retrieve it later. Here is a link to a resource on memory techniques:

The solution to our problem may have been found by someone else or they may have at least made a good start on it. One of our first steps should be a review of the literature. What have others said about our topic? What research has been done and what were the results. A good library is your best friend at this point. With the advent of computers and the Internet, searches using Google: or other search engines can save us many hours as we collect background information.

Direct observation is a tool used to gather information throughout the problem solving process. We vary greatly in our observation skills but can become more proficient through experience. Observing for details is the basis of all other observation skills. Regular attempts to describe in writing or by drawing a variety of objects in your surroundings will foster greater skill in this area. Size, color, relationship to other things, change, and texture are some of the things to look for. Having observed two or more objects or events, it is usually easy to distinguish differences or similarities but it takes skill to identify those that are relevant to a particular study or investigation. Additional information on the skills of observing can be found at

Many schemes have been devised to keep track of the many details and bits of information that one collects in the course of an investigation. One of the simplest is to use note cards to jot down the information (being careful to note the source). The cards can be easily rearranged to identify relationships, hierarchies, etc. Computer software is also available to do the same thing and may be more convenient for people with access to the equipment. An organizational skill of great value is the ability to classify or categorize the individual items in the large pool of data. Practicing this skill consists of finding similarities between various items in a list, then giving each grouping an appropriate name. This seemingly simple activity requires careful thought if one is to develop mutually exclusive categories based on relevant criteria.

Considering and Assessing Information

As information begins to pile up and is organized, the mind can begin to consider, manipulate, and assess it. What are the parts, how do they fit together and what does it all mean? Do patterns emerge, how is this like other things I know, and can I use this to predict how things might work in the future? The answers to many of these questions may direct us back to the need for additional information from the gathering stage or trigger a thought that leads to a creative insight.

When we search for the main idea of a paragraph, try to identify personality characteristics of a character in a play, or search for the structure of a protein we use a skill called analysis. Whether trying to develop an understanding of an historical event or learn a new game, taking information apart to look at the pieces and determine what is relevant, analysis comes into play.

One aspect of analysis is to identify essential properties of a concept. For example, when we picture the concept “chair” in our brain we must have an idea of the essential properties of a thing before it can be called a chair. Before you call something a chair it must have a surface to sit on and have some sort of backrest. You may think of other essential attributes of “chairness.” Practicing this skill consists of identifying concepts and attempting to list the essential properties. For example, what are the essential properties of “mind”? What about “government” or “protozoan”?

Each person brings their own point of view to every situation they encounter, but most are not aware that their perspective influences decisions they make, how they attack problems or how they approach various tasks. A powerful form of thinking is analysis of your own perspectives – to consider the principles you believe in and the basis for these principles or to do the same for another person with whom you are having a discussion or argument. Debate is a formalized activity that relies heavily on this skill. Here is a link to some graphic organizers to help with analysis:

Once the pieces of a concept have been revealed, we can begin to compare and contrast it with other things we know. How does the new idea fit into the scheme of things as perceived by our own brain? Tables, charts, and Venn diagrams have all been used to show where ideas contrast or overlap. Learning to use graphic organizers can be a big help in developing these skills.

Interpretation is another of the skills we use when considering input. It is easy to read too much into information, especially when it seems to confirm a previously held belief. Headlines and “sound bites” can easily lead to misinterpretation. For example, a headline that reads, “Local Teachers’ Pay Raises Triple the State Average,” might tempt the reader to believe that the raises were unreasonable. Without knowing what both local teachers made and the state average before and after the increases it is impossible to interpret the information fairly. It could be that even after the raises, local teachers were still well below the state average. If we interpret the information received and simultaneously think about other data we might need to avoid misinterpretation, our skill level is growing. Graphic organizers for compare/contrast can be found here:

Hypothesizing may be used either in an attempt to generate potential causes for an event or to predict the outcome based on a given set of circumstances. One way to develop your skills of hypothesizing is to play the game of hypothesizing. To play the game, one person merely asks a question beginning with “Why” or “What if” and other participants offer as many hypotheses as they can to answer the question. As in brainstorming, all answers are accepted, no matter how far fetched or “off-the-wall.” A recorder should keep track of all of the answers. For a more detailed explanation of hypothesizing and some sample exercises, see:

Many activities in school and in life require us to examine a large amount of information and try to make sense of it. Pattern recognition is a valuable skill that helps us identify relationships among the mass of data, thus helping us keep chunks of information in mind as we attempt to deal with the situation. For example, if we observe the following sequence of numbers: 5, 10, 20, 40, ___, 160, we see that each successive number in the sequence is double the previous number. We can surmise that the missing number is 80 with a great deal of confidence because we recognize the pattern of doubling. Patterns occur in speech, literature, mathematics, history, science, and virtually everything one can think of. Acquiring the habit of looking for patterns in everything we do helps us integrate the vast quantities of data we are subjected to in our daily lives.

Synthesis is the opposite of analysis. Instead of taking something apart and looking at the pieces as we do in analysis, synthesis involves looking at the pieces and generating a bigger idea. For example, after determining the speed of sound in air, water, and steel and knowing the molecular components and spatial relationships of each medium, one might synthesize a theory of sound propagation.

In the broadest possible sense, creative thinking is coming up with something new. To a young child, creativity is used to solve virtually every problem encountered because cultural patterns and common solutions have not yet been learned. As we grow older and our experience broadens, it becomes easier to use tried-and-true methods and we are, in fact, encouraged to do so by parents and school programs that reward such behavior. Still, we prize the alternative view, the unexpected revelation that cuts through to a new insight. As with the other thinking skills discussed above, creative thinking can be learned (and taught) through practice and experience. Developing the habit of generating off-beat solutions, looking at problems from multiple perspectives, or using tools like “Synectics” or “Lateral Thinking” enhances the chance of coming up with something new.

Application Strategies

What is described here as an end point of higher-level thinking, the combined use of the many skills to solve a problem or make a decision, is actually the starting point as well. Before we can strategically martial the discrete skills, we must perceive that we have a problem to address, a decision to be made, or an issue to be resolved. The nature of our difficulty helps determine our plan of attack.

Turning first to problem solving, we find that experts have discussed a variety of approaches, but most involve a stepwise approach. Polya in his little book, How to Solve It, suggests a 4-step model. First we have to clearly understand the problem and what is required. Second we must attempt to see how the pieces are connected in order to make a plan. Third, we carry out the plan, and fourth, we look back at the plan and process to see how it might be improved. This seems rather straightforward, but the devil is in the details. Complex problems may require breaking up into smaller chunks, for example, in order to even understand what is going on.

Other authors have suggested that when solving a problem identifying the goal you are trying to accomplish should be the first step followed by identification of limiting conditions or constraints. The third step would be to find different ways of overcoming the constraints that can then be tried out and evaluated for effectiveness. Here is a site with another model along with some practice exercises:

Decision-making models have also appeared in various formats. A couple will be presented here, but the reader is advised to adapt these and other models to your own needs. What is comfortable for one is not necessarily so for all. A six-step model has been proposed which includes the following steps: Specify the parts of what is at issue and define any unclear terms; determine appropriate areas of concern; predict both good and bad consequences for each concern listed in step 2; select the 3 or 4 most important good and the 3 or 4 most important bad consequences; assess the sources of each of the good and bad consequences identified by rating the confidence you have in the sources; make a decision by weighing the good side against the bad side.

A simpler model consists of merely making a matrix with the possible decisions at the top of the page with the factors we wish to consider listed down the left side. In the intersections on the matrix, a plus or minus can be used to indicate good or bad and a number from 1 to 5 to indicate importance or impact of the factor. Top-rated decisions will be those with a high positive score across a number of factors. It is sometimes necessary to weight the factors as well since some might be deal killers. Here is an example of how a decision matrix can be used:

Experimental inquiry, sometimes called the scientific method, is a strategy used in a wide variety of instances but is especially suited to the sciences. It uses both hypotheses (see above) and experiments to reach conclusions regarding puzzling or problematic circumstances. Core elements of the strategy include observation of a discrepant or puzzling event, thinking about and explaining why it happened, developing a prediction (hypothesis), designing and carrying out an experiment to test the prediction, and explaining the results of the experiment in light of the experimental results. Inquiry from the student’s viewpoint might look like this:


What is it? How does it work?

Could this explain?

How can I prove it?

Perform the experiment

What is my conclusion?

New Question?


519 N. Vancouver St.

Kennewick, WA 99336


How We Got Here (and what can we do about it?)

A presentation to AAUW, Enterprise, Oregon September 13, 2010.

Please take a moment to answer each of the following.  Base your answer on what you believe is appropriate for your own child:

1.         The primary purpose of reading instruction is:

  • Life-long love of reading
  • To read and comprehend the written word at a increasing level of sophistication
  • Score at a high level on a standardized, multiple-choice reading achievement test

2.              The primary purpose of school mathematics instruction is:

  • Become proficient in the math necessary for everyday living and for entry-level employment
  • Understand math concepts at increasing levels of sophistication allowing students to pursue careers in math, science engineering, or technology.
  • Score high on a standardized, multiple-choice math achievement test

3.              The primary purpose of the public school is:

  • Provide each child with a healthy, safe, supported and challenging learning environment
  • Prepare each student for life, the next level of education, and/or a career
  • Score high on standardized, multiple choice achievement tests in reading, math, and writing

4.              What do you want the schools to do for your children, grandchildren, or those of your neighbors?

For many years I did education futures and visioning workshops with school communities.  When I asked question 4 at these community gatherings I don’t recall ever hearing anyone say they wanted to narrow the curriculum, lower the graduation rate, or assess students (or their teachers) primarily on the basis of standardized test scores.  Unfortunately, that is where we are, mired in a senseless set of so-called reforms that are only driving us closer and closer to the cliff.

I have been troubled by the answers to this set of questions chosen by our business and political leaders for at least 10 years, but the general population has been indoctrinated by the mantra, “Our schools are a failure.  We must increase scores on international tests to remain competitive in the global economy.”  Last year a small group of Kennewick citizens came together to discuss what was happening in our town. We considered how best to inform the public regarding the research supporting or opposing the reforms being promulgated by the school board and the government with the backing of top business officials like Bill Gates and Eli Broad.  Kennewick School District Citizens is the result.

The rest of this presentation will explore how we got here and what a small band of Kennewick citizens is trying to do about it.

How We Got Here

The following section presents information about the key decision points leading to current education policy at the national level.  Each is then followed with research findings as reported on

A Nation at Risk-Reagan-1983

  • “The public schools are broken and the nation’s competitive economic position is compromised.”
  • “If an unfriendly foreign power had attempted to impose on America the mediocre educational performance that exists today, we might well have viewed it as an act of war.”

Rothstein’s 2008 Response (See

  • Was based solely on declining SAT scores which we now know was the result of adding non-college bound students to the pool of test takers.
  • The economic ills of 1980 were soon reversed and despite no real change in the school programs until the late 90’s, the U.S. economy was the strongest in the world for the entire period of time, even lasting until 2007 (The last year studied).
  • A Nation at Risk ignored the responsibility of other social and economic institutions for learning.

Ravitch’s  Findings (See <>).

  • Diane Ravitch served in the administration of the first George Bush and has, for 20 years or more, championed the No Child Left Behind Act and the gradual take over of the public schools.  As a consummate researcher, however, Ravitch has followed the implementation of these “reforms” and, voila, she found she had been wrong.  The reforms have not worked and in some cases made the situation worse.  She reports this in her book, The Death and Life of the Great American School System.

No Child Left Behind-Bush II- 2001

  • States must set standards and measurable goals
  • Requires assessments at many grade levels and that all students be proficient by 2014.
  • Claims increased accountability
  • Highly qualified teachers
  • School choice (Vouchers and charter schools)


My comments follow:


Kennewick, along with most cities and towns in the country, has relied on standardized testing as a way to improve achievement for the past twenty years despite the arguments of assessment experts (they designed the tests) who said they were not appropriate for this purpose and instruction experts (they know what needs to be taught), who argued for whole child solutions.  Kennewick School District has claimed vast improvement over the time period although the results don’t show up in higher graduation rates, greater college admissions, or any other measure than the same standardized tests at some grades, but not others.  TIME FOR TESTING IS INCREDIBLE.


States have gradually eroded the cut scores and lowered standards to increase test scores and remain “making AYP”

IMPACT ON ELECTIVES is committed to education of the whole child.  We believe that restricted curriculum that shortchanges the arts, music, science, history, physical education and vocational subjects is a disservice to our children and places the entire culture at risk.  Others share our views and are beginning to speak up on behalf of their children.  We think it is time for Kennewick residents to speak up as well.

EAST LANSING, Mich. – America’s increasing reliance on standardized testing as a yardstick for educational success is a flawed policy that threatens to undermine the nation’s strengths of creativity and innovation, according to a provocative new book from a Michigan State University scholar.

By grading student success on government-set standards in a limited number of subjects such as math, reading and science, Yong Zhao argues the United States is eager to “throw away” one of its global advantages – an education that respects individual talents and does not dictate what students learn or how teachers teach.


I am a test-weary parent/community member concerned for our schools and for the future of our children.

Prior to about 2000, important school policy decisions were deferred to local school boards and professional educators and administrators trained and committed to the education of our children.  Since the recent reform initiatives our schools are being driven by standardized tests and the mandates of the federal NCLB and RttT. I want my schools back, I want our teachers empowered again, I want our children to run off to school excited about learning and return home more prepared for the world we live in. It’s not a perfect system, but I believe our children are better off in the hands of professionally trained and committed teachers, than in the clutches of profit seeking test publishers, political rhetoric, and greedy, for-profit school advocates.


Lack of highly qualified” teachers, AYP, unfunded mandates.

RACE TO THE TOP-Obama-Current

  • Links student test scores to teacher/principal evaluation
  • Sets National Standards/Assessment
  • Expands school choice forcing more charter schools


My comments follow:


Economic Policy Institute report:

If new laws or policies specifically require that teachers be fired if their students’ test scores do not rise by a certain amount, then more teachers might well be terminated than is now the case. But there is not strong evidence to indicate either that the departing teachers would actually be the weakest teachers, or that the departing teachers would be replaced by more effective ones. There is also little or no evidence for the claim that teachers will be more motivated to improve student learning if teachers are evaluated or monetarily rewarded for student test score gains.  (the ten scholars whose names are on the document are some of the most eminent in educational circles, including among their midst former Presidents of the American Educational Research Association and the National Council on Measurement in Education, two of the three professional organizations most involved with psychological measurement,)


Standards advocates argue that common standards are necessary for keeping the nation competitive in a global economy. But Mathis points out that research does not support this oft-expressed rationale. No studies support a true causal relationship between national standards and economic competitiveness, and at the most superficial level we know that nations with centralized standards generally tend to perform no better (or worse) on international tests than those without. Further, research shows that national economic competitiveness is influenced far more by economic decisions than by test scores.

Mathis also raises questions about the rapid development of the common-core standards, the lack of field testing, and the overarching need for any high-stakes consequences to be “valid,” pursuant to established professional guidelines. Given these concerns, he says that the prospect of positive effects on educational quality or equality “seems improbable.”

CHARTER SCHOOLS            Stanford Study

Of the 2403 charter schools reflected on the curve, 46 percent of charter schools have math gains that are statistically indistinguishable from the  average growth among their TPS comparisons.

Charters whose math growth exceeded their TPS equivalent growth by a significant amount account for 17percent of the total.

The remaining group, 37 percent of charter schools, posted math gains that were

significantly below what their students would have seen if they enrolled in local traditional public schools instead.

What We Are Doing About It