Skip to:

Education Research

  • When The Legend Becomes Fact, Print The Fact Sheet

    Written on November 11, 2011

    The New Teacher Project (TNTP) just released a "fact sheet" on value-added (VA) analysis. I’m all for efforts to clarify complex topics such as VA, and, without question, there is a great deal of misinformation floating around on this subject, both "pro-" and "anti-."

    The fact sheet presents five sets of “myths and facts." Three of the “myths” seem somewhat unnecessary: that there’s no research behind VA; that teachers will be evaluated based solely on test scores; and that VA is useless because it’s not perfect. Almost nobody believes or makes these arguments (at least in my experience). But I guess it never hurts to clarify.

    In contrast, the other two are very common arguments, but they are not myths. They are serious issues with concrete policy implications. If there are any myths, they're in the "facts" column.

  • The Impact Of The Principal In The Classroom

    Written on November 3, 2011

    Direct observation is way of gathering data by watching behavior or events as they occur; for example, a teacher teaching a lesson. This methodology is important to teacher induction and professional development, as well as teacher evaluation. Yet, direct observation has a major shortcoming: it is a rather obtrusive data gathering technique. In other words, we know the observer can influence the situation and the behavior of those being observed. We also know people do not behave the same way when they know they are being watched. In psychology, these forms of reactivity are known as the Hawthorne effect, and the observer- or experimenter- expectancy effect (also here).

    Social scientists and medical researchers are well aware of these issues and the fact that research findings don’t mean a whole lot when the researcher and/or the study participants know the purpose of the research and/or are aware that they are being observed or tested. To circumvent these obstacles, techniques like “mild deception” and “covert observation” are frequently used in social science research.

    For example, experimenters often take advantage of “cover stories” which give subjects a sensible rationale for the research while preventing them from knowing (or guessing) the true goals of the study, which would threaten the experiment’s internal validity – see here. Also, researchers use double-blind designs, which, in the medical field, mean that neither the research participant nor the researcher know when the treatment or the placebo are being administered.

  • In Research, What Does A "Significant Effect" Mean?

    Written on November 1, 2011

    If you follow education research – or quantitative work in any field – you’ll often hear the term “significant effect." For example, you will frequently read research papers saying that a given intervention, such as charter school attendance or participation in a tutoring program, had “significant effects," positive or negative, on achievement outcomes.

    This term by itself is usually sufficient to get people who support the policy in question extremely excited, and to compel them to announce boldly that their policy “works." They’re often overinterpreting the results, but there’s a good reason for this. The problem is that “significant effect” is a statistical term, and it doesn’t always mean what it appears to mean. As most people understand the words, “significant effects” are often neither significant nor necessarily effects.

    Let’s very quickly clear this up, one word at a time, working backwards.

  • The Teachers' Union Hypothesis

    Written on October 21, 2011

    For the past couple of months, Steve Brill's new book has served to step up the eternally-beneath-the-surface hypothesis that teachers’ unions are the primary obstacle to improving educational outcomes in the U.S. The general idea is that unions block “needed reforms," such as merit pay and other forms of test-based accountability for teachers, and that they “protect bad teachers” from being fired.

    Teachers’ unions are a convenient target. For one thing, a significant proportion of Americans aren’t crazy about unions of any type. Moreover, portraying unions as the villain in the education reform drama facilitates the (mostly false) policy-based distinction between teachers and the organizations that represent them – put simply, “love teachers, hate their unions." Under the auspices of this dichotomy, people can advocate for changes , such as teacher-level personnel policies based partially on testing results, without having to address why most teachers oppose them (a badly needed conversation).

    No, teachers’ unions aren’t perfect, because the teachers to whom they give voice aren’t perfect. There are literally thousands of unions, and, just like districts, legislatures and all other institutions, they make mistakes. But I believe strongly in separating opinion and anecdote from actual evidence, and the simple fact is that the pervasive argument that unions are a substantial cause of low student performance has a weak empirical basis, while the evidence that unions are a primary cause of low performance does not exist.

  • What Are "Middle Class Schools"?

    Written on September 21, 2011

    An organization called “The Third Way” released a report last week, in which they present descriptive data on what they call “middle class schools." The primary conclusion of their analysis is that “middle class schools” aren’t “making the grade," and that they are “falling short on their most basic 21st century mission: To prepare kids to get a college degree." They also argue that “middle class schools” are largely ignored in our debate and policymaking, and we need a “second phase of school reform” in order to address this deficit.

    The Wall Street Journal swallowed the report whole, running a story presenting Third Way’s findings under the headline “Middle class schools fail to make the grade."

    To be clear, I think that our education policy debates do focus on lower-income schools to a degree that sometimes ignores those closer to the middle of the distribution. So, it’s definitely worthwhile to take a look at “middle class schools’” performance and how it can be improved. In other words, I’m very receptive to the underlying purpose of the report.

    That said, this analysis consists mostly of arbitrary measurement and flawed, vague interpretations. As a result, it actually offers little meaningful insight.

  • The Real Charter School Experiment

    Written on September 12, 2011

    The New York Times reports that there is a pilot program in Houston, called the "Apollo 20 Program" in which some of the district’s regular public schools are "mimicking" the practices of high-performing charter schools. According to the Times article, the group of pilot schools seek to replicate five of the practices commonly used by high-flying charters: extended school time; extensive tutoring; more selective hiring of principals and teachers; “data-driven” instruction, including frequent diagnostic quizzing; and a “no excuses” culture of high expectations.

    In theory, this pilot program is a good idea, since a primary mission of charter schools should be as a testing ground for new policies and practices that could help to improve all schools. More than a decade of evidence has made it very clear that there’s nothing about "charterness" that makes a school successful – and indeed, only a handful get excellent results. So instead of arguing along the tired old pro-/anti-charter lines, we should, like Houston, be asking why these schools excel and working to see if we can use this information productively.

    I’ll be watching to see how the pilot schools end up doing. I’m also hoping that the analysis (the program is being overseen by Harvard’s EdLabs) includes some effort to separate out the effects of each of the five replicated practices. If so, I’m guessing that we will find that the difference between high- and low-performing urban schools depends more than anything else on two factors: time and money.

  • Attracting The "Best Candidates" To Teaching

    Written on August 15, 2011

    ** Also posted here on "Valerie Strauss' Answer Sheet" in the Washington Post

    One of the few issues that all sides in the education debate agree upon is the desirability of attracting “better people” into the teaching profession. While this certainly includes the possibility of using policy to lure career-switchers, most of the focus is on attracting “top” candidates right out of college or graduate school.

    The common metric that is used to identify these “top” candidates is their pre-service (especially college) characteristics and performance. Most commonly, people call for the need to attract teachers from the “top third” of graduating classes, an outcome that is frequently cited as being the case in high-performing nations such as Finland. Now, it bears noting that “attracting better people," like “improving teacher quality," is a policy goal, not a concrete policy proposal – it tells us what we want, not how to get it. And how to make teaching more enticing for “top” candidates is still very much an open question (as is the equally important question of how to improve the performance of existing teachers).

    In order to answer that question, we need to have some idea of whom we’re pursuing – who are these “top” candidates, and what do they want? I sometimes worry that our conception of this group – in terms of the “top third” and similar constructions – doesn’t quite square with the evidence, and that this misconception might actually be misguiding rather than focusing our policy discussions.

  • Comparing Teacher Turnover In Charter And Regular Public Schools

    Written on August 11, 2011

    ** Also posted here on “Valerie Strauss’ Answer Sheet” in the Washington Post

    A couple of weeks ago, a new working paper on teacher turnover in Los Angeles got a lot of attention, and for good reason. Teacher turnover, which tends to be alarmingly high in lower-income schools and districts, has been identified as a major impediment to improvements in student achievement.

    Unfortunately, some of the media coverage of this paper has tended to miss the mark. Mostly, we have seen horserace stories focusing on fact that many charter schools have very high teacher turnover rates, much higher than most regular public schools in LA. The problem is that, as a group, charter school teachers are significantly dissimilar to their public school peers. For instance, they tend to be younger and/or less experienced than public school teachers overall; and younger, less experienced teachers tend to exhibit higher levels of turnover across all types of schools. So, if there is more overall churn in charter schools, this may simply be a result of the demographics of the teaching force or other factors, rather than any direct effect of charter schools per se (e.g., more difficult working conditions).

    But the important results in this paper aren’t about the amount of turnover in charters versus regular public schools, which can measured very easily, but rather the likelihood that similar teachers in these schools will exit.

  • If Gifted And Talented Programs Don't Boost Scores, Should We Eliminate Them?

    Written on August 8, 2011

    In education policy debates, the phrase “what works” is sometimes used to mean “what increases test scores." Among those of us who believe that testing data have a productive role to play in education policy (even if we disagree on the details of that role), there is a constant struggle to interpret test-based evidence properly and put it in context. This effort to craft and maintain a framework for using assessment data productively is very important but, despite the careless claims of some public figures, it is also extremely difficult.

    Equally important and difficult is the need to apply that framework consistently. For instance, a recent working paper from the National Bureau of Economic Research (NBER) looked at the question of whether gifted and talented (GT) programs boost student achievement. The researchers found that GT programs (and magnet schools as well) have little discernible impact on students’ test score gains. Another recent NBER paper reached the same conclusion about the highly-selective “exam schools” in New York and Boston. Now, it’s certainly true that high-quality research on the test-based effect of these programs is still somewhat scarce, and these are only two (as yet unpublished) analyses, but their conclusions are certainly worth noting.

    Still, let’s speculate for a moment: Let’s say that, over the next few years, several other good studies also reached the same conclusion. Would anyone, based on this evidence, be calling for the elimination of GT programs? I doubt it. Yet, if we applied faithfully the standards by which we sometimes judge other policy interventions, we would have to make a case for getting rid of GT.

  • In The Classroom, Differences Can Become Assets

    Written on August 4, 2011

    Author, speaker and education expert Sir Ken Robinson argues that today’s education system is anachronistic and needs to be rethought. Robinson notes that our current model, shaped by the industrial revolution, reveals a "production line" approach: for example, we group kids by "date of manufacture", instruct them "by batches", and subject them all to standardized tests. Yet, we often miss the most fundamental questions - for example, Robinson asks, "Why is age the most important thing kids have in common?"

    In spite of the various theories about the stages of cognitive development (Piaget, etc.), it is difficult to decide how to group children. Academically and linguistically diverse classrooms have become a prevalent phenomenon in the U.S. and other parts of the world, posing important challenges for educators whose mission is to support the learning of all students.

    It’s not only that children are dissimilar in terms of their interests, ethnicity, social class, skills, and other attributes; what’s even more consequential is that human interactions are built on the basis of those differences. In other words, individuals create patterns of relations that reflect and perpetuate social distinctions.



Subscribe to Education Research


This web site and the information contained herein are provided as a service to those who are interested in the work of the Albert Shanker Institute (ASI). ASI makes no warranties, either express or implied, concerning the information contained on or linked from The visitor uses the information provided herein at his/her own risk. ASI, its officers, board members, agents, and employees specifically disclaim any and all liability from damages which may result from the utilization of the information provided herein. The content in the Shanker Blog may not necessarily reflect the views or official policy positions of ASI or any related entity or organization.