Skip to content

What’s the Story?

September 9, 2010

Several years ago, I was watching the evening news and some reporters were covering a story about a new gasoline tax or surcharge.  I wish I could recall the details now, but what struck me was the reporting.  The main story included a summary of the facts – who did what, when, where and why, followed by some expert analysis by people in government and industry.  That whole report lasted about a minute.  The basic analysis seemed to minimize fears that people might have about the negative impacts of the new tax.

Then, a second reporter picked up the story, and what do you suppose was happening?  Interviews with random motorists.  Where?  At the gas station, of course.  How many times have you seen that report in your life?  (Sort of like the clichéd shots of overweight people viewed from the chest down whenever there are news reports about obesity).   For the next two or three minutes, they ran a package that had been edited together to show a series of apparently uninformed motorists, who didn’t know the details of the report and hadn’t heard any of the analysis, offering their opinions about what this new tax or surcharge was going to mean for drivers.  It’s almost needless to say, but the average person on the street was more inclined to take a negative, worried view, and to offer predictions that did not match the experts.

Well, there you have it.  Information, expert analysis, and random uninformed guesswork – the complete report.  I offer this little anecdote as a reminder that sometimes the story we see or read in the news is the story the media want to tell; for their purposes, that story doesn’t always need to be subjected to deeper analysis or filled in with enough background to make sense of the present or future.  By virtue of the fact that you’re standing at the gas pump at the right time, you can have as much air time as the economist and the policy analyst.

Now, bringing this back to education reporting, the story of the day is about using value-added measures for teacher evaluation.  How often have you seen a story that runs like this:

“Value-added measurement is a technique that some people say will allow for better evaluation of teachers by using their students’ test scores.  Here are some parents and politicians who like the idea.  Teachers and unions don’t like it.  Some studies say VAM can be a useful measure of teacher quality, and some experts disagree.”

It’s certainly balanced, and accurate, but is it complete?  I found myself quoted in one of those types of stories this week, and was reminded once again of the gas station report.  Sharon Noguchi, the reporter who interviewed me, was very professional and gave me ample opportunity to describe my experience and my views, by phone and by email.  She asked me follow-up questions, and even pushed back at some of my generalizations and forced me to clarify my views.  Her story summarizes various viewpoints on the issue, and at least where I’m included, I can say that my views were presented accurately (in part because Ms. Noguchi double-checked before going to print).  We spoke for about half an hour to produce those two little parts of the story, so naturally I would have liked to have more of that information make it into the story.

But what’s still bugging me is that I offered a number of sources of better information, and a number of important findings about value-added measurements.  While I think I know teaching better than the folks at the gas pump know energy and fiscal policy, I was actually encouraging Ms. Noguchi not to rely on me too much.  I suggested that she look into the joint statement (often cited on this blog) of the NCME/APA/AERA statement.  They offer warnings about test misuse, reiterated with more specific reference to VAM by the National Academies last year and the Economic Policy Institute this year.  The consensus is clear:  using VAM with state tests has too many flaws to be used as a reliable measure of individual teacher quality.

I practically begged Ms. Noguchi to report findings from the real experts on educational measurement and statistical analysis, and to her credit, she did talk to one researcher to add that perspective.   Still, I keep wondering why I see so little reporting on the weight of the evidence, so little investigation into the claims of VAM advocates.  Why is the story about everyone’s opinions, instead of focusing on the ones grounded in a deep understanding of research, psychometrics, and statistics?

Would it be too much to ask for a reporter to present the facts this way: “While some parents say they’d like to see teacher evaluations based on VAM, all of the leading educational research organizations say that such an analysis would likely be flawed or misleading.”  Too much news reporting presents our story as a debate of contrary, but equally valid opinions, rather than a debate in which one side is factually challenged, while the other can rely on the support of so many more expert opinions.

“Well, there you have it, Bob.  Dr. Jane Smith of the Energy and Economics Institute has just explained why this surcharge will have minimal impact on motorists.  Now, let’s send it out to Nick at the gas station, where he’ll ask this guy in the black Chevy what he thinks of today’s news.”

8 Comments leave one →
  1. Jan Dietzgen permalink
    September 9, 2010 9:48 am

    Dear David,

    Bravo for the metaphor! I also read Ms. Noguchi’s article. I immediately wanted to send her your recent postings on VAM. Now I hope that you have sent your Blog to the newspaper because it clarifies the frustration of many of us. In the same newspaper a week before Tom Friedman heralded the VAM, Michelle Rhee’s and Joe Klein’s use of VAM to rid their systems of those bad teachers who are the demise of our educational system. (I wanted to send Mr. Friedman your, “How do we account for the new principal” article and add, “How do we account for “cockroaches in the room sans whiteboards or a computer, built for 20 desks, holding 39”) The question is how do we inform the people on at the gas station the pitfalls of these tests. In my district testing is so pervasive that in the sophomore year, students take pre and post released CAHSEE tests in September and as their English Final. Add to that a district online pre and post test, the STAR tests and the real CAHSEE, meaning that ten graders take eleven standardized tests. I have always suggested for the proponents of testing: Take the Test with two colleagues, compare answers, and then check the official answers. I predict the question of validity and the level of testing will become apparent.

    • David B. Cohen permalink*
      September 9, 2010 9:41 pm

      Jan – thanks for the kind words, and I’ll your question back to you: how do we inform the people at the gas station – or the coffee shop, or the PTA, or anywhere else? I’m doing my best, and hope you are too. That’s part of the reason that ACT was founded – to get the story out from the classroom perspective. Hope you’re doing what you can too – that summary of testing you gave could make a pretty compelling story. The idea of using a CAHSEE as a final exam really bothers me, not only in terms of educational practices, but also in terms of the message it sends to students.

  2. September 9, 2010 10:43 am

    Here’s what kills me: Genuine Ed Policy experts–that is, people with considerable experience, knowledge and PhDs in Ed policy–claiming that while, yeah, VAM is seriously and deeply flawed, it’s the best thing we’ve got so we should use it. Collateral damage? (read: good teachers randomly losing their jobs) Just an unpleasant side effect.

    Quote from School Finance 101: The new rhetoric argues that VAM is still the “best available” method for evaluating teacher “performance.” Let me point out that if the “best available” automobile burst into flames on every fifth start, I think I’d walk or stay home instead. I’d take pretty significant steps to avoid driving.

    • David B. Cohen permalink*
      September 9, 2010 9:43 pm

      Thanks as always, Nancy, for reading, replying, and adding to the mix. California could sure use a Bruce Baker in the blogosphere. For those who don’t know his blog, check out:


  1. Reflections on recent posts « School Finance 101
  2. Appeal to Education Writers: Evaluate Opinions « InterACT
  3. Turning the Tables: VAM on Trial « InterACT
  4. VAM on Trial (Re-posted from InterACT) | DAVID B. COHEN

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: