One promise of open data is its ability to inform citizens and consumers about the quality of local services. At the Gov 2.0 Summit yesterday the US Department of Health and Human Resources announced it was releasing data on hospitals, nursing homes and clinics in the hopes that developers will create applications that show citizens and consumers how their local hospitals stacks up against others. In short, how good, or even how safe, is their local hospital?
In Canada we already have some experience with this type of measuring. The Fraser Institute publishes an annual report card of schools performance in Alberta, BC, Ontario and Washington. (For those unfamiliar with the Fraser Institute it is a right-wing think tank based in Vancouver with, shall we say, dubious research credentials but strong ideological and fundraising goals.
Perhaps unsurprisingly, private schools do rather well in the Fraser Institute’s report card. Indeed it would appear (and I may be off by one here) that the t0p 18 schools on the list are all private. This does support a narrative that private schools are inherently better than state run schools that would be consistent with the Fraser Institute’s outlook. But, of course, that would be a difficult conclusion to sustain. Private schools tend to be populated with kids from wealthy families with better educated parents and have been given a blessed head start in life. Also, and not noted in the report card, is that many private schools are comfortable turfing out under-performing or unruly students. This means that the “delayed advancement rate,” one critical metric of a schools performance, is dramatically less impacted than a public school that cannot as easily send students packing.
Indeed, the Fraser Institute’s report card is rife with problems, something that teachers unions and, say, equally ideological but left-oriented think tanks like the Centre for Policy Alternatives are all too happy to point out.
While I loath the Fraser Institute’s simplistic report card and think it is of dubious value to parents I do like that they are at least trying to give parents some tool by which to measure schools. The notion that schools, teachers and education quality can’t be measured, or are too complicated to measure is untenable. I suspect few parent – especially those in say, jobs where they are evaluated – believe it. Nor does such a position help parents assess the quality of education their child is receiving. While they understand, may be sympathetic to or even agree that this is a complicated issue it seems clear based on the success of Ontario’s school locator that many parents want and like these tools.
Ultimately the problem here isn’t the open data (despite what critics of the Ontario Government’s school comparison website would have you believe). Besides, are we now going to hide or suppress data so that parents can’t assess their kids schools? Nor is the problem school report cards per se. If anything is the problem it is that the Fraser Institute has had the field all to itself to play in. If teachers groups, other think tanks, or any other group believes that the Fraser Institute’s report cards are not too crude, why not design a better one? The data is available (and the government could easily be pressured to make more of it available). Why don’t teacher’s groups share with parents the metrics by which they believe parents should evaluate and compare schools? What this issue could use is some healthy competition and debate – one that generated more options and tools for parents.
The challenge for government is to make data more easily available. By making educational data more accessible, less time, IT skills and energy is needed to organize the data and precious resources can instead be focused on developing and visualizing the scoring methodology. This is certainly seems to be Health and Human Services approach: lower transaction costs, galvanize a variety of assessment applications and foster a healthy debate. It would be nice if ministries of education in Canada took a similar view.
But the second half of that challenge is also important, and groups outside of government need to recognize they can have a role, and the consequence of not participating. The mistake is to ask how to deal with groups like the Fraser Institute that use crude metrics, instead we need to encourage more groups and encourage our own organizations to contribute to the debate, to give it more nuance, and create better tools. Leaving the field to the Fraser Institute is a dangerous strategy, one that will serve few people. This is even more the case since in the future we are likely to have more, not less data about education, health and a myriad of other services and programs.
So, the challenge for readers is – will your organization participate?
eaves.ca 
The best approach I’ve seen to compare teachers (rather than schools).
http://www.latimes.com/news/local/teachers-investigation/
Wayne – I saw that piece via Tim O’Reilly on Slashdot a few weeks ago and thought it was fascinating. Agree that maybe teachers are the right metric…
I believe that teachers would argue that there simply aren’t metrics that show parents how schools are doing, and I would tend to agree. When I think of the stories I hear from teachers about making a difference in students’ lives, nothing I can think of in those situations is measurable.
While I do appreciate your point, the situation is quite simple and it is the Fraser Institute’s goal to obfuscate in their favour. They don’t care and don’t need to write accurate criticisms of our perfectly adequate school systems. They simply need to create doubt,
George Lakoff’s “Don’t Think of an Elephant” is important for understanding how political discourse is framed. It also clears up any uncertainties some may have had around our so called BC “Liberal” Gov’t. But more specifically it talks about school testing in this quote below:
Shepsil – I think we agree on lots but I don’t think the situation is that simple at all. I happen to believe that schools or possibly more specifically (as Wayne points out) teachers can fail. I’m not sure that should be the focus or point of using data to assess them but claiming otherwise feels deeply problematic from an accountability perspective. It seems like the end conclusion of your point is that we either can’t or shouldn’t measure or scorecard, schools. I think many (and possibly a strong majority) of parents would find that answer unacceptable and so I’m not sure such arguments will prove persuasive. I’m just trying to find an alternative solution.
David, do you have any suggestions for what to measure? My one idea would be to set an individual benchmark for each school and use that for future measurements rather than as a comparison between schools, but I’m not sure that’s what you are hoping for.
David, do you have any suggestions for what to measure? My one idea would be to set an individual benchmark for each school and use that for future measurements rather than as a comparison between schools, but I’m not sure that’s what you are hoping for.
David – As I see it, the Fraser Institute has no real credibility. Which you have only alluded to. In a civil society, it is important to stand up to bullies who try to discredit existing social systems for their own benefit. I can’t do this discussion justice without, once again, bringing in George Lakoff’s (also in Huff Post & YouTube) seminal works in cognitive science & linguistics. In his 2008 book, “The Political Mind – Why You Can’t Understand 21st-Century American Politics with an 18th-Century Brain”, he explains that we have entered:
He and his colleagues have shown thru “emphirical” evidence that as humans, among other things, our brains are wired to be empathetic. The last paragraph in the book goes on to say:
Keep up the good work.
David – As I see it, the Fraser Institute has no real credibility. Which you have only alluded to. In a civil society, it is important to stand up to bullies who try to discredit existing social systems for their own benefit. I can’t do this discussion justice without, once again, bringing in George Lakoff’s (also in Huff Post & YouTube) seminal works in cognitive science & linguistics. In his 2008 book, “The Political Mind – Why You Can’t Understand 21st-Century American Politics with an 18th-Century Brain”, he explains that we have entered:
He and his colleagues have shown thru “emphirical” evidence that as humans, among other things, our brains are wired to be empathetic. The last paragraph in the book goes on to say:
Keep up the good work.
David, you wrote:
“If anything is the problem it is that the Fraser Institute has had the field all to itself to play in. If teachers groups, other think tanks, or any other group believes that the Fraser Institute’s report cards are not too crude, why not design a better one?”
David Johnson and the C.D. Howe Institute have done exactly this. Rather than clumping together all schools irrespective of the composition of students, as the Fraser Institute does, Johnson adjusts school scores to reflect the socio-economic background of each school’s students. This adjustment provides something closer to an apples-for-apples comparison.
Johnson compares schools within Alberta, BC and Ontario. A recent example of his work is here: http://www.cdhowe.org/pdf/ebrief_100.pdf
As a parent and a citizen, I want this kind of data to be available and usable; but clearly Johnson, unlike the Fraser Institute, is making it meaningful, too.
Erik – you’ve made my day! I had no idea… Thank you for finding this.
Maybe I’ll reach out to C.D. Howe. Publishing this as a PDF is reaching all the wrong people and not the ones who need/want it. Would be nice to see this as a website… indeed this is one area where the Fraser Institute is outpacing them.
Greetings David et al,
I few comments if I may on a topic that is obviously of interest to me.
First, I would happily see many more report cards on schools and have many times encouraged union leaders and other interested parties to join in with their own evaluations.
Second, the alternatives to the Fraser Institute report card include both the CD Howe efforts as well as the report cards on schools in the Atlantic provinces published since 2004 by the Atlantic Institute for Market Studies.
While these two should be of some interest to educators, their weakness, in my view, is that, by design, they expect less of schools that serve kids with challenges of one kind or another.
The promise of “government schools” was, I think, to ensure that all kids–not just those of wealthy, motivated parents–acquire the basic skills that they will need to continue their education and go on to lead happy and prosperous lives in this great country of ours. To say, then, that a school is doing well even though half its kids are failing–as the CD Howe and AIMS studies are capable of doing–is selling a whole bunch of kids short.
I agree with the inimitable Prof. Bob Crocker of Memorial University when he remarked in reference to the AIMS report cards that he was not yet ready to give up on the poor.
Third, if any one has knowledge of other provincial results data that is generally objective, comparable, annually-generated, centrally available, and of interest to parents, please let me know as I would be happy to include them in future editions of the report card.
Finally, I think the Los Angeles Times effort, produced by Richard Buddin, a senior economist at RAND Corporation working for the Times as an independent contractor is a nicely done piece of work. It will be a long time before you will see its equivalent in Canada. Too bad.
Cheers,
Peter Cowley
Director, School Performance Studies
THE FRASER INSTITUTE
(Co-author of the Fraser Institute’s Report Cards series.)
Good article, thanks David. I think that many of the comments as well as your own points underline the fact that it’s very complicated to find the right way to measure performance in education, health, or elsewhere. Any one corporate style dashboard is only going to service a portion of interested parties’ questions, so anything that promotes more diversity than this is going to help. The promise of many people getting access to the data is that they can all adopt different approaches to measuring performance. Or perhaps better still for the more inquisitive minded, provide consumers with tools to build their own measurements.
I work often work with government agencies that are concerned about sharing data because of the risk that it will be misused in some way. Data collections often vary across states and even at local levels, and someone trying to join it all together can easily neglect significant differences in meaning. Making sure it’s all correctly harmonized is a nice goal, but often a very big undertaking. One way to counteract this kind of accidental misuse is for organizations to make their own representations of the data. This can help engage local communities on issues that matter most to them, as well as act as one authoritative source that people can refer to.
Cheers,
Don McIntosh