Tag Archives: census

Lies, Damned Lies, and Open Data

I have an article titles Lies, Damn Lies and Open Data in Slate Magazine as part of their Future Tense series.

Here, for me, is the core point:

On the surface, the open data movement was about who could access and use government data. It rested on the idea that data was as much a public asset as a highway, bridge, or park and so should be made available to those who paid for its creation and curation: taxpayers. But contrary to the hopes of some advocates, improving public access to data—that is, access to the evidence upon which public policy is going to be constructed—does not magically cause governments’, and politicians’, desire for control to evaporate. Quite the opposite. Open data will not depoliticize debate. It will force citizens, and governments, to realize how politicized data is, and always has been.

The long form census debacle here in Canada was, I think, a great example of data getting politicized, and was really helped clarify my thinking around this. This piece has been germinating since then, but the core thesis has occasionally leaked out during some of my talks and discussion. Indeed, you can see me share some of it during the tail end of my opening keynote at the Open Knowledge Foundation International Open Data Camp almost three years ago.

Anyways, please hop on over to Slate and take a look – I hope you enjoy the read.

Making StatsCan Data Free: Assessing the Cost

Regular readers of my blog will know that I’ve advocated that StatsCan’s data – and particularly its Census data – should be made open (e.g. free, unlicensed, and downloadable in multiple formats). Presently, despite the fact that Canadian tax dollars pay to collect (a sadly diminishing amount, and quality of,) data, it is not open.

The main defense I hear to why StatsCan’s data should not be free is because the department depends on the revenue the data generates.

So exactly how much revenue are we talking about? Thanks to the help of some former public servants I’ve been able to go over the publicly available numbers. The basic assessment – which I encourage people to verify and challenge – turns out not to be a huge a number.

The most interesting figure in StatsCan’s finances is the revenue it generates from its online database (e.g. data downloaded from its website). So how much revenue is it? Well in 2007/2008, it was $559,000.

That’s it. For $559,000 in lost government revenue Canadians could potentially have unlimited access to the Statscan census database their tax dollars paid to collect and organize. I suspect this is a tiny fraction of the value (and tax revenue) that might be generated by economic activity if this data were free.

Worse, the $559,000 is not profit. From what I can tell it is only revenue. Consequently, it doesn’t factor in collection costs StatsCan has to absorb to run and maintain a checkout system on its website, collect credit card info, bill people, etc… I’m willing to bet almost anything that the cost of these functions either exceed $559,000 a year, or come pretty close. So the net cost of making the data free could end up being a less.

StatsCan makes another $763,000 selling Statistics Canada publications (these are 243 data releases of the 29 major economic indicators StatsCan measures and the 5 census releases it does annually – in short these are non-customized reports). So for $1,422,000 Canadians could get access to both the online data statscan has and the reports the organization generates. This is such laughably (or depressingly) small number it begs the question – why are we debating this? (again this is revenue, not profit, so the cost could be much lower)

Of course, the figure that you’ll often hear cited is $100M in revenue. So what accounts for the roughly 100x difference between the above number and the alleged revenue? Well, in 2007/08 StatsCan did make $103,155,000 but this was from value added (e.g. customized) reports. This is very, very different product than the basic data that is available on its website. My sources tell me this is not related to downloaded data.

I think we should concede that if the entire StatsCan’s database were made open and free it would impact some of this revenue. But this would also be a good thing. Why is this? Let’s break it down:

  1. Increase Capacity and Data Literacy: By making a great deal of data open and free, StatsCan would make it easier for competitors to enter the market place. More companies and individuals could analyze the country’s census and other data, and so too could more “ordinary” Canadians than ever would be able to access the database (again, that their tax dollars paid to create). This might include groups like senior high school and university students, non-profits and everyday citizens who wanted to know more about their country. So yes, Statscan would have more competitors, but the country might also benefit from having a more data literate population (and thus potential consumers).
  2. Increase Accessibility of Canadian Data to Marginalized Groups: An increase in the country’s analysis capacity would drop the price for such work. This would make it cheaper and easier for more marginal groups to benefit from this data – charities, religious groups, NGO’s, community organizations, individuals, etc…
  3. Improve Competitiveness: It would also be good for Canadian competitiveness, companies would have to spend less to understand and sell into the Canadian market. This would lower the cost of doing business in Canada – helpful to consumers and the Canadian economy.
  4. StatsCan would not lose all or even most of its business: Those at StatsCan who fear the organization would be overwhelmed by a more open world should remember, not all the data can be shared. Some data – particularly economic data gathered from companies – is sensitive and confidential. As a result there will be some data that StatsCan retains exclusive access to, and thus a monopoly over analysis. More importantly, I suspect that were Statscan data made open the demand for data analysis would grow, so arguably new capacity might end up being devoted to new demand, not existing demand.
  5. It will Reduce the Cost of Government: Finally, the crazy thing about StatsCan is that it sells its data and services to other Ministries and layers of government. This means that governments are paying people to move tax payer money between government ministries and jurisdictions. This is a needless administrative costs that drives up everybody’s taxes and poorly allocates scarce government resources (especially at the local level). Assuming every town and city in Canada pays $50 – 1000 dollars to access statscan data may not seem like much, but in reality, we are really paying that, plus their and StatsCan’s staff time to manage all these transactions, enforce compliance, etc… all of which is probably, far, far more.

So in summary, the cost to Canada of releasing this data will likely be pretty marginal, while the benefits could be enormous.

At best, if costs half a million dollars in forgone revenue. Given the improved access and enormous benefits, this is a pittance to pay.

At worst, StatsCana would lose maybe 20-30 million – this is a real nightmare scenario that assumes much greater competition in the marketplace (again, a lot of assumptions in this scenario). Of course the improved access to data would lead to economic benefits that would far, far, surpass this lost revenue, so the net benefit for the country would be big, but the cost to StatsCan would be real. Obviously, it would be nice if this decline in revenue was offset by improved funding for StatsCan (something a government that was genuinely concerned about Canadian economic competitiveness would jump at doing). However, given the current struggles Statscan faces on the revenue front (cuts across the board) I could see how a worse case scenario would be nerve wracking to the department’s senior public servants, who are also still reeling from the Long Form Census debacle.

Ultimately, however, I think the worse case scenario is unlikely. Moreover, in either scenario the benefits are significant.

Bonus Material:

Possibly the most disconcerting part of the financial reports on StatsCan on Treasury Board’s website was the stakeholder consultation associated with access to statscan’s database. It claimed that:

Usability and client satisfaction survey were conducted with a sample of clients in early 2005. Declared level of satisfaction with service was very high.

This is stunning. I’ve never talked to anyone who has had a satisfactory experience on StatsCan’s website (in contrast to their phone support – which everyone loves). I refer to the statscan site where the place where what you want is always one click away.

I’m willing to bet a great deal that the consultations were with existing long term customers – the type of people that have experience using the website. My suspicion is that if a broader consultation was conducted with potential users (university students, community groups, people like me and you, etc…) the numbers would tank. I dare you to try to use their website. It is virtually unnavigable.

Indeed, had made its website and data more accessible I suspect it the department would engage Canadians and have more stakeholders. This would have been the single most powerful thing it could have done to protect itself from cuts and decisions like the Long Form fiasco.

I know this post may anger a number of people at Statscan. I’m genuinely sorry. I know the staff work hard, are dedicated and are exceedingly skilled and professional. This type of feedback is never flattering – particularly in public. It is because you are so important to the unity, economy and quality of life in our country that it is imperative we hold you to the highest possible bar – not just in the quality of that data your collect (there you already excel) but in the way you serve and engage Canadians. In this, I hope that you get the support you need and deserve.

The best moment in Canadian democracy in 2010?: the census debate

Over at Samara, my friend Alison Loat is asking people to answer the question “What was the best moment in Canadian democracy in 2010?” In what I think was a good decision, they’ve defined the terms pretty broadly, stating:

The moment could be one that took place inside or outside of Parliament or other legislative chambers.  It could have happened at the federal, provincial, territorial or municipal level.  It could include any number of things, such as an election with a historic turnout, a stimulating public debate, a rally or protest, a critical piece of news analysis, the creation of a new digital application, or an important Parliamentary motion or decision.

If you’ve got an idea I encourage you to hear over there and write it up and submit it! The Samara people are great and are up to good work, so definitely worth checking out.

I’ve got one answer the question myself – what follows is my write up. I think I may even have one more in me… but here’s my first effort:

The Census Debate as Canada’s 2010 democratic moment.

In a functioning democracy disagreement is necessary and healthy. But at its core there most be some basic agreement – some shared understanding of who we are, as a people and as a society. This shared understanding not only serves as the basic facts that must inform our debates but also the basis of our shared identity that keeps us together even when we disagree.

This is why the census is so important, and why it is my choice for the best moment in Canadian democracy for 2010. The census binds us together by creating a shared understanding of who we are. Even the most marginalized Canadians stand up and are counted and thus can be reflected and heard in our national discourse.

That’s why at a time when Canadian political coverage tries to cleave the country’s citizens into different, competing groups – rural versus urban, French versus English, left versus right – I think the best moment in Canadian Democracy was seeing over 500 groups including all levels of government, non-profits from across the country, business organizations, rural communities, and virtually all the major religious organizations come together and challenge the government with one voice.

What a great democratic moment that so many organizations, that often disagree on so many issues, can collectively agree on a core shared interest: that a functioning democracy and an effective government is built on a foundation of some basic information about who we are. Even more so when the government tried to make the decision in secret, announcing it quietly on a friday, during a long weekend in the middle of summer.

The decision and the process surrounding it may be one of the year’s darkest moments for Canadian democracy but the country’s reaction was definitely one of our brightest.

How you know a government is broken

Last Friday Gloria Galloway and Bill Curry ran an excellent piece about how the government’s promise to strengthen Canada’s access-to-information laws is now five years old.

It is of course all so laughable it is sad. Here we have an issue that the public is universally supportive of – making government more transparent and accountable – and yet the government contends the issue requires extensive consultation. And so… no action.

Meanwhile, on issues to which the public is almost universally opposed – for example the long form census – the government acts without consultation, without evidence and in the dead of night, hoping that no one will notice.

Again, it would be laughable if the implications weren’t so serious. It’s also a big reversal of what should have been and maybe the clearest sign yet this government is broken.

And it didn’t have to be this way. Looking back at the Conservative’s 2006 election platform under the header “Strengthen Access to Information legislation” The government promised it would (this is verbatim)

  • Implement the Information Commissioner’s recommendations for reform of the Access to Information Act. Give the Information Commissioner the power to order the release of information.
  • Expand the coverage of the act to all Crown corporations, Officers of Parliament, foundations, and organizations that spend taxpayers’ money or perform public functions.
  • Subject the exclusion of Cabinet confidences to review by the Information Commissioner. Oblige public officials to create the records necessary to document their actions and decisions.
  • Provide a general public interest override for all exemptions, so that the public interest is put before the secrecy of the government.
  • Ensure that all exemptions from the disclosure of government information are justified only on the basis of the harm or injury that would result from disclosure, not blanket exemption rules.
  • Ensure that the disclosure requirements of the Access to Information Act cannot be circumvented by secrecy provisions in other federal acts, while respecting the confidentiality of national security and the privacy of personal information.

How many of these promises have been implemented? To date, only one (the one that is italicized)

As an aside, take a look at that platform. Guess what isn’t mentioned once: The long form census.

One of the great pledges of the Conservative government was that they were going to make government more accountable and more transparent. So far, when it comes to managing information – the collective documents our tax dollars paid to create – today our government is more opaque, more dumb and less inspiring to Canadians than it has ever been. For a government that was supposed to restore Canadians confidence in their country, it has been a sad decline to observe.

Census Update: It's the Economy, Stupid

Yesterday during a press conference newly minted House leader John Baird announced “The next few months will be sharply focused on Canadians’ No. 1 priority: jobs and the economy… The economic recovery remains fragile and it is increasingly clear that we are not out of the woods yet.”

Fantastic news.

I just hope someone sends Industry Minister Tony Clement the memo.

The effects and impacts of ending the mandatory long form census continues to spill out with a number of Canada’s most senior business and economic leaders pointing out how the decision will negatively impact the economy and… job growth.

First, there was Bank of Canada Governor Mark Carney (voted one of the most influential people in the world by Time Magazine) noting that the bank relies on data found in the mandatory long form to assess the economy and, presumably, to inform decisions on interest rates and other issues. The bank’s capacity to make informed decisions has now been compromised – not exactly a win for jobs or the economy.

As an interesting side note, Carney goes on to say that this may cause the bank to have to supplement StatsCan’s research with its own. Expect to hear more and more statements like this from Government agencies (which are still allowed to talk to the press) as more and more ministries and agencies get plunged into the dark regarding what is going on in the country and are no longer able to assess programs and issues they’ve been tasked to monitor. Various arms of the government (and thus you, taxpayer) will be spending 10s if not 100s of millions to pay for Industry Minister Clement’s mistake.

Then, in the same Globe article in which Carney makes these statements, Roger Martin, dean of the Rotman School of Management notes that ending the long form census hampers Canadian companies capacity to both compete globally and boost productivity. More damning, and further echoing arguments I’ve been making here, he states it will prevent Canadians from having “a sophisticated economy that uses information to its best.” Unkind words from one of the world’s recognized business leaders.

Sadly, it doesn’t end there. The always excellent Stephen Gordon lists the emerging academic literature chronicling the havoc the demise of the long form census is about to wreck. Especially relevant is “The Importance of the Long-Form Census to Canada” by UBC economists David Green and Kevin Milligan. Interestingly, it turns out that the Canadian Mortgage and Housing Corporation uses long form data to fulfill its legislative mandate, and also by local governments and private sector actors to learn about trends in housing. Something that might be of interest to those concerned about the economy and jobs given Canada is rumored to possible have a housing bubble.

Still more damning is how Green and Milligan show the mandatory long form serves as the foundation for the Labour Force Survey (LFS) from which we derive unemployment levels. Compromising the long form survey has, in short, compromised our ability to assess how many Canadians actually have jobs, something that, if you really believed Canadians felt the economy and jobs were the number 1 priority, your government should care about measuring accurately.

Maybe John Baird will sit down with Tony Clement and the Prime Minister and explain to them how, if the economy and jobs are priority 1 then perhaps the government should rethink its decision on the long form census.

Just don’t hold your breath. Instead, do write another email or letter to your local MP. Our country’s economic recovery and competitiveness is being eroded by a government either too dumb to understand the implications of its decision and too stubborn to admit a mistake. Those of us who will be paying the price should remind them of how they can best serve their own priorities.

Least Shocking Headline: Our Government Ignored Data about the Census

Since the resignation of Deputy Minister Munir Sheikh and his public repudiation of voluntary long form census it has become clear that Industry Minister Clement has – at best – been misleading the public about the advice he received from statscan.

Now more evidence has emerged showing how the Government was well informed about the significant problems their plan would create. As Steven Chase of the Globe and Mail writes:

For instance, the real 2006 census long-form found that renting households as a percentage of the population in Canada had dropped by 3.08 percentage points from the 2001 census.

But when the Statscan study simulated the results of a voluntary 2006 long-form – which reflect the lower response rates expected in optional surveys – it got a markedly different answer. Calculations instead indicated that rented dwellings in Canada as a share of the population declined by 8.07 percentage points from 2001.

The difference – nearly five percentage points – suggests a voluntary survey in 2006 would have massively undercounted renting households.

So a mere 150% difference. Which, of course, might affect how every city in Canada considered zoning issues and adjust policies around the housing and rentals stock.

Again, if the Government wants to scrap the long-form census, that’s their prerogative. And I suppose we can’t be surprised that a government that wants less data and information to inform decisions would ignore data that showed them the negative consequence of their proposal. I mean, when you’ve already decided evidence doesn’t matter in crime policy, health policy and a myriad of other issues, you aren’t suddenly going to decide the collecting evidence is important…

But if that is your conclusion, stick with it! Don’t lie to me and the Canadian public and claim it won’t have a dramatic impact on the quality of data the Government collects or an impact on how policy and services for Canadians are affected.

And, of course, given how sensitive the decision is, and how it will cascade down and impact businesses, non-profits, local, provincial and federal government decision making, I wish we’d had a chance to debate the merits of it before the decision was made. Who knows, such a debate might not have just saved the long form census, it might have save government a 10-point decline in the polls.

Census Update and other chuckles

Sorry for the lack of posts this week, blog was offline for a bit. (For geeks out there, I now have a company managing my blog for me and we we’re moving from a shared hosting service to a virtual private server – I should have less down time in the future – very excited).

Sadly, in that time there have been a bunch of fascinating developments on the census. As some of you may be aware a new poll by EKOS emerged today that has the Liberals and Conservatives dead even. More interesting however is how the census is playing a key role in the shift:

In seeking an explanation for these movements, we need look no further than the government’s ill-received decision to end the mandatory long form census. Not only does the shift of the highly educated support this conclusion, but a direct question on public approval for this decision provides compelling evidence that this move precipitated the current woes that the Conservative Party now faces.

When asked whether they felt that the privacy intrusion of the census justified a voluntary census or whether the lack of representativeness would cost us vital data, a clear majority of the public (56%) picked the latter (compared 26% who felt the mandatory long form was a violation of privacy). Even among Tory supporters, this appeal is not selling and there is an overwhelming lean to disapproval in the rest of the spectrum. Opposition to this decision is strongest among the university educated.

Of course, one of the retorts from pundits in favour of scrapping the long form census has been that only a few people care about this issue, it won’t matter in the medium term and it certainly won’t impact any election. For example:

Two things: I still standby my thesis that I believe that chucking mandatory nature of the long-form is a move to dismantle the welfare state (and that this is a move in the right direction). And two, nobody cares outside of the beehive. It’s the media that is pushing the story outside of the beehive walls propelled by the loud buzz of special interests.

Sigh, I suppose that 56% of  Canadians represent “a special interest.”

For me, both groups (56% and 26%) have legitimate concerns. As such, efforts by those in favour of this decision opposition as “special interest” driven are wrong and, frankly, disingenuous. Happily, they have failed. Indeed, the more these pundits try, the more they seem to make this a wedge issue in favour of those opposed to the decision. Mostly, I just think it would have been nice to have the issue debated before a decision was made.

More interesting has been another effort to defend trashing the long form census. I think Jack Mintz has thoroughly damaged his credibility with a terrible, contradictory and misleading op-ed in the Financial Post. Rather than dive into it, I encourage everyone to wander over to Aaron Wherry’s fantastic (and, unlike this post, short) dismantling of it. He’s already done all the heavy lifting.

Finally, just because I could help but notice the irony… I see that Conservative MP Garry Breitkreuz has an oped in the Mark in which he is worried about the role that the police is taking lobbying to keep the registry alive:

Taxpayers should be incensed at the CACP for co-opting the role of policy-maker. When law enforcement managers try to write the laws they enforce, history has taught us we risk becoming a state where police can dictate our personal freedoms.

I, of course, agree that it is dangerous for the police to get involved in policy debates. I now eagerly away for Garry Breitkreuz to demand that the RCMP own up to the funding of fake “research” in an effort to distort the debate on Insite and harm reduction policies. It would seem that someone at the RCMP, or higher up, doesn’t believe that should happen.

But on further review, maybe we shouldn’t get to excited. Looking at Garry’s website, and specifically, this PDF he’s made available for download, it seems like he’s actually quite keen to have police force members be outspoken about the gun registry as long as they agree with his view.

Ah, hypocrisy. If only he didn’t make it so easy.