A word about numbers

By richard alldritt posted 17-05-2012 14:57

  

As users of official statistics will be aware, more and more government datasets are available, in fairly raw form, on the internet. This allows people to examine for themselves the data used by government.  Have you tried this armchair auditing?  Here is a link to the Treasury COINS database http://data.gov.uk/dataset/coins - not official statistics I should stress.  If you can find a usable statistical message in these databases you clearly do not need much help from government statisticians.  Or could it be that you do need help but have not realised it; and the messages you have inferred are not actually correct.

One user on the COINS website commented  although the raw data is useful for those with the time and skills to be ‘armchair auditors’ it would be wrong for this to become the only form of audit.”  That’s a good point, but not quite the full story.  The analysis of raw data certainly requires time and skills but it also requires a proper understanding of the data.  Are they what they appear to be; are they subject to revisions; are they the latest version; are they incomplete; are there documented weaknesses; for what uses are these statistics generally regarded as useful; are there other datasets that might modify our interpretation. And so on.  This is the reading light for the armchair auditor.  And producing it requires a different range of expertise.

You cannot (nor will you soon be able to) put official data in to a statistical app and get a summary of their strengths and weaknesses in relation to their principal uses.  Perhaps one day.  But if there ever is such an app it will most likely work by accessing the written advice and knowledge of the people who put the statistics together and who do actually know about their strengths and weaknesses, revisions, uses and the rest.

In short, developments in the release of data simply underline the importance of preparing and publishing professional statistical advice. We in the Statistics Authority call this ‘narrative commentary’, and expect to see it alongside a summary of the statistics at the time they are released. This does not mean that the release of detailed management data necessarily needs to wait on the preparation of summary statistics and considered commentary.  But we think it would be helpful to the user to, at least, offer a link from the latest detailed data back to the most recent summary and commentary.

 

Going a little deeper, two types of written advice are needed.  Firstly, descriptive text which sets out the relevant facts about the statistics such as time periods, geography, methods, definitions, etc.  Secondly, narrative text drawing out the main messages from the statistics, explaining the relevance of the statistics in terms of the policy context, stating assumptions about the use that will be made of the statistics and highlighting their strengths and limitations in those contexts.

Among the requirements of the Code of Practice for Official Statistics are that full and frank commentary and analysis should be offered, including:

·         information on the quality and reliability of statistics in relation to potential uses

·         information on methods, procedures and classifications

·         factual information about the policy or operational context

·         presentation of statistics in graphs, tables and maps that enhance clarity, interpretability and consistency.

 

The Statistics Authority has also offered the following advice in relation to the preparation of written commentary:

·         Official statistics only justify their production at public expense when they are used in ways that provide public benefit. Most official statistics require careful explanation. Government statisticians will usually know the strengths and weaknesses of the statistics better than other commentators.

 

·         The cost of supplementing statistics with text and guidance is relatively small, particularly in cases where statistics are produced in a regular series and the guidance is updated at each release. It is likely to be more efficient, in terms of the return on public investment in statistics, to offer written guidance to the user than not to do so.

 

·         Government statisticians share a common training and an ethos of impartiality. Whilst experts outside government are of course capable of writing impartial commentary, they are not under any obligation to do so.

 

·         Guidance to the user is needed most at the moment the statistics are first made public. The news media look to the statistical release to provide an objective account of the statistics. They may regard the absence of such text as indicative of partisan influence. Saying nothing is not necessarily politically neutral. 

 

·         The public has a right to know the statisticians’ understanding of the messages from the statistics, just as they have a right to the data itself. It is common for statisticians in government departments to brief policy colleagues on the substance of the statistics. That knowledge should, as a matter of principle, be shared more widely.

This is an unequivocal manifesto for statistics to be accompanied by helpful guidance. Armchair auditors may want to check that they have that guidance before they start auditing.

 

Richard Alldritt, UK Statistics Authority

 

 

 

3 comments
226 views

Tag

Comments

21-05-2012 12:45

David Walker makes a good point about the word 'narrative'. A set of statistics has. in practice, more than one narrative and the one that it is appropriate for the professional statistician may be seen, by the professional journalist or commentator, as quite narrow. But it is still important. In the case of statistics on violent crime for example, there is the trend iin the figures; the history of changes in the counting rules; the contrast between the police figures and the British Crime Survey; the fact that the victims are mostly from the same demographic as the perpetrators; the relationship between policing strategy and the amount of crime recorded. There is a story here and the person who wants to use the statistics to guide their actions or decisions needs to know it. But there is also a bigger story about what the statistics of violence are telling us about society and about government policy and police effectiveness. The statistician must tread carefully here. There are some factual things that might be said in statistical releases but mostly this bigger narrative is for others. My fear is that the existence of some things that are out of bounds for government statisticians has been taken as grounds for excluding far too much from the narrative.

18-05-2012 15:43

I hope many more people will join this important discussion. The mission to explain - to tell our audience "what is going on" - is at the heart of our professional mission.
The original RSS Charter tells us that the Society was established in 1834 to "collect, arrange, digest and publish facts illustrating the condition and prospects of society in its material, social and moral relations". Sir John Sinclair, oldest of the founding members of the Society, wrote that statistics was "an enquiry into the state of a country for the purpose of [Paul Allin will like this bit....] ascertaining the quantum of happiness enjoyed by its inhabitants and the means of its future improvement."
If official statisticians are to meet the first of the United Nations Fundamental Principles set out to guide them, all official statistics must pass a test of "practical utility". To be useful to any audience, inside or outside government, words and often pictures are needed to bring out the meaning contained within the numbers.

18-05-2012 11:29

Two points in response, Richard:
1. "narrative' requires interpretation and analysis. 'Impartiality' doesn't imply capacity to interpret, I fear. It's questionable whether official statisticians are equipped to do the work of telling the story in the figures a) by reason of temperament and training and b) because of where they sit in the food chain. Because they are official statisticians, the public might regard their interpretation as skewed towards the official line. Wouldn't the interpreters need to be more obviously at arm's length from Whitehall. Take, for example, data around the policing function - a much more complex set than just 'crime', though that's complicated enough. The interpreters would need to be 'engaged' in a highly controversial arena, this year made all the more involved because of police and crime commissioner elections.
2. When we move away from national statistics to official data at large, a question becomes: who assures this data. The Audit Commission (RIP) was moving towards extending the definition of audit to include verification of data production beyond financial information. Is this the job of the NAO, or of ONS? ONS has not, yet, seen it as part of its role to verify and assure.