As users of official statistics will be aware, more and more government datasets are available, in fairly raw form, on the internet. This allows people to examine for themselves the data used by government. Have you tried this armchair auditing? Here is a link to the Treasury COINS database http://data.gov.uk/dataset/coins - not official statistics I should stress. If you can find a usable statistical message in these databases you clearly do not need much help from government statisticians. Or could it be that you do need help but have not realised it; and the messages you have inferred are not actually correct.
One user on the COINS website commented “although the raw data is useful for those with the time and skills to be ‘armchair auditors’ it would be wrong for this to become the only form of audit.” That’s a good point, but not quite the full story. The analysis of raw data certainly requires time and skills but it also requires a proper understanding of the data. Are they what they appear to be; are they subject to revisions; are they the latest version; are they incomplete; are there documented weaknesses; for what uses are these statistics generally regarded as useful; are there other datasets that might modify our interpretation. And so on. This is the reading light for the armchair auditor. And producing it requires a different range of expertise.
You cannot (nor will you soon be able to) put official data in to a statistical app and get a summary of their strengths and weaknesses in relation to their principal uses. Perhaps one day. But if there ever is such an app it will most likely work by accessing the written advice and knowledge of the people who put the statistics together and who do actually know about their strengths and weaknesses, revisions, uses and the rest.
In short, developments in the release of data simply underline the importance of preparing and publishing professional statistical advice. We in the Statistics Authority call this ‘narrative commentary’, and expect to see it alongside a summary of the statistics at the time they are released. This does not mean that the release of detailed management data necessarily needs to wait on the preparation of summary statistics and considered commentary. But we think it would be helpful to the user to, at least, offer a link from the latest detailed data back to the most recent summary and commentary.
Going a little deeper, two types of written advice are needed. Firstly, descriptive text which sets out the relevant facts about the statistics such as time periods, geography, methods, definitions, etc. Secondly, narrative text drawing out the main messages from the statistics, explaining the relevance of the statistics in terms of the policy context, stating assumptions about the use that will be made of the statistics and highlighting their strengths and limitations in those contexts.
Among the requirements of the Code of Practice for Official Statistics are that full and frank commentary and analysis should be offered, including:
· information on the quality and reliability of statistics in relation to potential uses
· information on methods, procedures and classifications
· factual information about the policy or operational context
· presentation of statistics in graphs, tables and maps that enhance clarity, interpretability and consistency.
The Statistics Authority has also offered the following advice in relation to the preparation of written commentary:
· Official statistics only justify their production at public expense when they are used in ways that provide public benefit. Most official statistics require careful explanation. Government statisticians will usually know the strengths and weaknesses of the statistics better than other commentators.
· The cost of supplementing statistics with text and guidance is relatively small, particularly in cases where statistics are produced in a regular series and the guidance is updated at each release. It is likely to be more efficient, in terms of the return on public investment in statistics, to offer written guidance to the user than not to do so.
· Government statisticians share a common training and an ethos of impartiality. Whilst experts outside government are of course capable of writing impartial commentary, they are not under any obligation to do so.
· Guidance to the user is needed most at the moment the statistics are first made public. The news media look to the statistical release to provide an objective account of the statistics. They may regard the absence of such text as indicative of partisan influence. Saying nothing is not necessarily politically neutral.
· The public has a right to know the statisticians’ understanding of the messages from the statistics, just as they have a right to the data itself. It is common for statisticians in government departments to brief policy colleagues on the substance of the statistics. That knowledge should, as a matter of principle, be shared more widely.
This is an unequivocal manifesto for statistics to be accompanied by helpful guidance. Armchair auditors may want to check that they have that guidance before they start auditing.
Richard Alldritt, UK Statistics Authority