Transparency and Open DataLeading University of Southampton academics, Professor Nigel Shadbolt and the British inventor of the world wide web, Professor Sir Tim Berners-Lee, have led the development of a groundbreaking new portal - data.gov.uk 

Set to revolutionise the way we access services and do business, it provides a single point of access website containing a collection of almost 4000 (and growing) data sets of non-personal data from across Government, ranging from education to transport, from government spending to crime.

Introduction

Non-personal data underpins our economy and our society - data about how much is being spent and where; data about how schools, hospitals and police are performing; data about where things are; and data about the weather.

Extracting this crucial data from government departments was a continuing problem, until the introduction of the Freedom of Information (FOI) Act. However, FOI-derived information does not carry an automatic right to republish beyond the person who requested it.

Berners-Lee has observed: “If data can be published under a Freedom of Information request, why not publish it online? By releasing this data, we can unlock new ideas for delivering public services, help communities and society work better, and let talented entrepreneurs and engineers create new businesses and services.”

Development

In June 2009, Berners-Lee and his colleague from the School of Electronics and Computer Science at the University of Southampton - leading web science expert, Professor Nigel Shadbolt - were asked by then Prime Minister, Gordon Brown, to help transform public access to government information.

As this work progressed, the Prime Minister said: “We want to build on the outstanding work Sir Tim and Nigel Shadbolt have put in to making public data public. We are determined to go further in breaking down the walled garden of Government, using technology and information to provide greater transparency on the workings of Whitehall, and give everyone more say over the services they receive.”

In describing the ambition for open government data, Professor Shadbolt commented: “The vision is that citizens, consumers and Government can create, re-use and distribute public information in ways that add value, support transparency, facilitate new services and increase efficiency. We believe we can achieve this with the emergence of a new generation of web techniques and standards.”

Shadbolt and Berners-Lee worked closely with technical and delivery experts to oversee the development and implementation of data.gov.uk. During the initial six-month development period, they talked to members of the Cabinet and civil servants and involved over 2,400 people in the developer community to help determine the data they wanted, and the best way to access it.

Beta version

In September 2009, a beta version of data.gov.uk was released to developers. A beta version implied the site was still under development – this, in itself, was somewhat unusual for an official government website. It was a site developed using open source software and open standards and at this point contained more than 1,100 datasets, ranging from traffic counts on the road network to planning applications and from reference data on schools to the Farm Survey. Over 1,000 people began testing the site and creating applications to bring together information from different sources.

In a further development, Nigel Shadbolt was invited to chair a Local Public Data Panel that included local government chief executives, IT specialists, and entrepreneurs. With a two-year remit, until December 2011, the panel will oversee the release of local public data, ensuring it is linked effectively across relevant agencies, authorities and government departments.

Public launch

On 21 January 2010, the new portal - data.gov.uk - was launched as a new, single access-point website containing an initial collection of 2,500 sets of non-personal data from across the whole of Government. This was a huge achievement – and unusual in that it provided access to data across England, Scotland and Wales.

The data is in a format that can be reused by any individual or business to create innovative new software tools, such as applications providing information on house prices, local schools, amenities and services, or access to local hospitals. Another benefit is that openly available public data enables citizens to choose between public service providers, and lobby for improvements.

At the time of its launch, Professor Shadbolt commented: "For the first time, we have a single online place where those looking for government data can go to find it, without having to know which department holds what and where it is. It's not our job to say where data might be useful; it's our job to unleash it and allow businesses and independent developers to build innovative services which they can then deliver to users.”

Applications have continued to be created using the data feeds, including PlanningAlerts, a free service that combines local authority planning websites looking for planning applications. PlanningAlerts then automatically e-mails details of applications in the local area to anyone who has signed up for the service.

FillThatHole uses location data from the Office for National Statistics to allow people to report potholes and other road hazards across the UK. Another is a mobile phone application that locates your nearest NHS dentist, while ASBOrometer uses your phone location and reports the local ASBO rate profile and where that puts the area, nationally.

Berners-Lee has commented: “One of the really fascinating things about this work is that you can never anticipate the services that will actually turn out to be the killer apps. We must try to turn the service provision mentality on its head. We provide the basic, raw data, and the services are developed through people’s commercial and social and voluntary efforts. We need to move to an ethos where, if somebody in Government creates a database, then, by default, they will create a path so that data can be used by the public.”

On 1 April 2010, in a significant advance, the Government released a selection of geographical data held by the Ordnance Survey. The then Secretary of State for Communities and Local Government, John Denham, commented: “Increasing online access to Ordnance Survey data will attract a new wave of entrepreneurs and result in new solutions to old problems that will benefit us all. It will also drive a new industry, creating new jobs and driving future growth.”

The future

After the election in May 2010, the new Coalition Government signalled its commitment to continuing the open data initiative by inviting Professors Berners-Lee and Shadbolt to join the new Public Sector Transparency Board, which is charged with setting open data standards across the public sector and the development of the legal ‘Right to Data’.

Francis Maude, Minister for the Cabinet Office, who chairs the new board stated: “…we want transparency to become an absolutely core part of every bit of government business. That is why we have asked some of the country’s, and the world’s, greatest experts in this field to help us take this work forward quickly here in central Government and across the whole of the public sector.”

The coalition Government has been quick to release important new data sets. Prime Minister David Cameron’s letter on transparency laid out clear commitments, and has led to the release of comprehensive spending data from the Treasury. All government departments must publish spending above £25,000. Local authorities will have to provide data on each item of spending above £500. Police forces will be required to publish data about crime at the local level, and more. At the Department of Communities and Local Government, the new Secretary of State, Eric Pickles, has been promoting transparency and open data publication within his own department asserting: “Greater openness in spending is the best way to root out waste, spot duplication and increase value for money.” The department now has a dedicated Minister for Transparency and Open Data – Baroness Hanham.

Following advice from Professor Shadbolt, the Transparency Board has now published draft Public Data Principles that embody commitments to open government data. In pursuing their work, Shadbolt and Berners-Lee have adopted a pragmatic approach that sees data release progressing from simple publication formats, such as spreadsheets, to ones in which the data is directly accessible and ‘linked’. Linked Data is a set of emerging standards that will enable data to be directly accessed on the web. Not only do these open standards enable the description of data of all types, they also support interlinking of data. For example, a government generated web identifier for a local authority can be linked to other data from across Government – data that is relevant to that authority.

In a joint statement, Shadbolt and Berners-Lee have noted: “… the linked open data movement is a crucial tool, for Government, public and industry to get the most value from the important resources being opened up. During times of austerity, transparency is essential, and open data will play a crucial role."

And in broader context the Guardian, in a lead editorial, noted: “It (is?) … hazardous trying to envision how freer data will redraw the boundaries between different communities or recast their relationship with power. But it is reasonable to speculate that the uncovering and unlocking of so much information will drive improvements in public policy. It will level the territory on which voters meet politicians… Without the printed word, there would have been no informed electorate, no demand for accountability from our leaders – and indeed, no democracy at all. Open data will surely revive it, and in time could transform it too.”