Data Impact blog

This year, the UK Data Service reached an incredible milestone – over 10,000 data collections are now discoverable through our Data Catalogue. In this two-part series we discuss the journey to this milestone, why it matters and how it sets the stage for the next chapter of data innovation and impact.

A legacy of data stewardship

Since its founding in 1967, the UK Data Archive has been at the heart of social science data preservation and access in the UK.

What began as an initiative to safeguard key research data has grown into one of the world’s leading infrastructures for economic, population and social data.

Over the decades, the Archive has expanded its remit – from early household surveys to complex longitudinal studies, qualitative collections and international comparative datasets, to now reaching over 10,000 data collections discoverable through the UK Data Service.

This milestone reflects almost six decades of work to make high-quality, trusted data available across the UK and beyond.

Each of these collections represent a unique contribution to our shared understanding of society. Together, they tell a story of six decades of progress in data stewardship, research, collaboration and impact.

So, how did this legacy take shape?

Laying the foundations: 1963 – 1970

The roots of the UK Data Archive stretch back to 1963, when the Social and Economic Archive Committee (SEAC) was established.

The SEAC was formed to address a growing challenge: researchers were struggling to share information about social surveys and the data generated by them.

This meant that social survey work was often being duplicated, wasting both time and resources, with some data even being sold overseas to the United States. As a result, British researchers were missing out on opportunities to access vital data.

In 1967, the UK Data Archive was founded at the University of Essex as the Social Science Research Council (SSRC) Data Bank – the first step towards preserving and sharing research data more systematically.

In 1969, a fire destroyed the building that housed the Archive’s early administrative and clerical records, but the work continued undeterred – laying the foundations for what would become a trusted national data infrastructure.

Figure 1 – A timeline of the UK Data Archive and UK Data Service. 

Larger version of figure / Accessible version of figure.

Expanding access and influence: 1970s – 1980s

In 1972, the SSRC Databank changed its name to the Survey Archive. For the first time, the Government Statistical Service permitted its surveys to be stored at the Archive, following successful advocacy by academics who recognised the long-term value of this data.

Landmark collections soon followed, including the British Social Attitudes Survey, the General Household Survey and the Labour Force Survey.

Camilla Philpot-Thomsen, Associate Director of Data Operations at the UK Data Service, reflects on these landmark datasets that now form part of the UK Data Service’s 10,000 data collections – and the impact that they continue to have for researchers over time:

“Every dataset added to the UK Data Service represents an investment in future discovery – enabling researchers, policymakers and students to build on existing knowledge and address new challenges.”

The Archive’s early success helped inspire the creation of the Council of European Social Science Data Archives (CESSDA) in 1976 – now an international consortium that continues to shape social science data sharing today.

By 1982, the Survey Archive had evolved into the SSRC Data Archive, when it began to archive diverse data types beyond just survey data. When the SSRC became the Economic and Social Research Council (ESRC) in 1984, the Archive adopted the name of ESRC Data Archive.

During this period, the Archive also began engaging in major data-oriented projects funded by the European Commission, such as the BBC Domesday Project and the Rural Areas Database.

Figure 2 – A timeline of the UK Data Archive and UK Data Service: 1970s – 1980s.

Larger version of figure / Accessible version of figure.

Building trusted data infrastructure: 1990s – 2000s

Throughout the 1990s, the Archive continued to grow its national and international role through a range of pioneering projects funded by the European Commission, including Nesstar, FASTER, and the CESSDA-PPP project – all oriented towards research, development work and the advancement of technical solutions.

During this time, the Archive developed a number of discrete yet interlinked services, most notably

• the Economic and Social Data Service (ESDS)
• the Secure Data Service (now SecureLab)
• the History Data Unit
• the History Data Service

These innovations laid the groundwork for the integrated services that would later become part of the UK Data Service.

By 1996, the Archive began receiving direct funding from Jisc, in recognition of its support for teaching and learning, and became known simply as the Data Archive. In 2000, to recognise its UK-wide and international role, the organisation became the UK Data Archive.

The UK Data Archive was considered a crucial part of the data ecosystem in the UK at this time due to the strong reputation it had built up over the previous years.

It became clear that many surveys were asking the same questions, and the Archive was on one of the committees that encouraged the reduction of duplication in the sector.

In 2007, during the Archive’s 40^th year, it relocated to a new purpose-built building on the University of Essex campus along with the Institute for Social and Economic Research.

The Archive also achieved NHS Data Security and Protection Toolkit accreditation and it digitised all historical documentation. This was a vital step in ensuring that decades of valuable research data and documentation could be accessed, searched and reused by future generations of researchers.

Cristina Magder, Data Collections Development Manager at the UK Data Service, reflects on this journey that shaped the UK Data Service and highlights what has particularly driven its success:
 

“What truly drives our success isn’t just technology or infrastructure, but it’s the people. At the heart of all of this, behind every one of these achievements, is a team that deeply believes in the power of data to improve lives.”

Figure 3 – A timeline of the UK Data Archive and UK Data Service: 1990s – 2000s.

Larger version of figure / Accessible version of figure.

Other stories you’ll find interesting

Comment or question about this blog post?

Please email us!