Dr Javiera Cartagena-Farias (left) and Dr Catherine Henderson (right), researchers at the London School of Economics, share their insights into the benefits of linking survey and administrative data.
By linking survey and administrative data, we can unlock powerful insights to improve how health and social care services are designed, delivered, and evaluated – reducing burden on participants while strengthening the evidence base for policy and practice.
Effective policy and the strategic planning of public services rest on the foundation of reliable, high-quality data. Traditionally, large-scale surveys such as the English Longitudinal Study of Ageing (ELSA) and Understanding Society (UKHLS/USoc), among others, have provided valuable insights into the needs and outcomes of service users.
However, declining response rates in recent years have raised concerns about the robustness of this evidence base and its capacity to support policy and practice evaluation. Even where response rates remain acceptable, surveys face inherent limitations. They often lack sufficient sample sizes for certain subgroups of interest, and they are resource-intensive to administer.
In our work, we have seen firsthand how difficult it can be to gather the data needed to evaluate what works, when and for whom.
For participants, surveys can also be burdensome, requiring considerable time, sometimes addressing sensitive topics, and potentially creating stress and frustration. This burden is particularly acute when evaluations target individuals who are already under stress because they are in poor general health, have communication or memory difficulties, or have recently left hospital, to name a few such circumstances. In such contexts, traditional survey data may not always be the most appropriate or feasible source of evidence.
One promising alternative is to draw on the data already being collected by services themselves. Administrative records gathered by local authorities, voluntary and community sector organisations, and the NHS are unique in content and scale, and can offer rich insights into patterns of service use.
However, these records are rarely designed with evaluations in mind. They often lack information on key outcomes such as wellbeing, or on basic but vital measures like mobility and activities of daily living (ADL), which are inconsistently captured and seldom recorded in structured formats.
Linking survey and administrative data: a powerful combination
Recent methodological advances have demonstrated the value of combining administrative records with survey data through data linkage. With the appropriate consents, survey data can be linked to routinely collected local authority social care records and/or to health data from GPs and NHS hospitals, and we can build richer, more complete pictures of people’s lives and the support they receive.
This approach can greatly expand the analytical potential of both sources. For instance, linkage can support cost-effectiveness analysis by making available both participant-reported outcomes data and continuous data on health and social care service use.
Here, linkage can reduce the amount of information needed from research participants about the services they use, because data available from care records are used instead. This approach can be particularly useful for research with people in poor health, who might withdraw from long and potentially tiring surveys.
In practice, data linkage can draw on different administrative sources:
- Nationally warehoused datasets, such as Hospital Episode Statistics (HES), or Client Level Data (CLD), among others, provide large-scale coverage of the population, but require careful navigation of governance and access processes.
- Local authority datasets can capture important dimensions of social care provision, although these do not typically extend to NHS services.
- Regional initiatives, such as the Discovery project in East London, are pioneering the integration of NHS and social care records, providing further opportunities for joined-up analysis.
Lessons from practice
Researchers working on administrative data have documented both opportunities and challenges. Issues of data quality and coverage remain significant, as do the complexities of securing governance approvals, managing data-sharing agreements (DSAs), and contracting with multiple organisations. Finally, health and social care organisations will apply their own data protection impact assessment processes, another potential source of delay.
Technical expertise and robust data security infrastructures are essential, and the time spent negotiating and procuring access represents a substantial, albeit largely unquantified (and unpredicted) cost to research. Access and linkage to health data have been formalised via NHS Digital (see the grey box below); by contrast, access to local authority data must be negotiated individually. Here, IT systems are often fragmented and evolving, limiting longitudinal tracking and analytical use.
The UK Data Service has recently worked with NHS Digital to streamline access to several physical and mental health survey datasets, allowing researchers to largely avoid the complex DSA process.
This partnership represents a meaningful step toward improving access to secure, high-quality data for the public good. However, it currently does not extend to NHS Hospital Episode Statistics products, which offer larger sample sizes and detailed insights into healthcare services.
Data fields also vary widely across councils, complicating standardisation and national aggregation. Analytical teams face capacity constraints, with unpredictable and lengthy delays in extracting data.
These challenges are not unique to a few studies – they reflect broader system-level issues that researchers across England continue to face. In the course of our own work, we have applied for administrative datasets via bodies such as NHS England and for survey data via the UK Data Service. The publications we mention here by researchers working with administrative data have resonated with our experiences.
Yet, when it works, the benefits are enormous. Linked datasets can reveal patterns of service use, highlight inequalities, and identify what interventions make the most difference to people’s lives. This evidence directly informs policy and practice, supporting better decisions at both local and national levels.
There is hope!
Moves to improve data infrastructure are beginning to address some of these barriers. National initiatives, such as the imposition of data standards under NHS England and legislative changes in the Health and Care Act 2022, aim to ensure more consistent recording. Proposals for minimum datasets that capture not only service use but also critical outcomes such as wellbeing are particularly important.
The push on data digitalisation across the health and social care sector may also help to improve the quality, safety and personalisation of care and support services. These steps represent essential progress, but sustained government support will be required if they are to deliver meaningful improvements in data quality, coverage, and accessibility.
Evidence from the recent DACHA study shows that health and social care records data can be linked, despite issues arising around data quality, governance approvals, data access (particularly to primary care records), and the substantial time and expertise required to extract and manage data securely.
The potential benefits of data linkage are clear: freeing up research and operational resources while generating actionable evidence to improve care.
But also urgency!
The urgency of this task cannot be overstated. England’s health and social care systems are under growing strain from rising demand, workforce shortages, and financial pressures. In this context, the ability to evaluate what works is central to the equitable allocation of scarce resources and to the delivery of high-quality, person-centred care.
Without reliable and inclusive data, efforts to design responsive and effective services risk being compromised, leaving both policymakers and service users at a disadvantage.
A call to action
Researchers with experiences of applying for and linking health and social care administrative data and survey data should be encouraged to document these experiences, to illustrate the challenges encountered, and to identify what worked or didn’t work to resolve them.
By expanding the evidence base on the barriers and facilitators to data linkage, we can help build momentum within government and within research communities to streamline and improve these processes.
England holds some of the richest administrative and survey datasets in the world. To harness their full potential, we need:
- Continued investment in data infrastructure, governance, and analytical capacity across health and social care.
- Collaboration between government, local authorities, researchers, and data providers to simplify and standardise access.
- Recognition that secure, ethical data linkage is not just a technical process. It is essential to improving people’s lives and making care systems fairer and more effective.
About the author
Dr Javiera Cartagena-Farias is a Research Fellow at the Care Policy and Evaluation Centre (CPEC) at LSE.
Her research focuses on the quantitative evaluation of long-term care services and health and social care integration strategies, as well as the role played by housing conditions and the development of care needs. Javiera’s research often employs underused administrative data to help to gain a better understanding of the social care system in England.
Follow Javiera on LinkedIn or BlueSky.
Dr Catherine Henderson is an Assistant Professorial Research Fellow at the Care Policy and Evaluation Centre (CPEC) at LSE.
Her research interests include psychosocial interventions for people with dementia, the cost-effectiveness of assistive technologies in managing long-term health and social care needs, intermediate care, residential care provision, and resource allocation within residential and nursing homes.
Follow Catherine on LinkedIn.
Other stories you’ll find interesting
Comment or question about this blog post?
Please email us!