The potential of broadcast programming to reach millions of people holds obvious appeal to researchers looking to maximise the dissemination of their work. But when it comes to impact, having vast reach is just one part of the equation – how can the significance of broadcast research be determined? Melissa Grant, Lucy Vernall and Kirsty Hill developed a mixed-methods approach, using questionnaires and focus groups, that sought to measure the impact of health-related research broadcast in two programmes on prime time television. Follow-up work conducted after the broadcasts showed that participants’ understanding of the issues had subsequently been enhanced, with a number revealing that they had changed their behaviours as a result of the research.
When the media calls to say “we’re interested in your research”, it’s certainly a rather flattering moment. There may have been a bit of run-up and brokering for this to happen, and it may be just the first step on a long road, with quite a while passing before the research is broadcast. But at some point beyond this, the REF-aware researcher will begin to think about what impact this might produce and how exactly to measure that. The media has a pervasive role in the communication of research; whether a short announcement or a feature-length documentary, the content can be broadcast to millions of people in a very short space of time. So reach, one aspect of REF impact assessment, may be high – but what about significance? That short space of time might not be long enough to leave a lasting impression.
We have been collaborating on an evaluation of this type of significance. In January we published an article in Research for All exploring the evaluation of research presented on television. By using a pragmatic mixed-methods approach, we attempted to measure the impact of research broadcast in two one-hour factual health-related programmes on prime time television. The reach of engaging with TV programming did not disappoint, with an average of 2.5million viewers (11 per cent of the total evening’s viewers) across the two programmes, the majority of whom watched on the night of the broadcast (data obtained from the Broadcasters’ Audience Research Board) in the UK. To determine the significance of these broadcasts, our approach required participants for questionnaires (to be completed before and after the broadcasts) and focus groups, and Twitter data mining, as a specific hashtag was associated with the programmes.
There were many hurdles along the way, such as recruitment and retention of participants, and the effect of participants being aware they were taking part in research. These are well known and can be countered by using professional recruitment companies and through reimbursement of time and expenses. For our focus groups we employed a professional recruitment agency to target particular demographics. The targeting is important to ensure the evaluation is addressing the views of that programme’s likely audience in terms of age, gender, socioeconomic group, etc. If participants do not mirror the audience the evaluation may underestimate the impact. For instance our self-selecting participants were recruited through our own networks and had a slightly higher educational level compared to those of the focus groups and the likely viewers.
We were particularly interested in discovering if the broadcasts increased the knowledge of the participants and, in the case of the focus groups, if this translated into behaviour changes that could alter their health. Indeed we saw quite a leap in knowledge in some cases, as demonstrated by some participants correctly answering certain questions after the viewing compared to before. However we also found that in response to some questions the audience certainly initially knew more than we were expecting. The focus groups in particular allowed for new topics to come out that could be useful for expanding or targeting research in the future. In addition, focus group participants were contacted nine weeks later and asked if they had changed any habits to the benefit of their health: 30 per cent revealed that they had. This indicates that the impact did indeed have significance in addition to its documented reach.
Image credit: TVintage by Ajeet Mestry, via Unsplash (licensed under a CC0 1.0 license).
Mining Twitter elicited nearly 1,300 tweets but only one per cent were relevant to our research themes and it was impossible to say if there was lasting change in knowledge or habits, as the majority were made during the hours of broadcast and responded directly to what was shown on screen. So, fascinating as it was to see the response to the programmes without any kind of intervention, this route did not necessarily provide information about impact but rather on different audiences and topics to engage with in the future.
Overall we found an indication that broadcast programming did indeed have a significant impact, coupled with a wide reach. The evaluation went beyond what might be the norm for the assessment of a media broadcast – particularly in the eyes of the broadcasters, where this is largely just a numbers game. The evaluation required investment of both time and money but we feel this kind of engaged investigation is important not only to find out about the value of the initial broadcast, but also to help connect with the audience to find new ways forward, potentially in partnership in the future.
Note: This article was originally posted on the LSE Impact Blog and is reproduced under a Creative Commons Attribution 3.0 Unported License. It is an opinion piece which gives the views of the authors, and not the position of the LSE Impact Blog, the London School of Economics or the UK Data Service.
This blog post is based on the authors’ article, “Can the research impact of broadcast programming be determined?”, published in Research For All (DOI: 10.18546/RFA.02.1.11).
About the authors
Melissa Grant is a senior lecturer at the University of Birmingham’s School of Dentistry using quantitative methods, such as mass spectrometry of saliva, to study oral health, particularly associated with gum disease (periodontitis).
Lucy Vernall, Academic Ideas Lab, works at the interface of broadcast and research, helping to simulate new TV and radio programmes based on research, and to capture evidence of resulting impact.
Kirsty Hill is a senior lecturer at the University of Birmingham’s School of Dentistry using quantitative and qualitative methods to study oral health and its impact on the general population.