As Research England considers increasing the weighting of environment statements to 25% in REF2029, there is a heightened focus on what exactly constitutes a high-quality statement. Based on a textual analysis of REF 2021 environment statements, Matthew Inglis, Elizabeth Gadd and Elizabeth Stokoe highlight eight qualities associated with high-scoring statements and suggest how they might improve environment statements in REF2029.
Having been closely involved with Loughborough University’s submission to REF 2021, the drafting of REF Environment Statements (“REF 5b”) has occupied (too) much of our time and energy. These word-limited, discipline-level narratives formed 15% of the overall outcome and asked each institutional Unit of Assessment (UoAs) to describe their: (1) “unit context, research and impact strategy”; (2) “people, including: staffing strategy and staff development, research students, equality and diversity”; (3) “income, infrastructure and facilities”, and (4) “collaboration and contribution to the research base, economy and society.”
When the results came out in 2022, UoAs were provided with an assessment of their Environment Statement across the four star ratings. For example, an environment might be judged as 15% four-star, 50% three-star and 35% two-star. However, the feedback provided no clear sense as to which elements worked well, and which were weaker.
With this in mind, we decided to do our own analyses to understand what might lead assessors to score a research environment highly and what might not. We used latent Dirichlet allocation topic modelling to analyse all 1888 unit-level environment statements from REF2021 to identify their constituent topics and then ran a regression analysis to understand how these topics were associated with REF Environment GPA scores.
The results were fascinating and are described in full in our paper published in Research Evaluation. In summary, we found that beyond those topics that were regional or disciplinary in basis (Scottish, the North, Chemistry) a further eight key topics comprised environment statements (Table 1). Of these, four were significant negative predictors of environment GPA; two were significant positive predictors, and two were not significant predictors. Statements that had higher weightings from the “Immature Research Environment”, “Staff Ways of Working”, “REF-Focused Research Strategy”, and “ECR Development” topics were associated with lower environment GPAs. Statements that had higher weightings from the “Exemplification of Strategy and Processes” and “Industry Partners and Funding” topics were associated with higher environment GPAs. Perhaps most surprisingly, the “Career Development and EDI” topic had a nonlinear relationship with GPAs, meaning that those statements which did not focus much on these issues scored badly, but so did those which focused heavily on them.
In addition, for Main Panels A (medicine, health and life sciences), B (physical sciences, engineering and mathematics), and C (social sciences), environment statements that discussed more discipline-specific issues scored more highly than those that did not.
| Topic | Beta | Significance |
|---|---|---|
| Topic 4—Internal Structure of Research Units | −0.006 | Not significant |
| Topic 7—Career Development and EDI | −0.023 | Not significant |
| Topic 16—Immature Research Environment | −0.438*** | Significant negative predictor |
| Topic 18—Staff Ways of Working | −0.057*** | Significant negative predictor |
| Topic 28—REF-Focused Research Strategy | −0.054*** | Significant negative predictor |
| Topic 40—Early Career Researcher (ECR) Development | −0.112*** | Significant negative predictor |
| Topic 30—Exemplification of Strategy and Processes | 0.117*** | Significant positive predictor |
| Topic 34—Industry Partners and Funding | 0.068*** | Significant positive predictor |
* P < 0.05, ** P < 0.01, *** P < 0.001
How to write a “good” environment statement
With the important caveat that all our analyses are correlational, we are nevertheless able to make tentative recommendations as to how to write a “good” environment statement that might be positively evaluated by REF reviewers. Of course, whether REF environment statements themselves capture what others may regard as a good research environment is a separate question.
1. Avoid stating things that high-quality research environments would consider trivial
For example, we would not mention that most staff in our unit have doctorates, or that our staff attend academic conferences and write articles in academic journals. We would avoid using the phrase “research-active”, especially in an aspirational way, and we would not mention that our staff review articles for academic journals unless they had substantial editorial roles. In short, routine research activities should not be discussed in REF environment statements. Doing this is likely to give readers the impression that research is not a central feature of the unit’s work and reduce perceptions of the vitality of the unit’s research environment.
2. When discussing research strategy, do not give the impression that your research strategy is solely driven by the REF
Instead, our strategy would be organised around research centres, research groups, and departments. It would be focused on an academic discipline, not a “UoA”. The staff who led our submission would not be characterised as a “UoA Working Group”, and if we appointed other academic staff to REF leadership roles, we would not mention it in our submission.
3. Do not go into too much detail about the specific ways in which staff-related processes operate
For instance, we would not explain how decisions about sabbatical leave are informed by input from both the research committee and the teaching allocation committee. Similarly, details about which staff are involved at which stages in approving requests for conference travel funding would be omitted. Why might including detail of this sort be associated with lower GPAs? One plausible explanation is simply that including such details is a waste of space. REF environment statements are word limited, so including superfluous details might simply prevent the inclusion of content that would be causally associated with higher GPAs.
4. Do not focus too much attention on how you support the career development of ECRs
This might sound surprising in the light of the REF submission guidance’s statement that submissions should include “evidence of how individuals at the beginning of their research careers are being supported and integrated into the research culture of the submitting unit.” However, our analysis showed that whilst too little attention on ECRs might lead to a lower GPA, too much can have the same effect, possibly due to taking up space that could have been used for content that was more strongly associated with positive GPAs. Another possibility is that an excessive focus on ECR development might indicate to panellists that a unit feels that they have an unusually low proportion of senior established researchers in post.
5. Take care to discuss career development and EDI, but not too much
The REF guidance emphasised that EDI should be discussed throughout submissions, but our analysis showed that those that devoted too much or too little attention to EDI scored less well. Our analysis suggested that devoting ∼13% of the statement to this topic was optimal. Whilst not taking EDI sufficiently seriously would harm a submission, once a submission successfully demonstrated that career development and EDI was a matter of concern, then further discussions on the topic became unnecessary. As with ECR career development further detail may crowd out space that could have been productively used to discuss other issues. It is also possible that some statements mentioned EDI so often that it conveyed a tick box approach rather than an authentic embedding. It is also possible that an unusually high level of discussion of EDI issues creates the impression of a poor environment for minoritised colleagues.
6. Illustrate your research strategy by giving as many concrete examples as possible of how it has been implemented in practice
For example, if we provided pump-priming research funding to our staff, we would give an example of someone who had received funding, what they did with it, and what this led to.
7. Mention your research funding and industrial partnerships as much as possible
This might seem superfluous, as panellists were provided with each unit’s grant expenditure alongside the written environment statement. However, our data suggest that mentioning funding and partnerships explained significant variance in GPAs over and above standardised grant income per FTE. In sum, having high levels of grant income is insufficient: one must use it to provide evidence of a successful research strategy and environment as well as what the income enabled.
8. Discuss your discipline as much as possible, particularly if you are writing a statement as part of a submission to a STEM UoA
For example, we would illustrate the success of our research strategy by discussing some of the important research findings it facilitated, we would name our research groups using well-understood disciplinary terms, and we would describe the work that our research funding allowed us to do, rather than merely state grant funding amounts.
Although it is not yet clear what the format will be of the People, Culture & Environment element of REF 2029, we hope that given proposals to increase the weighting of this element to 25%, this analysis might help institutions reflect upon what it means to have a high-quality research environment.
This post draws on the authors’ article, What is a high-quality research environment? Evidence from the UK’s research excellence framework, published in Research Evaluation.
The content generated on this blog is for information purposes only. This Article gives the views and opinions of the authors and does not reflect the views and opinions of the Impact of Social Science blog (the blog), nor of the London School of Economics and Political Science. Please review our comments policy if you have any concerns on posting a comment below.
Image Credit: Carmel Arquelau via Unsplash.
