Soft Machines

1. On the discrepancy between ONS and HMRC estimates of business R&D.

In the UK, there are two ways in which the total amount of business R&D (BERD) is measured. The Office for National Statistics conducts an annual survey of business, in which a sample of firms is asked to report how much R&D has been carried out. Meanwhile firms can report what R&D they have carried out to the taxman – HMRC – in order to claim R&D tax credits, which according to circumstances can be a reduction of their liability for corporation tax, or an actual cash payment. In recent years, the two measures of business R&D have increasingly diverged, with substantially more R&D expenditure being claimed for tax credits than is reported in the BERD survey.

The divergence between HM Revenue and Customs (HMRC) and Business enterprise research and development (BERD) estimates of research and development (R&D) expenditure. Source: ONS.

The ONS has been looking into this divergence, and has recently published a note which concludes that the primary reason for the discrepancy is an undersampling of the small business population. On this basis, it has adjusted its previous estimate for business R&D substantially upwards – in 2020, the revision is from £26.9 bn to £43 bn. In future years, ONS will introduce improved, more robust, methodologies that will include a wider range of SMEs in the sample they survey.

In principle, there could be two possible causes for the growing divergence between the total business R&D recorded by the ONS BERD survey and the amounts underlying claims to HMRC for R&D tax credits:

a. The incentives of R&D tax credits have caused businesses to stretch the definition of R&D so they can get money for activities that are part of normal business (e.g. market research, working out how to use new equipment). This is exacerbated by the growth of an industry of consultants offering their services to firms to help them claim this money (in return for a %).

b. The ONS survey of firms (the BERD survey) has systematically undersampled a population of small and medium enterprises (SMEs), which turn out to have more R&D activity than previously believed.

In favour of (a) – the discrepancy between the two measures hasn’t been entirely static, as you’d expect if it was simply a question of missing a population of firms who had always been doing R&D at a constant rate, but who have only just been discovered. The gap has risen from £7.3 bn in 2014, to £16.6 bn in 2018. So for this explanation to hold, we need to believe not only that there is an existing population of SMEs carrying out R&D that has previously been undetected, but that this population has been substantially growing. Is R&D growth in the SME sector at a rate of £2.3 bn a year plausible? I’m not sure.

Moreover, the incentives for stretching the definition of R&D to claim free money are obvious. HMRC accept that some claims are outright fraudulent, estimating that 4.9% of the cost of the scheme is attributable to error and fraud. But there’s a big grey area between outright fraud and creative interpretation of the “Frascati” definitions of R&D.

ONS argues in favour of (b), backing this up with a detailed comparison of the microdata from the ONS survey and HMRCs returns. To add some anecdotal support, work in Greater Manchester in collaboration with a data science consultancy does seem to have identified a population of innovative SMEs in GM which has previously remained invisible, in the sense that they are firms who don’t engage with universities or with Innovate UK.

In truth, the real answer is probably some mixture of the two. We’ll learn more once the new methodology has produced a complete data set identifying the sectors and geographical locations of R&D performing firms.

2. Policy implications

Figures for total R&D spending (including both business and public sector R&D) as a proportion of GDP provide a useful measure of the overall research intensity of the UK economy and form the basis for international comparisons. The previous figure for R&D intensity – about 1.7% – put the UK between the Czech Republic and Italy. The new estimates suggest a revised figure of 2.4%, which would put the UK roughly on a par with Belgium, slightly above France, but behind the USA and Germany, and still a long way behind leaders like Korea and Israel. Of course, when making these international comparisons, a natural question is how accurate are the R&D statistics in these other countries. This is a good question that could be investigated by OECD, who collate international R&D statistics.

The international comparison has driven a target for R&D intensity that the government committed to – that it would achieve an R&D intensity equal to the OECD average. At the time when the target was formulated this average was indeed equal to 2.4%. However, the OECD average is a moving target since other countries are increasing their own R&D – it’s now above 2.5%. One can also ask whether a target to achieve international mediocrity is stretching enough.

There are more fundamental issues with the idea of having an R&D intensity target at all. One quirk of expressing the target as a % of GDP is that one can achieve it by driving down the denominator; certainly GDP growth in the UK has been disappointing for the last 12 years, as the Prime Minister has reminded us. One could argue that a numerical target for R&D is arbitrary and one should concentrate more on the instrumental outcomes one wants to achieve from the research – higher growth, more rapid and cost effective progress towards net zero, better population health outcomes etc. As I wrote myself recently in my survey of the UK R&D landscape:

“An R&D target should be thought of not as an end in itself, but as a means to an end. We should start by asking what kind of economy do we need, if we are to meet the big strategic goals that I discussed in the first part of this series. Given a clearer view about that, we’ll have a better understanding the necessary fraction of national resources that we should devote to research and development. I don’t know if that would produce the exact figure of 2.4%, but I wouldn’t be surprised if it was significantly higher.”

Perhaps the most problematic implication of a BERD upgrade is the enduring puzzle that productivity growth remains very slow. This extra, previously unrecorded R&D, doesn’t seem to have translated into productivity growth as we would expect.

This raises the broader question of why we think the government should support business R&D at all, whether through R&D tax credits or through other means. The classical argument is that private sector R&D leads to wider benefits from the economy that aren’t captured by the firms that make the investments, so in the absence of government firms will invest less in R&D that would be socially optimal. This leads to the question of whether all kinds of R&D, in all kinds of company (e.g. large and small) lead to equal degrees of wider spillover effects (and the same question can be asked of intangible investments more generally). If the kinds of R&D that are now being revealed with the new methodology do have smaller spillovers than other types, one might ask what kind of interventions could improve those.

3. Political implications

As others have observed, the chief danger of the revision is that in times of fiscal retrenchment, the government could declare “mission accomplished” and delay or cancel increases in public R&D. This danger seems very real given the direction of the current government. The opposition, on the other hand, has called for an R&D target of 3% of GDP, so there is plenty of room there.

There is an argument that the revision suggests that public R&D is even more effective than we thought in generating private sector R&D – the leverage effect is stronger than we thought. For this argument to be convincing, we’d need to understand the degree to which the companies doing this R&D are connected to the wider innovation system. But it doesn’t then support the wider argument for R&D as a driver of productivity growth – we have the R&D intensity we aspired to, so why aren’t we seeing the benefits in the productivity figures?

There are possible arguments that our focus in business R&D has been too much on the big incumbents – the GSKs and Rolls Royces – whose R&D is very visible. On the other hand, this connects to the long-running question of why we don’t have more of those big incumbents? At this point, we should recall that there are only two UK companies in the world top-100 of R&D performers – AstraZeneca and GSK. So why aren’t some of these previously unseen R&D intensive companies scaling up to become the new big players?

There is much yet to understand here.

In the first part of this series attempting to sum up the issues facing UK science and innovation policy, I tried to set the context by laying out the wider challenges the UK government faces, asking what problems we need our science and innovation system to contribute to solving.

In the second part of the series, I posed some of the big questions about how the UK’s science and innovation system works, considering how R&D intensive the UK economy should be, the balance between basic and applied research, and the geographical distribution of R&D.

In the third part, I discussed the institutional landscape of R&D in the UK, looking at where R&D gets done in the UK.

In the fourth part, looking at the funding system, I considered who pays for R&D, and how decisions are made about what R&D to do.

In the fifth part, I looked in more detail at UK Research and Innovation, the government’s main agency for funding academic science.

In the sixth part, I looked at the other routes that the UK government funds R&D, particularly through government departments.

In this, the final section of my survey of the routes by which the UK government funds R&D, I turn to two areas with the most uncertainty. The first of these is the future of the UK’s participation in the EU Horizon programme. I’ll discuss the distinctive roles of EU funding, and what might replace it in the increasingly likely scenario that the UK is not able to associate. The second is the new agency the Advanced Research and Invention Agency, set up by Act of Parliament in early 2022, and currently just establishing itself; here I’ll suggest some early thoughts about the role this might play in the overall system.

7.1. Horizon Europe – past participation and future prospects of association

In the past, the UK government has funded R&D indirectly through the EU Horizon programme, which provided research grants to UK researchers in HE and to UK businesses, often as part of larger collaborative programmes with researchers and businesses from elsewhere in Europe. EU research funding to UK universities and businesses has been on a very material scale; of course ultimately this money came from the UK’s contributions to the overall budget. In the UK’s national accounts, this was accounted for by a notional cost that reached a high point of £1.46 billion in 2019.

Because EU research money was allocated competitively, there wasn’t a direct relationship between the money the UK put into the budget and the research money the UK received. In fact, because of the UK’s relative research strength, the UK got back significantly more money than it put in. According to an analysis of the 2007-2013 cycle, the UK’s indicative contribution to the budget was €5.4 bn, but it received €8.8 bn of funding for research, development and innovation.

After the UK decided to leave the EU, a consensus developed that the UK should seek to stay associated with the EU’s R&D programmes, an option already taken up by other non-member states such as Switzerland, Norway and Israel. The Trade and Cooperation Agreement between the EU and the UK contained a draft protocol establishing the UK’s association with Horizon Europe (with the exception of the European Innovation Council). “The Parties affirm that the draft protocols set out below have been agreed in principle and will be submitted to the Specialised Committee on Participation in Union Programmes for discussion and adoption. The United Kingdom and European Union reserve their right to reconsider participation in the programmes, activities and services listed in Protocols [I and II] before they are adopted since the legal instruments governing the Union programmes and activities may be subject to change. The draft protocols may also need to be amended to ensure their compliance with these instruments as adopted.”

If the UK does associate, it will need to contribute financially to the Horizon Europe programme. In contrast to the situation when the UK was a member state, when it received more back from EU R&D programmes than it notionally contributed, as an associated country it would need to cover not only the full cost of R&D activities funded in the UK through Horizon UK, but also a substantial additional overhead. The money for this was set aside in the 2021 Comprehensive Spending Review; it amounted to £1.3 bn in 21/22, rising to £2.1bn 24/25.

As I write, the draft protocol has not yet been finalised by the EU side, and given the wider political situation, it seems increasingly unlikely that it will be finalised any time soon. The UK government made a commitment at the time of the 2021 CSR that, in the event of the UK not associating, the money set aside would be retained in the science budget, redeployed in a set of programmes that reproduced the benefits of EU association – the so-called “Plan B”.

On July 20th, the government released more details of “Plan B”, restating the commitment to use the Horizon money for alternative science programmes. “In the event we are unable to associate, we will use the funding allocated to Horizon Europe at the 2021 Spending Review to build on our existing R&D programmes with flagship new domestic and international research and innovation investments to support top talent, drive end-to-end innovation and foster international collaboration with EU and global partners.”

7.2. The Three Pillars of Horizon Europe

The EU’s R&D programmes are agreed for seven year cycles; the current cycle – Horizon Europe – assigns €95.5 billion for the period from 2021-27. The overall goals of the programme are specified in terms of the strategic goals of the European Union – tackling climate change, meeting the UN’s Sustainable Development Goals, and boosting the EU’s competitiveness and economic growth.

To support these broad goals, Horizon Europe supports three “Pillars”. The first of these is “Excellent Science”. This includes the European Research Council, together with schemes supporting early career researchers and collaborative research and training for PhD students. The European Research Council supports investigator led basic science and humanities research; this has a very high reputation in the scientific community, for reasons I’ll discuss below. However, it is important to remember that it is a relatively small part of the overall Horizon programme – it’s been allocated €16 bn in the current cycle.

The second pillar is for “Global Challenges and European Industrial Competitiveness”, which supports research collaborations built around sectors, challenges and missions. These typically involve both academic and industrial researchers in multinational collaborations.

The third pillar is new to the current cycle – “Innovative Europe” is focused on developing more high tech start-up companies, with a new “European Innovation Council”, a “European Institute of Innovation and Technology”, and support for regional innovation ecosystems. In the event of association, the UK will opt out of the “EIC accelerator” – that part of pillar 3 which provides investment funding to companies.

Underpinning the whole programme is an aspiration to create a “European Research Area”, with free and easy movement of people and research groups across the continent, lubricated by exchange schemes for scientists (particularly at early career stages) and cross-border transferability of grants. In the past the UK has benefitted from this, with a scientific and institutional infrastructure that has made the country an attractive destination for scientists from other European countries.

7.3. Why scientists love the European Research Council

Amongst elite scientists in the UK, the main driving force for an enthusiasm for the UK to associate with Horizon Europe is to be able to continue to participate in the European Research Council. This, in part, simply reflects how successful the UK has been in winning competitive funding through this route. For example, in the competition for the most established researchers – the Advanced Grants, which provide €2.5 million over 5 years for a single investigator and their team – UK based researchers won 22% of all grants between 2008 and 2020, compared to 16% and 12% to the two next most successful nations, Germany and France respectively (source).

But beyond the self-interest of UK scientists, why is the European Research Council so highly thought of? It has a clarity of purpose, with a single-minded focus on investigator driven basic research, with no predetermined priorities, but with an emphasis on supporting high risk/high gain proposals. It is correctly perceived as highly competitive, attracting proposals from the most outstanding researchers across Europe – currently its grantees have won nine Nobel prizes. Its decisions are made by a peer review process which is widely considered to be fair, rigorous and well executed.

Peer review isn’t easy to do well. In section 2 of this series, in discussing a possible world-wide slow-down in scientific productivity, I mentioned the suggestion that peer review can lead to conservatism and can suppress radical new ideas. In section 5, I suggested that there was a lack of confidence in the scientific community in the credibility of the peer review systems that the UK Research Councils run. In the light of these concerns, it’s worth asking what the European Research Council gets right about peer review (while recognising that even the ERC’s process is probably not perfect, for example in tricky areas like handling highly interdisciplinary proposals).

In my opinion, there’s nothing magic about the ERC’s approach to peer review. The process involves committees of experts (and, to declare a personal interest, I recently served on the expert panel for Advanced Grants in my own field of Condensed Matter Physics). Those panels invite written comments on proposals from worldwide specialists they choose for their appropriateness to judge individual proposals. In a final meeting, the panels consider the referees’ reports, with interviews with the proposers to give them the chance to respond to criticisms, and come to a collective judgement about which proposals to give highest priority for funding.

What makes this work? The starting point must be high quality panels, with a good range of expertise, the ability to take a broad view, and an effective chair. At its best, the ERC has developed a virtuous circle, in which the high quality of the proposals means that outstanding scientists are prepared to put the time in to serve on panels, while in turn it is the credibility of the process that attracts applications from the best scientists from across a whole continent. It is the researchers on the panels who select the remote referees, using their knowledge of the field to select the most appropriate ones, and then applying their own critical scientific judgement to resolve any discrepancies and differences of opinion between referees. Sufficient time is set aside for in-depth decisions – a single proposal round will involve two panel meetings, each of which can take up to a week.

Meanwhile administrative support is provided by high quality subject specialists working full-time for ERC as programme managers. In the UK, the research councils were forced to make serious cuts on their office staff in the early 2010s, because it was mistakenly believed that these subject specialists represented an administrative overhead, rather than being a precondition for the most effective allocation of R&D funding. This mistake should not be repeated (and, indeed, should be corrected).

7.4. “Plan B” for non-association

The “Plan B” document published this July (Supporting UK R&D and collaborative research beyond European programmes) usefully sets out some principles for how the money set aside for association with Horizon Europe will be used in the event that association doesn’t materialise. But details of implementation remain sketchy, and delivery may prove challenging to the existing agencies and bodies that will be charged with executing these schemes.

These agencies are mostly in UKRI, with a particularly important role for Innovate UK, with the National Academies potentially playing a role in the “talent” schemes. These are largely fellowships at various career stages, that will be in part fill some of the role of the European Research Council, though without the benefits of the institutional strength that ERC has developed, as outlined in the last section.

The emphasis of measures taken so far has been on stabilising the system, in particular keeping in the UK outstanding scientists who have been awarded ERC grants, but who can’t take them up without moving to an EU member state. The commitment has been made to guarantee the funding of any Horizon UK grant awarded to UK based researchers for the lifetime of the grant. It is going to be important to ensure that this happens without bureaucratic hurdles, in perception or reality, as HE institutions in the EU will be making energetic efforts to recruit these researchers.

The last point emphasises the importance of making sure the UK remains an attractive destination for overseas scientists, and promoting researcher mobility to make sure that the UK is centrally integrated in international networks of expertise. The plan here remains vague, but states the intention to fund “bottom-up collaborations with researchers in partner countries around the globe; multilateral and bilateral collaborations; and Third Country Participation in Horizon Europe”.

Measures for supporting business R&D will be funnelled through Innovate UK; it seems these will largely build on existing schemes. The aim is to support both domestic and international collaborations. The international dimension will be particularly important in supporting high technology SMEs to participate in trans-national supply chains and innovation systems, many of which, of course, involve EU member states.

The local and regional dimension of support for innovation systems is also important. EU funding – including structural funding as well as direct R&D funding – has been important in developing clusters in economically lagging parts of the UK, such as Northern England, Wales and Northern Ireland. The Shared Prosperity Fund is likely to offer only a partial substitute for EU structural funds, so it is encouraging to see a commitment to drive “the development of emerging clusters throughout the UK”, and the statement that the “Plan B” portfolio “will support our mission of levelling up the UK and build on our commitment to increase domestic R&D investment outside of the Greater Southeast by at least a third over the spending review period and at least 40% by 2030.”

Moving forward with the association of the UK with Horizon Europe would seem to require a breakthrough in wider EU/UK relations that currently doesn’t seem very likely. In the absence of such a breakthrough, the priority needs to be for the new administration to confirm the funding of plan B, and move very quickly to turn what are currently rather high level plans into deliverable programmes.

7.5 The Advanced Research and Invention Agency (ARIA)

The most recent addition to the UK’s R&D funding landscape is the new funding agency, the Advanced Research and Invention Agency. This was established by an Act of Parliament, finalised in early 2022. It was a personal priority of the Prime Minister’s former chief advisor, Dominic Cummings, who emphasised the need to have a funding agency with the freedom to take big risks, modelled loosely on the US agency ARPA. ARPA was set up in the late 1950’s to ensure technological supremacy for the US armed forces, and research it supported has underpinned world-changing technological innovations such as the internet, the satellite location system that GPS evolved from, and stealth aircraft.

The Act of Parliament establishing ARIA does indeed give a huge amount of latitude in defining its goals and modes of operation; much is left to the discretion of the CEO and the board. The major lever the government retains is the level of funding allocated; the initial commitment is to spend £800m by 24/25. This is a relatively small amount seen in the context of the £20 billion total R&D budget planned for 24/25. Nonetheless, given that we’re already halfway through 22/23, that leaves only two years to get some entirely new programmes off the ground.

The Act does give the Secretary of State powers of Intervention on grounds of national security, and it is easy to imagine that these could be used quite widely. Nonetheless, there is some irony in the way the independence from government that was taken away from the Research Councils has been given to this new agency.

Given that the appointments of the Chief Executive and Chair have only relatively recently been announced, there is not yet clarity about what the new agency will do. I outlined my own views about how such an agency should operate in a piece from January 2020, UK ARPA: an experiment in science policy.

As I wrote then, “If we want to support visionary research, whose applications may be 10-20 years away, we should be prepared to be innovative – even experimental – in the way we fund research. And just as we need to be prepared for research not to work out as planned, we should be prepared to take some risks in the way we support it, especially if the result is less bureaucracy. There are some lessons to take from the long (and, it needs to be stressed, not always successful) history of ARPA/DARPA. To start with its operating philosophy, an agency inspired by ARPA should be built around the vision of the programme managers. But the operating philosophy needs to be underpinned by as enduring mission and clarity about who the primary beneficiaries of the research should be. And finally, there needs to be a deep understanding of how the agency fits into a wider innovation landscape.”

My starting point would be to recognise that pluralism & diversity in funding agencies is a good in itself, and we need to innovate in the way we support innovation. ARPA at its best represented an approach to funding where the focus was on the programme manager – or better, programme leader as the creative force. These leaders should be tasked with assembling and orchestrating teams of talented people to achieve ambitious programmes with concrete goals.

The archetype of the visionary leader is perhaps J.C.R. Licklider, who accepted a position with ARPA in 1962, because if offered an opportunity to realise his vision of computer networking. The research he funded at ARPA laid many of the foundations of modern computing, including the principles of networking that led to the internet, and the principles of human/computer interaction that were further developed a the XEROX PARC laboratory to give us the graphical interfaces that we all take for granted together.

ARPA benefited from a complete clarity of mission – its role was to ensure that the US armed forces enjoyed technological supremacy over any potential rival. That makes clear who its beneficiaries should be – the US Armed Forces.

What should ARIA’s mission be, and who are its beneficiaries? This remains to be decided, but from my perspective it is important to make clear that its primary beneficiaries should neither be the academic community, nor industry. Both communities will be crucial in delivering the mission, but it should not be primarily for their benefit. Instead, I believe that ARIA should focus on one, or a subset of one, of the important strategic goals that the UK state currently faces, as I outlined in the first part of this series.

For me, the most obvious candidate is the challenge of driving down the cost of achieving net zero greenhouse gas emissions to a point where the global transition can be driven by economics, rather than politics.

Up next…

In the next and final part of this series, I will attempt to sum up, with some key priorities for the UK R&D system.

Edited 20 Sept to make clear that the proposed opt-out from Pillar 3 of Horizon Europe only covers the European Innovation Council Fund. My thanks to Martin Smith for pointing this out.

In the first part of this series attempting to sum up the issues facing UK science and innovation policy, I tried to set the context by laying out the wider challenges the UK government faces, asking what problems we need our science and innovation system to contribute to solving.

In the second part of the series, I posed some of the big questions about how the UK’s science and innovation system works, considering how R&D intensive the UK economy should be, the balance between basic and applied research, and the geographical distribution of R&D.

In the third part, I discussed the institutional landscape of R&D in the UK, looking at where R&D gets done in the UK.

In the fourth part, looking at the funding system, I considered who pays for R&D, and how decisions are made about what R&D to do.

Here, I go on to look in more detail at the agencies that allocate R&D money on behalf of the government, examining the range of different purposes those agencies must meet and the constraints they operate under.

This funding system has seen some really substantial changes over the last few years, with the introduction of a new agency, UK Research and Innovation (UKRI), in 2018, and will see yet more in coming years, for example through the new agency the Advanced Research and Innovation Agency (ARIA). The future of the UK’s association with the European Union research funding programme, Horizon Europe, also remains in question. This makes it all the more timely to try and think about the funding system as a whole.

In this section, (after a brief detour to talk about R&D tax credits) I will start with the agencies that make up UK Research and Innovation. The next section will go on to consider the new agency ARIA, European funding and what might replace it, and R&D funding by other government departments.

5.1 Tax incentives for R&D

I begin my discussion, not with UKRI, but with HM Treasury, which is now one of the UK government’s largest direct funders of R&D, through the R&D tax credit schemes, which had a cost to government of £7.4 billion in FY 2019/2020. This cost has increased dramatically in recent years – in 2010/11, it was just £1.1 billion, rising to £3 billion by 2014/15. This is as a result both of increases in the scheme’s generosity, and of increases in take-up by business.

The way R&D tax credits work is that firms report their spending on R&D to the government, which partially offsets the cost of the R&D by reducing the amount of corporation tax the firm has to pay. Corporation tax is paid on the profits made by a company; this is potentially a difficulty for start-ups in the stage where they are investing money in R&D before significant revenues arrive. In these circumstances, where companies are not yet making a profit and aren’t liable for corporation tax, the government partially recompenses them for their R&D expenses with a cash payment.

The logic behind R&D tax credits is simple. Economists argue that firms don’t capture all the benefits to society that doing R&D brings, so left to itself the private sector will invest less in R&D than is optimal for the economy more widely. R&D tax credits correct this market failure, with the state in effect paying for the spillovers that benefit the economy more widely. But, the argument goes, the market knows what people want better than the government, so it’s necessary to funnel money directly to businesses seeking to exploit innovation to meet market opportunities that they have detected.

Supporting business R&D this way means that there’s no need for the government to make any decisions about what kind of R&D to support; there’s no danger of being accused of trying to “pick winners”. This means that the scheme is very cheap to administer, and there is no need for the government agencies to have any specialist expertise or to develop a strategy.

What are the downsides? One is that the scheme almost certainly has a significant deadweight cost – in effect, giving companies money for R&D they would have done anyway. There’s a huge incentive for companies – and the industry of consultants that has grown up to help them claim this government money – to stretch the definition of R&D to include the kind of business as usual that isn’t likely to generate much in the way of spillovers for the economy more widely. And, of course, there is a very real risk of outright fraud.

One strong signal that all is not right with the scheme is a growing mismatch between the total amount of business R&D that forms the basis for these claims, and the independent ONS estimate of Business R&D that comes from survey data. In 2014/15, R&D tax credits were claimed based on £24.4 billions worth of R&D, a bit more than ONS’s estimate of £20 billion for business R&D. But by 2019/20, the comparison dramatically diverges: the ONS estimate for business R&D was £25 billion, but businesses told the taxman that they’d done R&D worth nearly double that, £47.5 billion.

There are differences in definition between the two measures of R&D. For example, it is possible to claim tax credits for R&D that is carried out abroad. This doesn’t make a lot of sense from the economic point of view. It is suggested that the ONS survey undercounts R&D in the financial services and insurance sector; this, however, accounted for only £3 billion (7%) of R&D expenditure for tax credits in the year ending March 2019. But neither of these factors seem sufficient to explain the gulf between the two measures.

The R&D tax credit scheme is currently under review by HM Treasury. Crucial background for this review is a recent paper by David Connell, Is the UK’s flagship industrial policy a costly failure? (PDF). (Connell’s answer to his own question? “Yes”).

But even if this kind of scheme was perfectly run, there is a more fundamental question. This approach to funding R&D in business leaves the choice of what research to do entirely to the businesses, with no attempt at all to align the spending with the priorities of the state. This makes it attractive for governments that don’t have any priorities or strategic goals.

5.2 UKRI: the Research Councils

The 2017 Higher Education and Research Act created UK Research and Innovation (UKRI) as an umbrella organisation into which were folded the existing seven research councils, The innovation agency Innovate UK, and Research England, which looks after the higher education research system in England (though not in the devolved nations). I’ll discuss the goals of this reorganisation, and the degree to which it has succeeded, below. But first I want to discuss the research councils themselves.

As discussed in the last section, the Research Councils arose as expert panels to advise government on how to fund research outside government departments. For most of the twentieth century, they constituted a relatively small part of the government’s overall R&D effort. However, in recent years their relative importance in the system has grown – my plot shows the fraction of government R&D expenditure managed by the research councils from 1986 onwards. This shows a marked increase from 1986 to 1994, reflecting the run-down of government applied research in the late Thatcher period, and a further increase in the early 2000’s, reflecting the New Labour government’s simultaneous increase in research council budgets and decrease in departmental R&D budgets.

Fraction of total government R&D spending in the control of research councils. Data: ONS Research and Development expenditure by the UK government (2009 – 2019), BIS Science Engineering and Technology Indicators (before 2009).

As the research councils have become more dominant in the UK’s research system, their visibility – and the expectations placed on them – has increased. This has led to a progressive widening in the range of organisational goals they have.

For most academic researchers, the core goal of the research councils is to keep research in the academic disciplines moving forward. The very organisation and naming of research councils reflects this; the seven research councils are arranged on a disciplinary basis, BBSRC, covering biology & biotechnology, MRC, focusing on medical research; EPSRC covering chemistry, maths, engineering, and physics, (apart from astronomy & high energy physics, which are covered by STFC); NERC ,environmental science; ESRC, social science; and AHRC the arts and humanities.

The core mechanism for achieving this goal is called “responsive mode” – essentially, inviting proposals from researchers and seeing what comes in, allocating funding to the best proposals as judged by peer review. The difficulty of running an effective peer review process shouldn’t be underestimated – quite a lot of domain knowledge is needed to be find the right referees for a given proposal, and in putting together expert panels to rank a batch of proposals in a single sub-discipline on the basis of the referees’ reports.

Much of the most innovative research is not to be found in single disciplines, though, but where the insights of different disciplines are brought together. But interdisciplinary research is more difficult to judge and support; there’s always the potential for proposals to potential for proposals to fall between the cracks. The cracks can be between the single discipline based panels within research councils; for interdisciplinary proposals, it’s all too common to find referees who will dismiss the part of the proposal they understand, while failing to see the added value that bringing together ideas and methods from different disciplines.

This is perhaps even more difficult for proposals that fall between the remits of different research councils, where the budgetary incentives are against such research. Take a proposal to apply machine learning to a problem in medical science; it’s all too tempting for the Medical Research Council to say, this is all very interesting, but it’s the business of the Engineering and Physical Research Council to support machine learning proposals (and it should come from their budget), while the EPSRC says the same thing in reverse.

In the past, interdisciplinary proposals were handled through cross-council programmes overseen by the coordinating body RCUK. Most of these cross-council programmes were motivated by societal issues rather than academic priorities; they can be thought of as “mission-driven” research.

Following the formation of UKRI, the Strategic Priorities Fund was introduced to fund such proposals in thematic areas – my example would have fallen within the remit of “AI and Data Science for Science, Engineering, Health and Government”. Over the current budget period, according to the recent UKRI allocations document (PDF), the Strategic Priorities Fund is being wound down; there is not yet clarity in the recent UKRI strategy how cross-council research will be funded in the future.

Research Councils don’t just respond to the demands of the UK’s research community; they also actively shape the overall landscape. This is inevitable given their importance in the overall system; by what the Research Councils choose to fund, they influence the priorities both of institutions and individual researchers. If a university perceives that a particular discipline is no longer in favour with the research councils, they will be less likely to recruit academics in that area, while the choices of research direction of individual researchers will be influenced by what they think they can get funded.

The influence Research Councils have on the research landscape is inevitable. The issue is how purposefully the Councils use it, and how much their choices are informed by any kind of larger national strategy.

Beyond their role funding individual researchers and groups in universities, the Research Councils are also responsible for maintaining hard and soft infrastructures. Hard research infrastructures include research facilities that are too large for an individual research group to manage – for example high performance computing, large scale neutron and x-ray sources, telescopes and research ships; soft infrastructures, including large scale data sets and long-running observational programmes, are arguably less well cared for.

In addition, there are a number of national laboratories, often with very long histories, that have ended up under the care of individual research councils – for example the British Antarctic Survey and the British Geological Survey fall into NERC’s stewardship, while the BBSRC’s Pirbright Institute has been a leader in studying animal diseases for a century.

The issue here is that these institutes often have national strategic purposes distinct from the focus on the high-status discovery science that many in the academic community think should be the core function of the research councils. This puts the survival of these institutes in periods of tight budgets under pressure, and over the years a number of such institutes have shrunk, merged, or been transferred into the university sector, where they end up under inevitable pressure to conform to the norms of academic research. On the other hand, entirely non-scientific pressures to keep certain facilities funded can lead to political influences overriding scientific factors in decision making.

Research councils also play an important role at the upper end of the skills system, by supporting PhD programmes in universities. A high proportion of UK-domiciled PhD students are supported financially by the research councils, and recent years have seen this aspect of their work being put on a more formal footing, by the creation of “Centres of Doctoral Training”, often in collaboration with businesses, where the research training that has traditionally formed the core of a PhD programme is combined with training in more transferrable and business-focused skills.

The role of research councils in driving economic growth has become much more politically prominent in the last couple of decades as their relative importance in the overall research system has grown. The case for sustaining science spending during the early 2010’s period of austerity was based on its claimed importance for economic growth, and the increases in public spending that began in the May government were explicitly tied to the need to restore productivity growth to the UK economy.

The economic role of research councils operates both at a strategic level, shaping the research landscape to meet the needs of R&D intensive businesses, and at the operational level, encouraging research partnerships between academia and business in research projects and PhD training programmes, and promoting spin-out companies that use intellectual property created by the academic research they support.

This growth in prominence of the “impact agenda”, as it has come to be known, has been unpopular with many academics, who interpret it as a shift away from basic, discovery science in favour of more applied research. But it’s more accurate to see it in the wider context of the way much larger government applied research programmes were run-down in the 80’s, 90’s and 00’s, and the perceived vacuum that left.

To sum up, Research Councils have the following core goals:
1. Advancing disciplinary-based research through competitively awarded research grants
2. Promoting interdisciplinary and mission-driven research
3. Shaping the UK’s overall R&D landscape
4. Maintaining an infrastructure of research facilities and institutes
5. Driving economic growth by supporting research in collaboration with UK businesss and promoting spin-outs exploiting IP developed through research they support
6. Maintaining a pipeline of highly skilled people by supporting PhD programmes

In balancing these goals, they need to satisfy four quite different constituencies.

• They need the research community, as the source of insights about the directions in which science and technology are heading, to provide the expertise that the peer review system depends on, and, most importantly, as the people who actually carry out the research they fund.
• But they have to respond to the government, as the organisation that writes the cheques, making the case for supporting research, in competition with the many more pressing political priorities that governments may have.
• If they are to be convincing in their arguments that the research they support contributes to economic growth, they have to work in partnership with the businesses that can turn research progress into new and improved products and services.
• And they must reflect the wider values of society.

It is a challenge to create a structure which maps onto this many goals, and which needs to respond to such a wide variety of stakeholders.

5.3 UKRI: Innovate UK

Besides the research councils, UKRI now incorporates the innovation agency Innovate UK. This was formed as a free standing agency in the late 2000’s, as the Technology Strategy Board, taking a number of funding instruments for R&D in business from the Department of Trade and Industry. Now it operates a combination of sector based networking organisations (the Knowledge Transfer Networks), small scale collaborative grants for industry and academia (Knowledge Transfer Partnerships), and larger scale grants for business R&D (including the SME focused, responsive SMART grants). It also is responsible for core funding of the Catapult network of translational R&D institutions.

The key principle underlying Innovate UK is that it is “business-led”. This puts clear water between it and the research councils, with their focus on funding research in universities. But it does lead to some tensions and dangers of its own. A close connection to businesses in R&D intensive sectors can lead to the danger of capture by incumbents, and raises the question of who speaks for emerging companies and sectors.

It would be natural to think of Innovate UK as a vehicle for implementing an industrial strategy (and its original name – the Technology Strategy Board – reflects this). A good case can be made that its sustained support for the automotive sector has played a significant role in that sector’s relative recovery.

But being perceived as an instrument of industrial strategy carries political risks. Innovate UK received a significant setback as an organisation in the first period of the 2015 Conservative majority government, when a Secretary of State opposed in principle to the idea of industrial strategy – Sajiv Javid – imposed significant cuts, and introduced a policy of replacing grants by loans.

A more fundamental question remains: who or what is a business-led innovation agency like Innovate UK for? One doesn’t go far in discussions like this without hearing the phrase “UK plc”, and the assertion that the role of Innovate UK is to make sure “UK plc” benefits from new technology.

But there is no UK plc. Fifty years ago, one might have talked about a national capitalism consisting of major industrial concerns based in the UK, quoted on the London stock market and largely owned by UK residents or their fund managers and pension funds. But we left that world behind in the 80’s and 90’s, when the UK embraced globalisation with an enthusiasm unmatched anywhere else.

Today, around half of our business R&D is done by overseas owned firms; this is a very high proportion in comparison to other similar sized developed economies. Very few UK owned firms are to be found amongst world R&D leaders – according to the EU R&D Scoreboard, only two UK firms are in the world top 100, the pharma companies GSK and AstraZeneca.

Large UK technology intensive companies, like GEC and ICI, were broken up and sold in the early 2000s. Exit and entry of companies isn’t a bad thing in a dynamic economy, but the UK hasn’t done well in sustaining and growing new companies. In ICT, the chip design house ARM was sold to the Japanese fund Softbank in 2016, while AI start-up DeepMind was bought by Google in 2014. In life sciences, the Cambridge spin-out Solexa, which developed the currently dominant technology for sequencing DNA, was bought by US company Illumina in 2007. A next generation sequencing technology has been developed by Oxford Nanopore, which remains a rare example of a non-software technology start-up determined to scale-up as a UK owned, UK based company, but its R&D investment remains about a factor of ten less than Illlumina’s.

The trajectory of two privately held companies is instructive. The electrical goods company Dyson was founded in 1991, and while it maintains significant manufacturing and R&D presence in the UK, it moved its headquarters to Singapore in 2019, together with a significant fraction of its R&D and engineering effort. The chemical company INEOS emerged from a buyout of the commodity chemical operations of ICI and BP; it moved its HQ from the UK to Switzerland in 2010 for tax reasons. It did move its tax domicile back to the UK in 2016, but it is today a global company whose manufacturing and R&D are mostly now in overseas locations.

So, with the UK’s industry base so dominated by multinationals with little or no natural allegiance to the UK, what is the role of a business-led innovation agency? Given the very high dependence of the UK’s innovation system on R&D carried out by overseas owned firms, Innovate UK’s role in attracting inward R&D intensive investment and keeping it anchored in the UK remains important. A focus on supporting new companies in scaling up is also crucial, but the possibility of these companies relocating to the USA or mainland Europe is a constant risk – and such a move may be entirely logical from a business point of view, by giving access to bigger markets and deeper ecosystems.

On the other hand, a new focus on resilience and security of supply, driven by the experience of the pandemic and much more threatening geopolitics, presents a whole set of new challenges for an innovation agency. While an attempt to retreat into some kind of “Juche UK” vision of self-sufficiency is obviously doomed, there may be a need to purposefully build industrial capacity in a few key areas where that capacity has been lost – as we have already seen with vaccine manufacture. In this environment, Innovate UK may need move a little away from being business led, and be more proactive in leading business.

5.4 Place based research and innovation funding

In section 2.4 of this series, I discussed the geography of innovation, highlighting the very regionally unbalanced distribution of R&D spending in the UK, and the relation this has to the UK’s profound regional disparities in economic performance. This has been recognised by the government, with a commitment to an increase in R&D intensity outside the Greater Southeast being identified as “Mission 2” in the Levelling Up White Paper (PDF). To support this, UKRI has been given a new organisational objective, to “Deliver economic, social, and cultural benefits from research and innovation to all of our citizens, including by developing research and innovation strengths across the UK in support of levelling up”.

This is a more significant change than it might appear, because in the past the key elements of UKRI have been committed to a “place blind” approach to funding. For the research councils, the primary consideration has always been “excellence”, while Innovate UK and its predecessor the Technology Strategy Board has up to now, always focused on the innovation landscape at a national level. These agencies now have an instruction to “increase consideration of local growth criteria and impact in R&D fund design.”

The one part of UKRI that does have a track record of thinking about local and regional innovation systems is Research England. Research England was formed in 2017 from the part of the Higher Education Funding Council of England that dealt with funding research in universities. As its name suggests, its writ runs only in England. Its function is devolved in Scotland, Wales and Northern Ireland, exercised there respectively by the Scottish Funding Council, the Higher Education Funding Council Wales, and the Northern Ireland Executive’s Department of the Economy.

Research England is responsible for the formula driven funding discussed in section 3.2 of this series; it runs the “Research Excellence Framework”, and then administers the formula by which the results of this exercise are converted into block grants to universities. In addition, it awards strategic funding for research infrastructure.

Research England has been responsible for delivering a specifically place-based funding mechanism, the “Strength in Places Fund”. The aim of this is to support existing or emerging innovation clusters across the UK (including in the devolved nations). After two funding rounds, to a total value of £316m, UKRI has decided not to continue this scheme beyond the 12 currently supported projects.

This means the only currently open explicitly place based intervention is the “Innovation Accelerator” pilot programme announced in the Levelling Up White Paper. In this £100 m is split between three city regions, Greater Manchester, Glasgow and West Midlands, “intended to boost economic growth by investing in R&D strengths, attracting new private investment, boosting innovation diffusion, and maximising the economic impact of R&D institutions.”

In practise, Innovate UK’s Catapult Centres have played a significant role in developing regional innovation clusters. But this has happened largely in an unplanned way; developing regional innovation capacity has not been an explicit part of their mission. Eoin O’Sullivan and I argued in this paper that it should be.

Finally, it’s worth mentioning the important role the European Union’s structural funds have played in supporting innovation activities in economically lagging parts of the UK, especially Wales, Northern Ireland, Cornwall, and parts of northern England. These funds will be partially replaced by the Shared Prosperity Fund, though it’s still not clear how that will work in practise, and how much emphasis on innovation it will have.

5.5 UKRI four years on

UKRI formally came into being on April 1st 2018, a product of the 2017 Higher Education and Research Act; its formation was prompted by the 2015 Nurse Review. To what extent have the goals of the Nurse Review been realised?

The central recommendation that Nurse made was to merge the seven existing research councils into a single organisation. The Research Councils were government organisations, but with a degree of institutional autonomy conferred by their status as “Royal Charter” bodies. The research councils were formally independent of each other, but in practise they would present a common front to central government for spending reviews, and an umbrella organisation – “Research Councils UK” – acted as a coordinating body, developing joint interdisciplinary programmes.

The effect of the Higher Education and Research Act was to merge all seven research councils into a single body, with one accounting officer. Two other, rather different, organisations were also folded into the overall structure – Research England, with its systemic oversight and funding of research in English universities, and Innovate UK. As I discussed in section 4.4, the act imposes much more direct control from central government on UKRI than had been the case for the research councils.

What was the Nurse review trying to achieve? In part, it was to create a closer strategic connection between the research landscape and central government, with a single organisation being better able to engage with and influence departments across the whole of government. Other motivations were operational – “reducing the complexity and increasing the agility of operations”. There was a hope that a single organisation would reduce bureaucracy and strengthen governance.

But a key motivation was to break down the walls between different parts of the scientific endeavour – “Establishing mechanisms to deal with cross-cutting issues such as the support of multi-disciplinary and inter-disciplinary research, grand challenges and the redistribution of resource between Research Councils in response to new developments, advances and priorities in the research endeavour”.

How effective has UKRI been at achieving these goals? The government has just published an independent review of UKRI by Sir David Grant.

I don’t intend to summarise the findings of this report in detail here; it’s well worth reading in full. In short, there are findings both of operational shortcomings, and a lack of strategic coherence. One very worrying finding is a combination of high staff turnover with poor results from staff surveys; any knowledge-based organisation relies on the commitment of highly qualified and experienced staff.

The lack of strategic coherence is associated with a muddled organisational architecture. The Grant reports concern “about the extent to which the board makes strategic decisions around the direction of UKRI which then translate into meaningful activity within the organisation. For example, there is little evidence that UKRI has made strategic decisions to prioritise particular goals and the bulk of spending has not shifted between different councils, activities within councils or activities across UKRI.”

The role of the “councils” of the constituent Research Councils is now not clear. Before UKRI was established, these were, in effect, the governing bodies of the individual Research Councils; in UKRI they are in effect advisory bodies to each council, but their role within the wider organisation isn’t well defined: to quote the Grant review, “across UKRI, there are over a hundred council members sorted by domain expertise but with no clear way to engage with UKRI strategic decision-making and governance and with uncertainty over if they need to”.

How have these difficulties affected the way UKRI has operated? I would identify five key issues.

The first is that I don’t think that the promise of UKRI to improve support for interdisciplinary research has been realised. The Industrial Strategy Challenge Fund (ISCF), which did bring together research councils and Innovate UK in support of some interdisciplinary areas, had some successes, but is now being wound down. To quote the Grant Review again: “the potential for interdisciplinary research has not been fully realised. The most successful example is the ISCF which put new money into the system to support inter-disciplinarity. In practice, with most councils’ budgets committed into future years and systems that limit cross-council working, UKRI is unable to maximise the full potential for interdisciplinary research or transform the collective UK approach to this outside of specific programmes such as the ISCF.” Meanwhile, it was never clear how the eight themes and thirty four (generally small scale) programmes supported by the Strategic Priorities Fund were arrived at, and this fund is now being wound down with no clarity on its successor.

The second is that there doesn’t seem to have been much integration between Innovate UK and research councils. As the Grant Review says, “the advantages of having Innovate UK within UKRI have not been fully realised. With the exception of specific programmes such as ISCF we note that there have been examples and pockets of joint working between councils and Innovate UK, however this was often driven by passionate individuals and not by a strategic plan.” Innovate UK’s new plan for action barely mentions the research councils, making few connections between its own technology priorities and the upstream science priorities of the research councils. Meanwhile the research councils have their own priorities for engagement with industry, both in the university research they support and in their own institutes and research campuses, but there is a risk that this is seen as being in tension with a lead role in innovation for Innovate UK.

The third is a patchy degree of connection between skills policy and innovation policy, which reflects some wider difficulties in policy in England (the situation is different in the devolved nations, though here a lack of high level connectivity between UKRI and devolved nations causes other problems). The splitting of HEFCE into Research England, within UKRI, and a free-standing Office for Students, conceptualised as a regulator of higher education as a consumer service, means that no-one owns responsibility for the HE system in England as a deliverer of the skills needed for the innovation agenda. Historically Innovate UK has not regarded skills development as being part of its brief; there is some change here, with more involvement of the Catapult Network with regional skills systems, but this is hampered by the disconnect between BEIS and its agencies and a chronically neglected FE sector. Only in the provision of PhD training is there evidence of UKRI being able to take a more holistic view than its predecessors.

Fourthly, there still seems to be some lack of conviction within UKRI on addressing regional imbalances in R&D. If nothing else, the signalling doesn’t look good; as we’ve seen, UKRI’s only dedicated instrument for place-based R&D up to now, the Strength in Places scheme, is being wound down, with around £70m a year allocated for continuing funding of existing programmes. The three new “Innovation Accelerators” are allocated £50m a year, but only for two years, with no commitment to continuation beyond 24/25 or to expansion beyond the three cities funded in the pilot scheme. These figures look like very small commitments in the context of an £8 billion/year budget. If the emphasis now is going to be on adapting existing programmes to deliver UKRI’s new organisational goal, of “developing research and innovation strengths across the UK in support of levelling up”, there needs to be some clarity about how this is going to work in practise.

Fifthly, and a little more tentatively, I do sense a decreasing level of confidence in the wider scientific community in the ability of the research councils to run a credible peer review system, that does manage to support excellence in the core disciplines. One symptom of this is, perhaps, the great anxiety in the scientific community about the UK being cut off from the European Research Council, and the lack of confidence in the community that UKRI could run a credible replacement. I’ll discuss the ERC more in the next section; it does offer some important lessons about what it takes to run a credible and effective peer review system, which is more difficult than it might first appear.

Finally, what are the broader implications of the way in which the Higher Education and Research Act removed the autonomy of research councils, giving government more direct control over them? The goal was to make the system more responsive to the strategic goals of the government, and in turn give the science community a stronger voice in influencing those strategic goals. But the risk was that it would hobble the research councils’ freedom to operate and experiment, by imposing more Whitehall bureaucracy.

We’ve certainly seen quite a lot of the latter. According to the Grant Review, “UKRI reports receiving a high volume of ad-hoc requests from government”, and “UKRI has identified a non- exhaustive list of 40 different reports they must produce for government either annually, quarterly or monthly”.

The impression is of a whole set of extra hoops UKRI is made to jump through, absorbing management attention and creating friction and delays. Again, from the Grant Review: “the business case for the second wave of COVID-19 funding went through UKRI approvals in a week, BEIS in two weeks and HMT in six weeks consecutively, which is less than ideal in an emergency response situation”, and “UKRI’s SHARP programme must go through internal controls in addition to external assurance from four separate organisations (GIAA, IPA, BEIS Portfolio Office Gateway Reviews, CDDO) and approvals from BEIS commercial board, BEIS investment board, and ministers from BEIS, Cabinet Office and HMT”.

Yet there doesn’t seem to be much evidence of a strong strategic connection to government priorities that is influencing the operation of UKRI. Once again, the Grant Review comments that “there is little evidence that budget allocation advice from UKRI is made on a clear analysis of its goals and what the right allocation is to achieve those goals.”

What has happened by the removal of autonomy, though, is that UKRI is more exposed than the research councils were to rapid political shifts, due to the inability of recent governments to sustain consistent policy over the long term.

For example, in November 2020 the government announced that it was suspending the target of spending 0.7% of GDP on foreign aid. This led to large and abrupt cuts to UKRI’s Global Challenges Research Fund, which supported collaborative R&D with developing countries in support of international development. This in turn led to many grants being cut-off in midstream, and substantial damage to the UK’s international reputation as a reliable research partner.

One way in which there had been a connection between UKRI programmes and wider government strategy was through the Industrial Strategy Challenge Fund, which responded to priorities set in the 2017 Industrial Strategy White Paper and did bring together research councils and Innovate UK in support of some interdisciplinary areas. But this industrial strategy was in effect replaced in March 2021 by a Treasury driven “Plan for Growth”, with a subsequent Innovation Strategy defining priority “technology families”. The Industrial Strategy Challenge Fund is now being wound down, with little clarity on what might replace it.

Perhaps we will now enter a period of political stability, where long term priorities, informed by the science and innovation opportunities identified by UKRI, are set by government, and these in turn set long term directions for the UK’s public research enterprise. Maybe the long-term missions and 2030 goals defined by the Levelling Up White Paper will form a basis for some of these directions. Perhaps the new National Science and Technology Council (about which, more later) will give a clearer way of connecting UKRI strategy with wider government priorities. We shall see.

In the next section of this series, I will move on to consider the other ways in which the UK government supports science, covering other spending departments, the new agency ARIA, and EU R&D programmes – and whatever might replace them, in the increasingly likely event that the UK does not associate with Horizon Europe.

In the first part of this series attempting to sum up the issues facing UK science and innovation policy, I tried to set the context by laying out the wider challenges the UK government faces, asking what problems we need our science and innovation system to contribute to solving.

In the second part of the series, I posed some of the big questions about how the UK’s science and innovation system works, considering how R&D intensive the UK economy should be, the balance between basic and applied research, and the geographical distribution of R&D.

In the third part, I discussed the institutional landscape of R&D in the UK, looking at where R&D gets done in the UK.

In this part, looking at the funding system, I consider who pays for R&D, and how decisions are made about what R&D to do.

4.1. The broad flows of research funding in the UK

The flow diagram I reproduced in my last post summarises the overall way in which R&D is paid for in the UK. In 2019, total spending on R&D was £38.5 billion. The largest single contribution to this was from business, which spent £20.7 billion, mostly on R&D carried out in the business sector.

The government spent £10.4 billion, including £1.8 billion to support R&D in industry, £6 billion on university R&D, and £2.3 billion in its own laboratories.

Overseas sources of funding accounted for £5.5 billion. £1.5 billion of this overseas money went into universities; of this, I estimate around half of this came from the EU (and is thus properly thought of as originally coming from the UK government), with the rest from overseas companies, charities and other governments.

Finally, £1.8 billion was spent by the non-profit sector, dominated by the Wellcome Trust and medical research charities such as CRUK.

It’s worth adding two glosses to these official figures. Firstly, businesses receive a substantial subsidy for their R&D spending through the mechanism of R&D tax credits. These were worth £7.4 billion in FY 19/20. Although there isn’t an exact alignment between the R&D tax credit statistics and the Business R&D statistics, we can estimate, putting together the cost of tax credits and direct government funding of industry research, that roughly 35% of business spending on R&D is ultimately paid for by the state.

Secondly, as we discussed in the last section, research carried out in universities isn’t fully funded by the government, but in effect is cross-subsidised by other activities, especially teaching overseas students. It’s difficult to precisely quantify this additional contribution to university-based research, but it’s likely to be of order an additional £1 billion across the whole HE sector.

4.2. Who should decide what science is done?

Science funding is about making choices and deciding priorities. Who, in principle, should be making these decisions?

(a) Scientists. One view is that it is only that scientists who are in a position to judge the quality of the work of other scientists, and to make informed choices about what science should be done. This view underlies the prevalence of “peer review” as a mechanism for judging the validity and quality of scientific publications, and the practical procedures by which science project proposals are judged and ranked by science funding agencies. Typically, a project proposal will be sent to referees from the science community, who will make a critique of the proposal, and a panel will rank a set of proposals by reference to these referees’ reports.

From a practical point of view, the argument is that it is only expert, practising scientists who are in a position to assess the novelty of a proposal in the context of the existing body of scientific knowledge, and who can make a judgement of a proposal’s technical feasibility. The potential counterarguments are that reliance on the judgement of other scientists promotes conservative, consensus-driven research rather than projects with truly transformative potential, and disadvantages cross-disciplinary research, because of the difficulty of finding potential referees whose expertise ranges across more than one area.

This view was given an ideological framework by Michael Polanyi, who compared the international scientific enterprise at its best as a “market-place of ideas” in which the best and most profound ideas would naturally prevail. In this view any attempt by non-scientists to steer this “independent republic of science” is likely to be counterproductive and destructive. This point of view is popular with elite scientists.

(b) The Government. On the other hand, if the reason the government funds science is because it believes this supports its strategic objectives, then one can argue that the government should direct science in ways that support those objectives. In fact, over the last century and a half, this is exactly what has happened for most government supported science. The most pressing strategic goal behind which the government has directed science has always been military power; behind that at various times support for agriculture, for colonial activities, and for civil industry has also been prominent.

(c) The people. In a democratic society, the government’s support for science should reflect widely held societal priorities. There’s an argument that representative democracy doesn’t provide a very effective way of translating those societal priorities into decisions on science funding, simply because so many other issues – health, crime, the economy etc – are likely to be much more salient in influencing citizens’ votes. This makes the case for giving more direct forms of deliberative democracy – citizens’ assemblies and such like – a role in setting science priorities.

Often this has been framed in a defensive way, to head off potential public opposition to controversial new technologies. But there is case for thinking of the direct involvement of citizens in setting priorities in a more positive way, challenging expert group-think and bringing new perspectives to set a direction that commands widespread public support.

(d) The market. According to many economists, if you want to find out what people want, you should look at what they do, not what they say. In this view, the true test of whether people want some innovation is whether they buy it. Following Hayek, one can regard the market as the most efficient way of aggregating information about societal wants and needs. In this view, the government should simply step out of the way, and let private firms explore the space of possible innovations, with the market deciding which are successful and which not.

The difficulty with this view is that many radical innovations need large investments to get to the point at which they can be brought to market, with no certainty not just as to whether demand for them materialises, but as to whether they will work in the first place.

There’s a more general point here; what works for applied research, with a clear and relatively short route to commercialisation, is likely to be less useful for more basic research, where any applications are highly uncertain, unpredictable and often don’t manifest themselves for many years after an initial discovery.

4.3 What kind of science policy choices are we talking about?

In thinking about who makes decisions about what kind of science gets done, and who influences those priorities, we should distinguish between some different levels of decision-making.

We’ve seen major strategic shifts – for example the shift from applied research to “curiosity driven” research in the late Thatcher government. In effect this involved shifts of many £billions, and was driven from the centre of government, under the influence of a single powerful advisor (George Guise, in that case). The post-cold war shift of emphasis from defence to health and life sciences was on a similar scale, though this is probably more difficult to pin on a single individual or agency.

On a slightly smaller scale, we have long-term strategic programmes, with funding at the level of £100m’s. Examples of this could include the fusion programme, recent initiatives on batteries and quantum technology, and the new funding agency ARIA. Here the initiative usually does come from some part of central government, with Government Chief Scientific Advisors and other influential policy actors (for example, in the recent case of ARIA, Dominic Cummings) often being a driving force.

Programmes at the level of £10m’s have generally been initiated by research councils, though they may form part of the research councils’ pitch to government in the budget negotiations around spending reviews. Priorities at this level may emerge from the scientific community through the various advisory bodies that research councils draw on; there may also be an element of research councils anticipating what they think the government of the day is interested in, whether that is driven by formal government strategy documents or more informal interactions with key actors.

Within these programmes, it is the individual projects that are awarded to researchers that are awarded through peer review.

4.4 The “Haldane principle” and the political independence of science funding agencies

How closely should politicians be able to direct research priorities for government funded science? The conflict between the long-term nature of science and the short-term imperatives of electoral politics has long been recognised, and makes the case for inserting some distance between science funding agencies and central government. It’s not just in science that this conflict between the long term interests of the state and short term electoral politics is recognised; the decision to give the Bank of England the power to set interest rates independently of the Treasury presents an analogy. In the UK the symbol of this distance in science policy is a semi-mythical arrangement known as the “Haldane principle”.

The Haldane principle is interpreted in different, often self-serving, ways by different constituencies. Some scientists interpret it as meaning that the government should have no involvement in any aspect of science policy, apart from signing the cheques. For government ministers, on the other hand, it legitimates their right to make big funding announcements while leaving operational details to others. The historian David Edgerton has stressed (see e.g. The Haldane Principle and other invented traditions in science policy) its relatively recent rise to prominence in science policy discourse, and the fact that most government funded science has never been within its orbit.

The origins of the “Haldane principle” are purported to lie in an important and influential report from Lord Richard Haldane published in 1918 . This defined many of the principles by which the modern civil service is run, including principles for both the way state-funded science should be administered and the way scientific evidence should be used in government.

The principle that the government should be involved in science had been established in the late 19th Century, through reports such as that of the Royal Commission on Scientific Instruction in 1870, and the establishment of institutions such as the National Physical Laboratory and the Laboratory of the Government Chemist. The First World War brought new urgency to government driven science, both to meet the technological demands of the new industrial warfare, and to accommodate the medical demands of dealing with its terrible human cost.

This was the context of the Haldane review, which brought attention to the slightly ad-hoc way in which different government departments had ended up supporting scientific research in support of their various goals. The report focused on two new bodies that had arisen to deal with these pressures; the Medical Research Committee and the Department of Scientific and Industrial Research. In each case a pattern had been established – a minister taking responsibility, but with decision making devolved to a committee of experts, taking advice from a wider advisory council. This did set the pattern for something like a modern research council, and indeed the Medical Research Committee morphed into the Medical Research Council, which survived until its incorporation into UKRI in 2018. Other research councils – first the Agricultural Research Council, followed. But the focus of the Haldane recommendations was on the best way to bring expert advice to bear onto the problems of government, rather than any principle of scientific autonomy.

As David Edgerton has stressed, in the postwar period, the research councils were relatively small parts of an overall R&D system dominated by the requirements of the “Warfare State”. One major innovation was the introduction of the Science Research Council in 1965, which first evolved into the Science and Engineering Research Council, and then was broken up following William Waldegrave’s 1993 White Paper. This co-incided with the big shift in UK policy I’ve referred to before, where the state substantially withdrew from applied research. The 1993 White Paper did invoke a “Haldane principle”, but reasserted a right for the government to make strategic choices: “Day-to-day decisions on the scientific merits of different strategies, programmes and projects should be taken by the Research Councils, without Government involvement. There is, however, a preceding level of broad priority-setting between general classes of activity where a range of criteria must be brought to bear.”

The government asserted much more direct control over the research system in the 2017 Higher Education and Research Act, which incorporated all seven research councils into a single organisation, UK Research and Innovation (UKRI). The act does give a nod to a “Haldane principle”, which it defines in a rather diluted form: “The ‘Haldane principle’ is the principle that decisions on individual research proposals are best taken following an evaluation of the quality and likely impact of the proposals (such as a peer review process).”

However, the act makes explicit where it thinks power should lie. Section 102 of the Act states, “The Secretary of State may give UKRI directions about the allocation or expenditure by UKRI of grants received…”, and, in case the situation isn’t already clear enough, “UKRI must comply with any directions given under this section.”

My own view is that the government does have a right – indeed, a duty – to steer the overall science enterprise in support of the strategic goals of the state. I discussed some of those big issues in the first section of this series – the need to return to productivity growth, to manage the energy transition to net zero, to keep the nation secure in a hostile world, to support the health and well-being of its citizens. The government has given itself the power to do this.

The danger, though, is that nobody does the strategy, but that instead governments succumb to the temptation of micromanaging the implementation for short-term political advantage.

To come in the next instalment: on the bodies that fund science in the UK – UKRI, the research councils, Horizon Europe (and whatever may replace it). How well do they work, what challenges do they face?