Let the Data Do the Talking

Human history has been defined by scientific innovation, from the invention of electric lights to the dawn of modern medicine. Now, in an age defined by complex crises such as climate change and the COVID-19 pandemic, scientific progress is more important than it’s ever been. It’s only through research and innovation that our societies will be able to navigate the challenges ahead: developing cleaner forms of energy, distributing vaccines globally, growing enough food to feed a growing population, and so on.

But nothing in life is free, and science is no exception. In Canada, the Federal Government provides hundreds of millions of dollars of funding for critical research and development (R&D) infrastructure each year, driving innovation and economic development. The National Research Council (NRC) is responsible for strategically allocating this funding, parceling it out to around 150 research facilities across the country. 

These choices have massive consequences, for people in Canada and beyond. Research projects need funding from bodies like the NRC in order to get off the ground, and if high-potential projects are overlooked, everybody misses out  — imagine what might have happened if so many public agencies hadn’t funded the development of COVID-19 vaccines like the Oxford–AstraZeneca shot.¹ But with so many problems in the world clamoring for attention, deciding how to allocate research funding is no simple task.  

The problem with this decision-making process is that it involves an overwhelming amount of information. Before the NRC can decide who should get how much funding, they first have to score each facility along 11 different dimensions, looking at attributes like how unique their work is in the research ecosystem, the utilization rate of their research, and so on. They also have to make some tricky decisions about how to weigh all these different concerns — which considerations are most important? How can these determinations be made in a way that’s not biased? 

In the past, the NRC had made these decisions by gathering stakeholders together for a series of workshops, where everybody make their case about how they thought funding should be distributed. Unfortunately, beyond being time-consuming and frustrating, this process of debate isn’t so conducive to evidence-based decisions. Instead, participants are more likely to end up compromising and bargaining with each other to ensure that their strategic priorities are represented, regardless of what the data says.

TDL partnered with the NRC to help them take the human error out of their investment decisions. On top of making the funding allocation process more efficient and less maddening, the approach we developed will also help maximize the impact of R&D spending in Canada, and ensure that the country’s best researchers have the resources they need to create real change.

At the end of the day, the problem NRC was facing stemmed from simply having too much data to work with — more than our human brains are capable of managing all at once. Being the mortals that we are, we have limited cognitive bandwidth when it comes to processing large amounts of data, and we’re prone to information overload. This not only makes the decision-making process more stressful and inefficient, but it can also introduce cognitive bias into the mix, skewing our choices even further. 

It followed that the key to smart decision-making was to simplify, condensing all of the information that NRC was working with into a single score that would be easier to work with. TDL built out a customized methodology for integrating all of the NRC’s facility evaluation metrics into a single composite metric, weighted according to their top strategic priorities. We crash-tested our system in a workshop with 12 of NRC’s director generals (DGs), and reviewed by the Senior Executive Committee.

Trying to choose among 150 options using 11 variables is chaos. But what if we were to show you only two options, compared across 3 variables? Much easier. 

This is the first thing we did to figure out how we should weigh all of the NRC’s different strategic concerns. We presented all of the DGs with 2 hypothetical research facilities to choose between, alongside their scores on each of the 11 dimensions. Over 5 sessions, we collected over 2500 data points, which we used to compute both the level and stability of DG preferences for each of the 11 criteria. 

This let us identify a few key areas where there was a lot of variability — that is, where DG opinion was split. These were the pain points that were causing so many problems during funding workshops, and the issues that we needed to help the group reach consensus on. To get everyone onto the same page, we ran a brief workshop of our own, featuring evidence-based exercises for aligning stakeholder preferences. Over the course of an hour, we were able to bridge everyone’s preferences and reach a consensus, so that everyone in the room — some of whom had come in with diametrically opposed viewpoints — was happy with the model we had developed.

No matter how big your organization or what industry you’re operating in, data analytics is the key to making robust, informed decisions. Research has confirmed that highly data-driven organizations are more productive and more profitable than their competitors. 

But it’s not enough just to have access to data. Decision-makers also need to know how to make those data points work for them. That’s especially true for big, one-time decisions that need to consider a multitude of perspectives. In situations like these, an abundance of data can do more harm than good, unless decision-makers have a system to streamline and debias the process. 

A customized framework like the one we built for NRC takes the guesswork and cognitive overload out of complex decisions, without leaving organizational priorities or values behind. The result is a much smoother, more objective process for allocating hundreds of millions in funding, helping us speed up the pace of crucial innovation. 

1. Cross, S., Rho, Y., Reddy, H., Pepperrell, T., Rodgers, F., Osborne, R., Eni-Olotu, A., Banerjee, R., Wimmer, S., & Keestra, S. (2021). Who funded the research behind the Oxford–AstraZeneca COVID-19 vaccine? BMJ Global Health, 6(12). https://gh.bmj.com/content/6/12/e007321