How Do We Make Decisions?

Humans are near constant decision-makers. Every day, we make upwards of 35,000 decisions; 285 of which are about food. We like to think we are rational actors who can parse through information to arrive at the best or right answer, but what if we don’t always think before we decide? In 2002, Daniel Kahneman won the Nobel Peace Prize for his work with Amos Tversky on human judgement and decision-making under uncertainty. They demonstrated how we humans use heuristics, or mental shortcuts, to come to judgments quickly, without having to think too hard. We do it unconsciously: it’s just the way the brain works. Much of the time it works very well for us, but sometimes it causes us to be wrong. Why? Because those shortcuts have biases to them. He identifies:

Kahneman says it’s not that we are incapable of thinking more deeply to make a better decision, but that we rarely turn on that mode of our brain -- a slower, more effortful, and intentional mode -- if we can avoid it. These biases are employed intuitively and without reflection.

Of the 35,000 decisions we make a day, not all are of the same type. Some are personal preferences: chili sauce or ketchup? Others are moral in nature: give $5 to someone who asks? How we make moral decisions also comes down to default logics, informed by our sense of distributive justice.* Podcast guest Josh Rottman studies the logics children and adults use by default when making distributive decisions, and whether it’s possible to shift those defaults. (Spoiler: it’s a lot easier in children!)

Ferric C. Fang and Arturo Casadevall looked at how granting committees at the USA’s National Institutes of Health (NIH) decided which research projects to fund. Fang and Casadevall wanted to know whether grantmakers were able to identify the applications that would go on to produce greater value to the scientific community. The output of research projects is academic articles, so they defined successful projects as ones that produced papers that are more frequently cited.

Turns out, grantmakers weren’t any better at selecting successful projects than if applications were chosen at random. A number of research studies have confirmed grantmakers’ poor predictive skills. Why? Too few reviewers per application leads to random scoring; reviewers may not agree on criteria or their weighting; and discussion panels are often ineffectual, failing to improve the reliability of decisions.

• 
  While experts were unable to make good predictions, they vastly overestimated their ability to do so. Reviewers were able to differentiate between the strongest and weakest applications, but they found few good reasons to differentiate between projects in the middle of the pack.
• 
  Experts trend cautious in climates of resource scarcity. One tension affecting reviewers was the mandate to select potentially revolutionary research (which requires risk taking) but with limited dollars and high demand. As Nobel Laureate Roger Kornberg has observed, “In the present climate especially, the funding decisions are ultraconservative. If the work that you propose to do isn’t virtually certain of success, then it won’t be funded. And of course, the kind of work that we would most like to see take place, which is groundbreaking and innovative, lies at the other extreme.”
• 
  Even small amounts of personal bias in reviewers can have a significant impact on funding outcomes.

In the end, Fang and Casadevall make the case for the NIH to switch to a modified lottery to decide which research proposals should be funded.

A modified lottery is a method for maximizing the good reasons we have to make a decision, while sanitizing a process of the bad reasons or biases. It looks like applying filters to a pool of candidates before conducting a lottery, or sorting candidates into separate lotteries (eg. to ensure a particular representation of geographic zones). In the case of the NIH grantmakers, they were most reliably able to identify proposals that were infeasible, badly conceived, or unable to advance knowledge/practice. Reviewers also did better at identifying the strongest proposals. The remaining, middle-of-the-pack applications would be entered into a lottery to randomly determine what will get funded.

Research suggests that the results will, on average, be just as good as if the reviewers made the decisions. The results will also be more time efficient, and without the unintentional introduction of bias, or sending misleading signals to applicants about the quality of their proposal compared to others’.

A modified lottery approach might seem alarming in a context where grant review committees have been an important way to have community oversight of decisions. However, in a modified lottery there is still a role for humans to use their knowledge to make the best decisions, without asking people to go beyond what they feel sure about. Institutions that use modified lotteries may also transfer their resources to another stage in the process: doing more outreach to attract applications from underrepresented groups, or supporting first-time successful applicants to do their best work.

• 
  Mental shortcuts & moral decisions don’t mix

  Humans make a ton of decisions, so we develop mental shortcuts to help us avoid thinking too hard. But these shortcuts produce decisions marked by unconscious bias, so we should slow down our thinking for more consequential choices.
• 
  Questions of distributive justice are moral decisions

  Distributive justice is about how we decide the fairest way to divvy up burdens and benefits (like who pays taxes at what rate, or who gets seconds at dinner) and these decisions get to the core of our moral values
• 
  Unconscious bias creeps in more than we think

  The research is in: We overrate our ability to make sound decisions in complex situations where we don’t have all the information. Our biases can prevent us from perceiving relevant information, and influence our choices without our knowing.
• 
  Lotteries can be fairer than humans

  In a granting scenario, once we’ve run out of good reasons to select or reject applications, it might be fairer to decide the rest by lottery - keeping bias out of the process.

  
  

Experiences & Observations

What do you think of as the most important and consequential decisions you make in your work? Thinking of a decision you’ve been part of, to what extent, and in what ways, was the process and outcome informed by organizational values and purpose? Thinking about decision-making as it is practiced in the social sector, where have you seen or experienced the most transparent, values & purpose-led practice? Describe what has impressed you.

Reactions & Impressions

How do you feel about Fang and Casadevall’s recommendation of a modified lottery process as applied to grantmaking? What are some of your worries and curiosities?

Questions & Hunches to test

Next time you are in a group decision-making process, how might you test ways for participants to reflect on and gracefully note when they are coming to the end of their good reasons to make a choice (ie. those that do not rely on bias?)