Humility engineering is the process of recognizing a systematic human fallibility—cognitive or behavioral—and designing a method to account for it, rather than relying on the person to overcome it.

Humility engineering is a term that I coined; it is not (yet) a term of art. The activity it describes, however, is not new. Many fields, including accounting, aviation, medicine, nuclear engineering, and process safety have developed methods and institutions to account for observed sources of human error. But science is the field where the most humility engineering has been done. The scientific method is not a simple question-hypothesize-experiment-analyze loop. It is not a collection of facts arranged by category, as it is often presented in science museums. It is, at its core, a collection of safeguards engineered to help scientists (human beings) avoid fooling themselves.

The tools of science—the safeguards which are the products of humility engineering—are the ideas at the heart of the Behrens Science Museum. Facts are interesting, but how we know those facts, and how we can be confident that we haven’t fooled ourselves into believing them, that is the most interesting and most poorly understood part of science.

To highlight the point, let’s consider Richard Feynman. Feynman won the Nobel Prize in 1965 for his work on quantum electrodynamics, the most precisely tested theory in physics. On June 14, 1974, Feynman gave a now famous commencement speech, “Cargo Cult Science” at Caltech. Here is an excerpt of his speech:

We have learned a lot from experience about how to handle some of the ways we fool ourselves. One example: Millikan measured the charge on an electron by an experiment with falling oil drops and got an answer which we now know not to be quite right. It’s a little bit off, because he had the incorrect value for the viscosity of air. It’s interesting to look at the history of measurements of the charge of the electron, after Millikan. If you plot them as a function of time, you find that one is a little bigger than Millikan’s, and the next one’s a little bit bigger than that, and the next one’s a little bit bigger than that, until finally they settle down to a number which is higher.

Why didn’t they discover that the new number was higher right away? It’s a thing that scientists are ashamed of—this history—because it’s apparent that people did things like this: When they got a number that was too high above Millikan’s, they thought something must be wrong—and they would look for and find a reason why something might be wrong. When they got a number closer to Millikan’s value, they didn’t look so hard. And so they eliminated the numbers that were too far off and did other things like that. We’ve learned those tricks nowadays, and now we don’t have that kind of a disease.

But this long history of learning how to not fool ourselves—of having utter scientific integrity—is, I’m sorry to say, something that we haven’t specifically included in any particular course that I know of. We just hope you’ve caught on by osmosis.

The first principle is that you must not fool yourself—and you are the easiest person to fool. So you have to be very careful about that. After you’ve not fooled yourself, it’s easy not to fool other scientists. You just have to be honest in a conventional way after that. 

Feynman’s story about the measurement of the charge of the electron illustrates the problem of “researcher degrees of freedom.” Many choices have to be made in deciding how to analyze data. If any of those choices are made after looking at the data, there is no way for anyone—even the researchers themselves—to know if the choices they made were influenced by the data. That explains what happened with the measurement of the charge of the electron, despite our near certainty that the scientists making the measurement wanted to be accurate.

In the second paragraph Feynman says something really interesting: “It’s a thing that scientists are ashamed of—this history…” He says that scientists are ashamed! Shame is “a painful emotion caused by consciousness of guilt, shortcoming, or impropriety.” But he concludes the paragraph saying, “We’ve learned those tricks nowadays, and now we don’t have that kind of a disease.” Problem solved! Now (1974) scientists know better than to make those kinds of mistakes!

Feynman specifies the mistake in the third paragraph. It is “fooling ourselves.” And he equates “not fooling ourselves” with “utter scientific integrity.” Integrity that is not taught in a course. Integrity that is absorbed through “osmosis.”

The last paragraph holds Feynman’s famous line, “The first principle is that you must not fool yourself—and you are the easiest person to fool.” Even in context, it’s not exactly clear what the first principle refers to. Is it the first principle of science? Out of context, hat would be my guess. However, in context, I think he’s saying that it is the first principle of scientific integrity.

So, the entire excerpt is about “how to handle some of the ways we fool ourselves.” And Feynman’s answer seems to be that it is your job as a scientist. You must be careful.

The problem with Feynman’s advice is that scientists are human beings. They make mistakes just like everyone else, and, of course, they don’t know when they’re fooling themselves. There is no shame in being human. The solution is not to expect people to have superhuman “integrity.” The solution is to recognize our limitations and design safeguards around them. Science is a collection of safeguards: blinding and double-blinding, control groups, randomization, placebos, statistical tests, replication, peer review, and many others. Each of these exists, at least in part, because someone discovered a way in which scientists have fooled themselves.

Bruce Behrens
Humility Engineer (in training)