617: Test your gut, don’t trust it –the scientific mindset for product teams – with Stefan Thomke, PhD

Experiments to inform product decisions Watch on YouTube

TLDR

Stefan Thomke, professor at Harvard Business School, and author of The Scientific Mindset: How Thinking Scientifically Drives Better Business Decisions, shares five elements of a scientific mindset, four types of uncertainty in innovation, and three tests of a good hypothesis. Learn why good product decisions are rooted in testing more and assuming less.

Introduction

When Stefan Thomke studied almost 22,000 online experiments (mostly A/B tests) run by more than 1,300 companies, only about one in five changes made a measurable difference. Half of the changes made things worse. If experienced teams can’t predict the effect of a small change, how confident should any of us be about a creating a new product?

Stefan is the William Barclay Harding Professor of Business Administration at Harvard Business School, where he has studied experimentation and innovation for more than three decades. He started as an engineer, learning experimental design to solve a production problem in a semiconductor plant. He is the author of Experimentation Matters and Experimentation Works. His new book, The Scientific Mindset: How Thinking Scientifically Drives Better Business Decisions, completes the trilogy.

Stefan was on the show in 2020 to explain how to run product experiments. This time we look at the thinking that decides which experiments are worth running and what you do with the answers. You’ll learn how to find the assumptions hiding in a new product idea using four kinds of uncertainty, how to turn an assumption into a strong hypothesis, and why “customers want this feature” isn’t yet a hypothesis.

Summary of Concepts Discussed for Product Managers

Five Elements of a Scientific MindsetStefan opens with a question: What did you do the last time you didn’t know what to do? He quotes Richard Feynman: “The first principle is that you must not fool yourself, and you’re the easiest person to fool.” The human brain is a belief engine. We see connections between unrelated things and confuse correlation with causation. To overcome this, we need to think scientifically about decisions.

Stefan identifies five elements of a scientific mindset that apply to business decisions:

1. Be a knowledgeable skeptic.
2. Articulate testable hypotheses.
3. Produce hard evidence.
4. Probe cause and effect.
5. Create an experimentation culture, a shared scientific mindset.

Skeptics vs DenialistsA skeptic is not a denialist. A skeptic is neutral. They look at the evidence with an open mind and want the truth. A denialist starts with a conclusion, picks evidence that supports it, and dismisses evidence that contradicts it. Stefan calls this running the scientific method in reverse. It’s easy to fall in love with a conclusion and only question the evidence that disagrees with us. Product managers do this when they run customer research to validate a solution instead of finding unmet needs. Instead, fall in love with the customer’s problem, not