How to Fail Fast Safely: A/B Testing Lessons From Booking.com

Failing fast is only useful if failing is cheap and safe. Booking.com, which runs some 25,000 tests a year according to Stefan Thomke in Harvard Business Review, shows how to do it in seven steps: write the hypothesis first, wrap every change in an experiment, add an automatic circuit breaker, expect most ideas to fail, check data quality, record every result, and turn the habit into crisp interview answers. Within one sprint you can ship your next change this way. This guide is for software engineers who ship product code or face product-sense and experimentation questions in interviews. It is not a statistics course.

What you need before you start

• A feature flag or experimentation tool that can split traffic randomly (in-house or off the shelf).
• One primary metric your team already trusts, such as conversion, completed bookings or task success.
• A few guardrail metrics: error rate, latency, crash rate.
• Basic comfort with p-values and confidence intervals.
• About an hour to set up your first experiment template.

Step 1: Write the hypothesis and metrics before you build

Before writing code, write down which user behavior you expect to change, in which direction, and which metrics will prove it. One sentence for the hypothesis, one list for the metrics.

Why it matters: in their paper on democratizing experimentation at Booking.com, Kaufman, Pitchforth and Vermeer explain that experiment owners must specify up front which customer behavior they want to impact and which metrics support the hypothesis. Combined with peer review of successes, they write, this considerably reduces p-hacking.

Example hypothesis: "Showing free cancellation on the search results card will increase completed bookings, because users hesitate when cancellation terms are hidden. Primary metric: completed bookings per visitor. Guardrails: page latency, error rate, cancellations."

Common mistake: picking the success metric after looking at the results.

Step 2: Wrap every change in an experiment, not just the big ones

Put new code behind a flag that assigns users randomly to control or treatment, including refactors and bug fixes where you expect "no change".

Why it matters: the Booking.com paper states that every change, from entire redesigns and infrastructure changes to bug fixes, is wrapped in an experiment. A "safe" refactor that quietly drops conversions is caught in days, not months.

Example:

if experiment("search_card_free_cancellation").variant(user) == "treatment":
    render_card_with_cancellation_badge(hotel)
else:
    render_card(hotel)

Common mistake: shipping infrastructure changes without an experiment because "users will not see them".

Step 3: Add an automatic circuit breaker for the first minutes

Monitor guardrail metrics in near real time right after a release and abort the experiment automatically when errors or broken pages spike. Speed matters more than statistical elegance here.

Why it matters: Iskra and Lukas Vermeer describe Booking.com's circuit breaker, which watches each new experiment for the first three minutes, tests for significance roughly four times per second, and can stop an experiment within a second. In one case it stopped an experiment that caused errors for 47% of users. The authors note they deliberately trade some statistical rigor for speed.

Example rule: "If the treatment's error rate is significantly higher than control at any point in the first three minutes, set the experiment to 0% and page the owner."

Common mistake: relying on a human to notice a dashboard turning red.

Step 4: Expect most ideas to fail and plan for it

Budget for the fact that most experiments will be flat or negative. Judge the team on learning speed and the quality of experiments, not on the share of winners.

Why it matters: Harvard Business School Working Knowledge reports, from Thomke's research on Booking.com, that on average only 10 percent of experiments yield results that improve the bottom line. If you only ship ideas you are sure about, you will rarely find the 10 percent.

Example reframing in a sprint review: "Three of our four experiments were flat. We learned that badge placement does not matter on mobile, so we are dropping that line of work and testing cancellation wording instead."

Common mistake: rerunning a failed experiment with small tweaks until it "wins".

Step 5: Check data quality before you trust a result

Before reading the primary metric, verify that traffic split as designed (a sample ratio mismatch check), that users were not dropping out unevenly, and that the numbers match a second source.

Why it matters: the Booking.com paper describes computing a set of common metrics in two entirely separate data pipelines to validate decision data, plus statistical checks for selective attrition, which it says renders all experiment results void. A beautiful lift on broken data is worse than no result.

Example check: "We expected a 50/50 split. Control has 100,400 users and treatment 97,100. That gap is too large to be chance, so we stop and debug assignment before looking at conversion." (Illustrative numbers.)

Common mistake: debugging the data only when the result looks bad.

Step 6: Record every result where everyone can find it

Log each experiment's hypothesis, metrics, result and decision in a searchable place, and let anyone on the team start an experiment, not just a central analytics group.

Why it matters: Booking.com's platform acts as a searchable repository of all previous successes and failures, dating back to the first experiment. And according to HBS Working Knowledge, 75 percent of Booking.com's core employees take part in experimentation, with the ability to launch tests independently.

Example log entry: "2026-09 search_card_free_cancellation. Hypothesis: visible cancellation terms raise bookings. Result: flat on mobile, positive on desktop. Decision: ship desktop, retest mobile copy."

Common mistake: keeping results in slide decks that nobody can search.

Step 7: Turn the habit into structured interview answers

When an interviewer asks "How would you know if this feature works?" or "Should we launch this?", walk through the same loop out loud: hypothesis, metrics and guardrails, randomization, safe rollout, data checks, decision.

Why it matters: experimentation at scale is a real engineering problem, not just a product one. Booking.com's teams run more than a thousand concurrent experiments, which raises questions about assignment, logging, interaction between tests and rollback that come up in product-sense and system design interviews.

Example answer to "We want to add a 'price drop alert' button. How would you decide whether to launch it?":

Example answer: "First I would state the hypothesis: alerts bring users back, so 30-day bookings per user go up. Primary metric is bookings per user, with guardrails on latency, error rate and email unsubscribes. I would randomize at the user level, ramp from 1% with an automatic kill switch on errors, check the sample ratio before reading results, and run for full weekly cycles. If bookings are flat but unsubscribes rise, I would not ship, and I would log the result so the next team does not repeat it."

Common mistake: jumping straight to "I would A/B test it" without naming metrics, guardrails or a decision rule.

How to know it worked

• Within one sprint: every new change ships behind an experiment with a written hypothesis and guardrails.
• Within a month: at least one bad release is auto-stopped or caught by guardrails before users report it.
• Your experiment log has entries for flat and negative results, not only wins.
• In mock interviews: you can answer "how would you measure this?" in under two minutes with metric, guardrail and decision rule.

Troubleshooting

• If every experiment is flat: your changes may be too small or your metric too noisy, so test bolder ideas or a more sensitive metric.
• If the traffic split looks off: stop the analysis and fix assignment or logging before reading any result.
• If stakeholders want to ship a "loser": show the guardrail data and the logged hypothesis, then agree on a follow-up test.
• If the circuit breaker fires too often: check for false alarms, but keep it biased toward stopping; a wrongly stopped test costs far less than a broken page.

FAQ

What does "fail fast safely" mean in software?

It means trying many ideas quickly while limiting the damage of each failure. In practice that means randomized experiments behind flags, small initial exposure, automatic rollback on guardrail breaches and a shared record of results. Booking.com describes the goal as moving fast while unbreaking things as quickly as possible when the user experience breaks badly.

How many experiments does Booking.com run?

Stefan Thomke's 2020 Harvard Business Review article says Booking.com runs some 25,000 tests a year. A paper by Booking.com authors describes more than a thousand concurrent experiments. Figures change over time, so treat them as an order of magnitude, not a current count.

How do I answer A/B testing questions in a software engineer interview?

Structure your answer: hypothesis, primary metric, guardrail metrics, unit of randomization, rollout plan with a kill switch, data quality checks and a decision rule. Mention what you would do if the result is flat or mixed. Interviewers look for structured reasoning and trade-offs, not formulas.

Do I need statistics knowledge for experimentation questions?

You need the basics: randomization, statistical significance, confidence intervals, sample size and why peeking at results inflates false positives. You rarely need to compute anything by hand. Being clear about metrics, guardrails and data quality usually matters more than reciting formulas.

To recap: write the hypothesis first, wrap every change in an experiment, add a circuit breaker, expect most ideas to fail, check the data, record every result, and use the same loop in interviews. To practice explaining it under pressure, book an anonymous mock interview with an engineer from a top tech company. For related prep, read what helped in the system design round and browse our interview guide.