Ronny Kohavi

The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)

Source 2552023-07-2714,338 words

Use this source for the original language, context, and comparison across the transcript set. The excerpts below keep the conversation readable before any interpretation.

Source moments

Read the transcript before the interpretation.

These excerpts keep the original conversation visible. Read the quote first; any interpretation should stay attached to what the language actually does.

Opening
Ronny Kohavi (00:00:00): I'm very clear that I'm a big fan of test everything, which is any code change that you make, any feature that you introduce has to be in some experiment. Because again, I've observed this sort of surprising result that even small bug fixes, even small changes can sometimes have surprising, unexpected impact. Ronny Kohavi (00:00:22): And so I don't think it's possible to experiment too much. You have to allocate sometimes to these high risk, high reward ideas....

The segment is an original transcript moment first. The interpretation should stay attached to what the language actually does.

Low-ego framing
much. You have to allocate sometimes to these high risk, high reward ideas. We're going to try something that's most likely to fail. But if it does win, it's going to be a home run. Ronny...

Uses we/us, uncertainty, or learner framing instead of performing authority.

Accept praise cleanly
Lenny (00:01:53): Ronny Kohavi (00:04:33): Thank you for having me. Lenny (00:04:34): So you're known by many as maybe the leading expert on A/B testing and...

Accepts praise without shrinking from it or turning it into a performance.

Return warmth
Lenny (01:21:47): Amazing. I appreciate that. Two final questions. Where can folks find you online if they want to reach out, and is there anything that...

Matches the guest's warmth and keeps the social temperature generous.

Low-ego framing
Ronny Kohavi (00:09:18): And by the way, I don't know if you know the history of this, but I tell about this in class. We did this experiment way back around 2008 I think....

Uses we/us, uncertainty, or learner framing instead of performing authority.

Accept praise cleanly
Ronny Kohavi (01:22:39): Thank you so much. Lenny (01:22:40):...

Accepts praise without shrinking from it or turning it into a performance.