The Leadership Letter

Real correspondence from the people running real companies — and what it reveals about leadership.

Never Run an Experiment You Can't Measure

If you can't measure it before you ship it, you're not experimenting — you're guessing with revenue.

So I had a mild heart attack today when I saw the new Nexus launcher without the QSB. I fully understand the urge to innovate, but taking such a potentially high revenue impact decision which a) despite the low Nexus volume carries the risk of other OEMs asking for a similar placement over time and b) cannot really be supported by robust testing data regarding query impact (as far as I know the frameworks aren't in place yet; I have been asking for this for a long time) feels audacious - to give it a positive spin.

Hiroshi Lockheimer replied: Here's the logic: 1) Users generally are tired (visually) of our widget. It's in the middle of the screen, obscures your family picture, etc. 2) OEMs are VERY tired (visually) of our widget, because of 1) and also because they feel like they don't get to differentiate. 3) Now that we think like an OEM (the Nexus team), 2) really resonates. 4) We're going to experiment with the Nexus population. ~2M over a year. 5) In terms of OEM reaction, we've already briefed Samsung that we're going to be doing this 'experiment' on Nexus and that we'll be updating them on usage / potential revenue (share) impact.

Sundar Pichai wrote: We should always do these things with the ability to measure. Can I get a firm timeline and schedule for the A/B testing framework for this.

John Giannandrea responded: The Android cal says MR1 rolls out December 5th so if we make all the milestones in the A/B plan we will have data by the end of the year from the Search analyst teams. Not ideal, but it's where we are on this.

This edition is for members — but the daily letter is free.

One real corporate email, and the lesson it teaches, in your inbox every weekday morning. Free. Unsubscribe in one click.

Want this edition and the full searchable archive?

Start your free week →What members get →
Court Exhibit
United States v. Google LLC (Search)
1:20-cv-03010 (DCD), Trial Ex. UPX0128 — DOJ public archive
August 26, 2016
Public domain
View the primary source →