Court Exhibit·26 AUG 2016
Never Run an Experiment You Can't Measure
If you can't measure it before you ship it, you're not experimenting — you're guessing with revenue.
Source document — Fwd: Nexus Launcher Without QSB · United States v. Google LLC (Search) · 1:20-cv-03010 (DCD), Trial Ex. UPX0128 — DOJ public archive (page 1)
Excerpt · In Tom Oliveri & Philipp Schindler & Hiroshi Lockheimer & John Giannandrea & Sundar Pichai's own words
So I had a mild heart attack today when I saw the new Nexus launcher without the QSB. I fully understand the urge to innovate, but taking such a potentially high revenue impact decision which a) despite the low Nexus volume carries the risk of other OEMs asking for a similar placement over time and b) cannot really be supported by robust testing data regarding query impact (as far as I know the frameworks aren't in place yet; I have been asking for this for a long time) feels audacious - to give it a positive spin.
Hiroshi Lockheimer replied: Here's the logic: 1) Users generally are tired (visually) of our widget. It's in the middle of the screen, obscures your family picture, etc. 2) OEMs are VERY tired (visually) of our widget, because of 1) and also because they feel like they don't get to differentiate. 3) Now that we think like an OEM (the Nexus team), 2) really resonates. 4) We're going to experiment with the Nexus population. ~2M over a year. 5) In terms of OEM reaction, we've already briefed Samsung that we're going to be doing this 'experiment' on Nexus and that we'll be updating them on usage / potential revenue (share) impact.
Sundar Pichai wrote: We should always do these things with the ability to measure. Can I get a firm timeline and schedule for the A/B testing framework for this.
John Giannandrea responded: The Android cal says MR1 rolls out December 5th so if we make all the milestones in the A/B plan we will have data by the end of the year from the Search analyst teams. Not ideal, but it's where we are on this.
The daily letter · Free
This edition is for members — but the daily letter is free.
One real corporate email, and the lesson it teaches, in your inbox every weekday morning. Free. Unsubscribe in one click.
How this surfaced
- Source type
- Court Exhibit
- Case / record
- United States v. Google LLC (Search)
- Citation
- 1:20-cv-03010 (DCD), Trial Ex. UPX0128 — DOJ public archive
- Date authored
- August 26, 2016
- License
- Public domain
- Original
- View the primary source →
More from Sundar Pichai
Don ·Google ·14 MAR 2023
Bundle Deals Across Divisions to Save a Weak Position
When one business unit can't justify the check alone, stack the strategic value across the whole company — or lose the partner entirely.
Tim Sweeney & Hiroshi Lockheimer ·Google ·22 JUL 2020
Why Platforms Refuse the 'Just One Exception' Ask
The moment a platform grants one developer a private deal, it loses the argument for every other developer.
Ruth Porat & Jamie Rosenberg ·Google ·21 JUL 2017
Conditional Payments: How Google Kept Samsung Honest on Defaults
The real deal isn't the revenue share — it's what you have to give up to earn it.
Shanna Velez & Mike Marchak & Purnima Kochikar & Alyssa Ablao & Josh Kim & Randy Gelber ·Google ·15 DEC 2021
Discounts Are Cheaper Than Exceptions to Your Pricing Rule
When you can't grant a carveout, invent a program — same economics, but the rule stays intact.
Don ·Google ·14 MAR 2023
Bundle Deals Across Divisions to Save a Weak Position
When one business unit can't justify the check alone, stack the strategic value across the whole company — or lose the partner entirely.
Karen Aviram Beatty & Lawrence Koh ·Google ·11 DEC 2019
Bundling Deals Can Backfire When Your Counterparty Wants Simplicity
When you stack asks to justify a payment, the other side sees complexity — and offers you a simpler exit.