Court Exhibit·26 AUG 2016
Never Run an Experiment You Can't Measure
If you can't measure it before you ship it, you're not experimenting — you're guessing with revenue.
Source document — Fwd: Nexus Launcher Without QSB · United States v. Google LLC (Search) · 1:20-cv-03010 (DCD), Trial Ex. UPX0128 — DOJ public archive (page 1)
Excerpt · In Tom Oliveri & Philipp Schindler & Hiroshi Lockheimer & John Giannandrea & Sundar Pichai's own words
So I had a mild heart attack today when I saw the new Nexus launcher without the QSB. I fully understand the urge to innovate, but taking such a potentially high revenue impact decision which a) despite the low Nexus volume carries the risk of other OEMs asking for a similar placement over time and b) cannot really be supported by robust testing data regarding query impact (as far as I know the frameworks aren't in place yet; I have been asking for this for a long time) feels audacious - to give it a positive spin.
Hiroshi Lockheimer replied: Here's the logic: 1) Users generally are tired (visually) of our widget. It's in the middle of the screen, obscures your family picture, etc. 2) OEMs are VERY tired (visually) of our widget, because of 1) and also because they feel like they don't get to differentiate. 3) Now that we think like an OEM (the Nexus team), 2) really resonates. 4) We're going to experiment with the Nexus population. ~2M over a year. 5) In terms of OEM reaction, we've already briefed Samsung that we're going to be doing this 'experiment' on Nexus and that we'll be updating them on usage / potential revenue (share) impact.
Sundar Pichai wrote: We should always do these things with the ability to measure. Can I get a firm timeline and schedule for the A/B testing framework for this.
John Giannandrea responded: The Android cal says MR1 rolls out December 5th so if we make all the milestones in the A/B plan we will have data by the end of the year from the Search analyst teams. Not ideal, but it's where we are on this.
The daily letter · Free
This edition is for members — but the daily letter is free.
One real corporate email, and the lesson it teaches, in your inbox every weekday morning. Free. Unsubscribe in one click.
How this surfaced
- Source type
- Court Exhibit
- Case / record
- United States v. Google LLC (Search)
- Citation
- 1:20-cv-03010 (DCD), Trial Ex. UPX0128 — DOJ public archive
- Date authored
- August 26, 2016
- License
- Public domain
- Original
- View the primary source →
More from Sundar Pichai
Ruth Porat & Jamie Rosenberg ·Google ·21 JUL 2017
Ruth Porat and Jamie Rosenberg Discuss Samsung Search Revenue-Share Deal
A two-line executive sign-off on a Samsung search agreement captures, in near-real time, the commercial machinery at the heart of the U.S. government's landmark antitrust case against Google.
Jeff Shardell ·Google ·4 JUN 2007
Price Your Distribution Power Honestly
When a partner asks for the same price on a worse deal, the right answer starts with naming what you're actually buying.
Joan Braddi & Sundar Pichai & Sergey Brin & Jonathan Rosenberg ·Google ·7 JUN 2007
Don't Pay for Customers You Already Own
If your partner brings no real users to the deal, they aren't a partner — they're a toll booth.
Anthony Levandowski & Suzy Sammons ·Uber ·28 MAR 2017
Levandowski and Uber Coordinate Messaging Days Before His Firing
This internal email captures Anthony Levandowski, at the center of the most significant trade-secret lawsuit in autonomous-vehicle history, calmly dividing talking points with Uber colleagues just weeks before he was pushed out of the company.
Travis Kalanick & David Drummond ·Uber / Google ·7 MAR 2015
When Your Partner Stops Taking Meetings, Believe the Silence
A calendar tells you more about a partnership than any press release ever will.
Ed Baker & Jessica E. Lessin ·Uber ·26 DEC 2014
When Your Investor Becomes Your Biggest Competitor
The moment a board member starts leaving the room, the partnership is already over — the question is just who moves first.