Evan: How were you tracking though, because that was still earlier days, and still companies are struggling with how to track the actual 'one word is better than the next.' Were you guys hacking together stuff behind the scenes or was there actually a whole analytics platform, or flying blind with the blindfold on?
Josh: This was 2004. There were none of these analytics platforms or anything like it. We had a data warehouse, so every night the entire LinkedIn database was cloned into a data warehouse. Then the next day I could go in and tinker around the data warehouse-
Evan: What kind of tinkering? What kind of coding? What were you doing?
Josh: I was writing a bunch of SQL queries that were basically selecting the group that received invite A, versus invite B. Because we actually logged the entire chain - if I sent an email to an email address, I could then track what text they got, and then when they came and signed up - I could track their conversion rate and their behavior. I could join all these things.
I could actually test from the original invitation, how many people actually signed up and how many people then sent more invites. I could start to then go the second degree too. It started to get too complicated for my level of SQL knowledge. Now we have much more robust virality platforms. We were having to scratch and claw and make it up as went back then.
Evan: Before you joined there, did you know anything about SQL or did you think it was a kind of food from Europe? What's the deal? Did you just quickly teach yourself one day and say, "Hey we need this. We need this, and we don't have resources," and you did it, or what?
Josh: My first job was as a programmer. I worked on a product called RealPlayer that some of you might remember. Trust me, when I meet 22 year old founders these days and I say RealPlayer it's a total black stare.
Evan: I mention dial up waiting for an image for half an hour and they're like, "What?"
Josh: Yeah. Re-buffering video. I'd been a Windows programmer that was building the RealPlayer client and running the team doing that. I had a lot of coding experience, but I hadn't done much database work. Given that I had already been writing code and shipping software, picking up SQL to do the basic queries I was doing, was fine. As I said, I did kind of reach my limits and every once in awhile I would write an inner joint that they would come and yell at me for, for taking down the data warehouse.
Evan: We heard from Jack Levin yesterday talking about the first infrastructure employed at Google, and would press a couple buttons and the whole system would go down, and you had to take a bike to the office and go pull off the plugs and put them back in. So times have changed.
Josh: Yeah times have changed.
Evan: What did you learn ... First, it's easy to talk about the good things, sometimes we learn from the good things, sometimes we learn from the bad things. What's a mistake you really screwed up that you learned the most from at one of those early companies that surprised you but you learned the most from?
Josh: That's a great questions. One of the biggest mistakes in, and just one of the great learnings. I like to think of working these companies as you're constantly making experiments. Everything you do is an experiments. If you basically say, "We have a thesis and it's either going to work or not, and we're going to build this thing and we'll see what happens." That way I try not to think of it as a mistake because we went in with the thesis and we tested our thesis. Rather than, "I know this is going to work," and then when it doesn't work, you're disappointed.
Evan: It's that psychology again. Did you study psychology?
Josh: I actually did.
Evan: I knew it. It's a perspective.
Josh: I did this great program at Stanford called Symbolic Systems...
Evan: Okay. Now that makes sense.
Josh: Linguistics, psychology, computer science and philosophy. At Twitter, when I joined Twitter it was 2009 and Twitter had already been on Oprah and a bunch of people had heard of Twitter. We had this massive problem where every signup didn't stick around. We did a bunch of analysis, interviewed and called a bunch of users who had done stuff. One of the big things in the company was, we have to change the Twitter homepage. If we just make Twitter easier to understand when you show up on Twitter.com then it will grow so much better.
This was just this common thread in the company. I think it still is today. "If we just change the homepage, everyone will do it." The first project that I got started on, we rebuilt our onboarding flow. That actually worked really well. I got some credibility in the company, and they were like, "Oh, we can change things and actually make progress-"
Evan: “Josh actually knows what he's doing-”
Josh: I made a good guess and it worked. Then they were like, "The next thing, let's go make that homepage." We were like, "Let's go rebuild and design the homepage. Let's make it fresh with like new tweets coming in that show what's happening live. We'll show trends. We'll show all this stuff to make the Twitter homepage much more dynamic so you'll get a taste of Twitter," and this was replacing a page that had a big search box.
If you don't know how to use Twitter and the first thing you do is type something into a search box to search Twitter, that's probably the absolute worst way to try to figure out what the heck is going on on Twitter. We did all these changes. We made this page. It took us several weeks because we had to build a new algorithm that would surface the top tweets and build some editorial things, for instance, that we could immediately take something out if something inappropriate happened to show up, even if it was there.
Then we shipped it, and we actually did an AB test. We're like, "Okay. We'll ship it to half the users." It's very tricky to do AB tests on logged out pages because you just have to cookie people, but they might come from a different computer, but “we’ll try to do our best analysis.” We shipped it and it made zero difference. Literally what we learned or what I assumed that we learned is everyone who showed up to Twitter had so many preconceived notions of what it was about, that actually just clicking to go sign up and hopefully we could teach them through the signup flow made much more of an impact than whatever we put on the homepage. It turned out as we kept doing more testing, just removing all options except for login and signup, actually performed mildly better than everything else-