Skip to content

Personalization at Scale with Pandora Media and a16z

At an invitation-only event hosted at the San Francisco Museum of Modern Art, ActionIQ Founder and CEO Tasso Argyros was joined by two of Silicon Valley’s finest — Pandora Media CMO, Aimée Lapic and Andreessen Horowitz General Partner, Andrew Chen on the topic of personalization at scale. Together they discussed:

  • The critical role of personalization at scale in the digital age
  • The right strategic investments to support personalization at scale
  • Why personalization at scale requires transforming organizational structures.
Tasso Argyros, Founder & CEO, ActionIQ:
Thank you, everybody, for being here. I'm very, very excited about this conversation. We're going to start with brief introductions, and then we'll get into the discussion I'll be moderating, and then we'll make sure to have 10 or 15 minutes for questions from the audience, because we want to make this as relevant as possible for all of you. Aimee, do you want to start by introducing yourself?
Aimée Lapic, CMO, Pandora Media:
Sure. I'm Aimée Lapic. I'm the CMO at Pandora Media. I've been at Pandora about a year and a half, and I'm primarily responsible for marketing, but also for driving our subscription business. So, it's a little over a half-billion dollar business. Prior to Pandora, I was the CMO and head of the eCommerce business for Banana Republic. Before that, I was at the Gap for a million years. Michael Braga is actually in the audience. He and I worked together a while ago. I think I spent 14 years at the Gap, in a variety of different roles. Interesting thing about me, I grew up in Louisiana, and so I love all things Cajun and also Southern. I am a client/customer of ActionIQ. So you can talk to me about how well that's going. Because it's going really well.
Andrew Chen, General Partner, Andreessen Horowitz:
Hi everyone, I'm Andrew Chen. I'm a general partner at Andreessen Horowitz, where I'm on the consumer team. It's always funny, because we also have a biotech team that works on saving people's lives. But my job is to figure out how to entertain teenagers using all the new investments that I'm working on. So I spend a lot of time in next-gen media, like podcasting. As many of you know, we went from zero to 200 million users in a fortnight last year. So I'm spending a lot of time in the games industry, and then a lot of my attention goes to marketplaces, transportation, travel. And that's because in my prior role, I ran many of the growth teams at Uber, where I spent most of my time on the rider side of the company—the product teams. I ended up working on everything from the new user experience, engagement, retention, incentives, etc. I spent time on the driver's side, as well.
Tasso Argyros:
That's great. And briefly my name is Tasso Argyros. I'm the co-founder and CEO of ActionIQ. Very briefly, at ActionIQ, we help brands personalize at scale by helping connect a lot more data, help teams become a lot more agile, and finally orchestrate experiences across all channels. If you want to learn more, you're welcome to talk to us. Both of your backgrounds are pretty fascinating. Andrew, you worked at a brand like Uber Ride, but now you're looking at the market and what's new, and you're probably getting a ton of information every day about what's new and hasn't been possible before. And Aimee, you are absolutely super-passionate about your brands, right? I mean, every time I've heard you talk about personalization at scale for Pandora, your passion comes out.
Tasso Argyros:
So as we think about the macro level and the specifics of applying some of these principles to the brand, I must say I am honored to be here with both of you. The topic today is personalization at scale, but ultimately, it is about consumer expectations. And the people I talk to say, what consumers expect today is very different from what consumers were expecting even five years ago, not to mention 10, 20 years ago. So Andrew, let me start with you. You probably get pitched by companies all the time. They have a new idea about engaging consumers in one way or the other. What have you learned in terms of what drives consumer engagement now, versus how it was 10 years ago?
Andrew Chen:
Yeah, that's a great question. First, I'm going to do a little preamble and then I'll get to that question. So, many of you have heard the phrase "Software eats the world," right? I think Marc Andreessen came up with that. When I heard it, I would think, what does that actually mean? Why is that such a deep thing? And I think it's for the following reason. For a really long time, for decades, software was its own category. Tech was its own thing. We would use websites, but when it was time to drive a car or go eat out or something, then we would just go to the non-tech world to do that thing.
Andrew Chen:
All of sudden, I realized the slogan "software eats the world" means, software ends up actually being a core part of all these other experiences in the real world. And so a lot of the time that I spend in investing, we end up talking about marketplaces for everything, like used cars to pet adoption to lawn car—you name it. It's about how you bring software to each one of these things. And so what ends up happening is, there are all these really interesting second- and third-order effects that then really reinvent the experience and are going to continue to transform it over the next few decades.
Andrew Chen:
I'll just zoom in on food delivery as one particular example of this. When you look at food delivery right now, you hit a button and a burrito comes to you. Amazing, right? But I think what is really fascinating to think about is, all of the searches people are making. Think about all of the individual menu items when people tap into a particular thing, and then they abandon cart, maybe they add it back in. Maybe people are searching for Korean food at places that don't have any Korean restaurants, right? So what ends up happening is, all of a sudden all of this data creates the next generation of companies that are able to take advantage of that data and to just rethink the interaction around food in completely new and different ways.
Andrew Chen:
So, for example, one of the newest companies that I've invested in—it's actually unannounced right now—has the goal of literally figuring out what it is that people are searching for and what people are eating, what people want and aren't able to get. And that's because currently the model is being delivered by this restaurant 1.0 infrastructure, which is 75% retail and 25% kitchen. But what if we put these dark kitchens all over the city that just make whatever kind of food that consumers want? So, it literally feeds on that data and translates it into these virtual restaurant brands that then can be delivered with no retail presence at all. I think that's a really, really interesting example of just how technology can take an experience like the restaurant eating experience and take it in a totally new, different direction. And five years from now? Well, then we'll just be complaining some other little aspect of things.
Andrew Chen:
Because consumers get spoiled, and then off we go and the next wave of innovation begins. It never stops. So that's very interesting. So, if I can paraphrase, there's the core product. A lot of data is generated through interactions around the core product, like people that are searching for different foods, or in the case of Pandora, people that are consuming music. There's value in that data. That was probably data that used to be thrown away, but now people are realizing that you can actually do something very useful. Like understanding what the expectations of Pandora users, and how that ties to this kind of second or third level of data generation.
Aimée Lapic:
So who here listens to music? Okay. It makes you feel joyful hopefully, or sad, or some intense emotion. Usually, the expectations of our customers for our experience is intense. The expectation is all about an intense emotion, either a very joyful emotion or a very sad time, etc. But also it's about getting the right music for you at the right time. And personalized discovery is the reason Pandora has been so successful over time. Pandora is a 14-year-old brand in the music space. And the digital music split a pretty long time. But in that space, we were at forefront of this idea of using machine learning, which at the time we called the music genome project.
Aimée Lapic:
In other words, the art and science of picking music that people would love and bringing it to them. And the next wave of musicians and songs, etc., based on your behaviors, based on what you've loved, based on your thumbs up, thumbs down, skips, replays, etc. That idea of being able to serve up a very personalized experience is the reason Pandora is still in business and still continues to be the number-one online music player in the US, because you can consume a particular song in many, many different places today. And what's interesting is that, because we're really good at figuring out who should get what type of music, our response, our engagement, our time-on-platform is really high. People spend on average well over a day each month listening to music on Pandora.
Aimée Lapic:
That's a lot of time, given the number of users we have. Yes, it's an enormous number. We have over 70 million US listeners in a given month. That's a lot of music being consumed. And that idea of giving you the right type of music because of who you are and what you listen to is really powerful. We talked a little bit about personalization at scale. So in marketing, we try to continue to bring people back to the platform, right? We make money when people are on the platform listening to music. We're an advertising-based model predominantly. So, we will figure out what you love and pair that back to the kind of email or a push notification we send, to tell you that that type of music is available on the platform.
Aimée Lapic:
When we get it right, our response rate is 300% higher than when we don't get it right. So for example, we've done a head-to-head test targeting listeners who like to listen to rhythm and blues. So for example we targeted Janell Monet's latest album to R&B listeners versus known Janelle Monáe listeners. Our response rate to Janelle Monáe listeners was 300% higher than the response rate to average R&B listeners. That kind of personalization at scale—we've sent the message out to thousands and thousands of people—is how Pandora brings this very idea of personalized discovery to our customers.
Tasso Argyros:
So, essentially personalization is the product for you.
Aimée Lapic:
Yes, 100% of the product. It's why people stay. I mean, we talk to our customers a lot both in person through focus groups but also through surveys, and we actually watched them, what they do when they leave, etc. The reason they stay is, personalized discovery. They're showing us that they love what we have. The reason they leave is, they don't know yet that we also offer on-demand listening as well. And so that's our challenge—how do we do both really effectively for our customers.
Tasso Argyros:
It's fascinating that we're having all these conversations about data and personalization, and you're the CMO that's doing that, right? Historically, you had people like Don Draper from Mad Men, where the CMO is very brand/creative focused. But you wanted to be an astronaut when you were a kid, if I understand correctly. And you grew up through the data analytics CRM track. Tell us a little more about how you evolved as a leader.
Aimée Lapic:
Sure. The key to being a decent leader is being authentic. So, here you go. Here's my truth. I was pretty analytical earlier in my career. I did a lot of CRM marketing, and my boss at the time said, I was good at my job, but I'd never be a CMO because I hadn't worked for an agency. I was not a creative person, etc. So he said I should figure out where I'm going. How do you become a GM through the CRM track? Well frankly, for the last seven years of my life, I've been a CMO, plus a GM of a business, because what's happened over the last 10 years. The analytical parts of marketing have become more important than the creative parts of marketing. Not that creative is not important. You've still got to tell stories, you've got to connect with listeners or consumers. But how you do it can be so much more targeted and so much more relevant. And so basically, that same person who gave me the advice then promoted me to be the CMO of Banana Republic, despite what he had told me five years earlier.
Tasso Argyros:
And the times change super-fast. I mean that was only a few years ago. So, how does the role of the CMO change in terms of marketing versus product, or P&L owner versus just managing creative or advertising?
Aimée Lapic:
The #1 ways CMOs are successful is that they're actually driving the business. That means figuring out what the key business metrics are, whether it's revenue or profit, etc. That's got to be at the forefront of everything you do as a marketer. It can't be, I'm here to create brand relevance for the sake of creating brand relevance. I mean, that's important, but it's got to be a means to drive the overall business. So frankly, what's happened over the last few years is, a lot of CMOs have become P&L owners at the same time, because now they have these levers at their disposal to really drive the business. And that has been phenomenal for most businesses, because then you're thinking about the customer and the customer experience first, and how that plays into driving the overall business results.
Aimée Lapic:
It also happens to be my passion—understanding what the customer's wants and needs are, first and foremost, and then figuring out how to deliver on that. In terms of the difference between product and marketing, it's interesting, because I've worked in retail, I've worked in eCommerce and now I work in a digital app environment. And in my experiences, that product is critical in each of those different environments, but it plays a very different of role, and you will not succeed if you don't have the customer lens around it as well. So figuring out that interplay of how marketers and product folks work together to get to the end objectives and the better customer experience—that is really critical. It needs to be stated and worked on together as shared objectives, because otherwise you'll have a phenomenal product that no one knows about. Or, you'll have a really good story that doesn't hold up when consumers actually use the product being developed.
Tasso Argyros:
That's great. So, Andrew, to bring it back to you. You talk a lot about growth, right? You've built an amazing brand around yourself for being the growth expert in the industry today. And you've been talking about growth as its own discipline with its own principles. So, help us bring everything together. We've talked about marketing becoming more growth-focused. Also, we're blurring the line between what's a marketing communication and what's a product. So we have marketing, we have product, and we have growth. How does it all work together?
Andrew Chen:
It's been really fascinating that over the last five years that we're starting to see these teams that call themselves growth teams at the really new companies, the Ubers and Airbnbs. And Google has a team that's like a central growth team. So I think it's a really fascinating movement. I think the reason for that is really simple—startups are constantly broke. They have no money. So what ends up happening is, they're initially growing in order to get customers to use their product, so the product has to grow itself basically, right?
Andrew Chen:
The users have to help grow the products. As much as a company like Dropbox might spend on paid marketing, hundreds of millions of people end up using Dropbox because someone shared a folder with them. The same with Instagram. You started using it literally just because it was in a feed, a photo from one of your friends. The same with Slack. And with Uber. And so I think these startups first get started and then they realize, we actually need a lot of users.
Andrew Chen:
And they don't typically reach for the traditional marketing tools, because often they are uneducated about them, in the case of these young technical founders. But also they don't have the budget for them. And so they do the thing that they know how to do, which is to use software to try and solve this problem. And so over the years what's happened? You end up with these growth teams that are primarily thinking about how to grow the product. And they're doing it by thinking about onboarding flows, notification systems, optimizing engagement loops, etc. And they are thinking about that from a software/product lens, as opposed to the traditional marketing lens. And I think that is really interesting.
Andrew Chen:
For example, you can talk to the LinkedIn team about how they think about email. Yes, they do have some humans thinking about doing campaigns. But the vast, vast majority of the billions upon billions of emails that they send each month are completely machine learning-driven. It's personalized, it's driven by data. They build these really sophisticated systems, the score, the efficacy of each of the potential stories that they can put in an email, net the unsubscribe rate that might actually happen from it. They've tried to figure out frequency caps and all that, and it's all in software, and that's how it's set up. And when you think about that, that's what's happening both in email notifications, but also in algorithmically generated feeds.
Andrew Chen:
It ends up becoming a very different lens than that of folks who, for example, might come into tech from the agency side of the world. Now, I think the good news is that the whole industry is getting smarter. And so whether you're coming from a "marketing" background or a quote "product" background, the skills around being data driven, everyone is thinking in a more, quantitative way and more iterative way. I think that is something that is rapidly converging. I've talked to a lot of the folks that hold VP growth titles. And it's funny, because I think if this movement continues, we're going to eventually call these growth teams part of the product team or part of the marketing team.
Andrew Chen:
The real question is whether or not you give engineers to the marketers to figure out what to do with them, or do you always want PMs and data scientists, et cetera? At Uber, one of the first things that happened was, they started the growth team. It was 10 people. This was when the company was about 300 employees total. And the first thing that the team did was build the internal A/B testing framework. That was literally the first move. After they did that, they were focused on mostly user acquisition, and then they grew it and grew and grew it. And eventually the growth team at Uber had over 500 people. And then it was the most iterative, data-driven organization in the company. But they would often work with marketing by having a product marketing matrix on the growth teams. And at various times, performance marketing actually reported into the growth teams. So there are a lot of different ways to do it. I don't think anyone's figured out exactly the right setup.
Tasso Argyros:
This is very interesting, and is actually very near and dear to our heart, because at the end of the day, we try to make it easy for teams to become a lot more agile, more nimble, able to iterate a lot more quickly and then make it a lot easier to measure every action. For some people, it seems obvious, but for others it's not. Aimée we have spoken about A/B testing, right?,
Aimée Lapic:
We have four different arms of every single buy, and we optimize to get to one, and then we challenge it again with three different arts, like literally. And so everything is a test.
Tasso Argyros:
Why do you think that hasn't that spread to the broader industry?
Andrew Chen:
It's still something that we see maybe in top 10%. But it's still like a top-tier kind of skill to be testing driven. But within the, startup tech industry, it has been increasing in the last 10 years. We used to talk about investing in companies based on how many total signups they have. Literally just signups, not even active. Oh, you've signed up a million users. That's awesome. Right. And that would literally be just like a joke now. When we see a cumulative user graph on a pitch, we laugh cause it's funny, because it's not truthful, right?
Andrew Chen:
Instead, you have to measure some fundamental business value that's there. And so I do think that overall the startup ecosystem is getting much smarter. I think the hard part is literally just getting your data to a place that you can even do any of this analysis. It can be insanely hard. There was a period a long time, even at Uber, where our data warehouses were so disorganized and messy that just pulling out normal bits of data to try to be data-driven was difficult. And so people would just have to use their best guess. So I think there's always these cycles, but I am actually very optimistic and bullish, because I think everyone's getting a lot smarter.
Aimée Lapic:
Yeah. And the tools are, to your point, are a lot better. When I was at Banana Republic running eCommerce, we would do a ton of different tests. But the ability to act on the information in a timely manner was not there. It was mostly about accessing the data and being able to use it for the next level of testing, whether it was product related or inventory related or marketing related. So the accessibility and the availability go hand in hand.
Tasso Argyros:
That sounds great. The actionability. So, I'm curious because you've been in the space for a long time, right? My perception is that the old school of marketing people would market based on profiles of customers. In the new world of marketing, people market or personalize based on interactions, right? Because there's just so many more interactions, and they are so much richer. I mean, for Pandora it's all interactions. And I'm curious, first of all, do you agree with this, and if so, was there a point in the last 15 years that the switch happened in your mind or in the industry? Did they think, we have to stop looking at what someone bought, and start looking at every interaction they're having with us? What does it tell us about the customer? Are we still going towards that point as an industry?
Aimée Lapic:
I do think it depends on the actual industry. Not all tech is the same, for example. So for example in apparel, there is still a focus on the type of customer segment according to tastes, how they use the clothes, etc. So you would design clothes for that type of person, versus just everything being reactionary based on their behavior. It has to be forward-looking, if you will. And so that's a little bit different than music, where you can serve up the next song based on the behavior you were exhibiting within the app itself. So if I skip a song, you're going to serve up a song that's more like the songs I've thumbed up in the past. If I replay that song, you're going to serve up a song very similar to that. So it's all behavioral based, if you will.
Aimée Lapic:
So, it depends on the type of industry you're talking about. I think there's still a ton of marketing that's based on customer profiles, on who are our aspirational profiles. Because that's inspirational to people in terms of their storytelling and their creativity. Yes, it has to be mirrored by real data. What if you can create all of those profiles, and yet they only represent 1% of your population? So, it needs to be matched to real data. And then, how it performs needs to be matched to what actually happens when you put a campaign against that or actions against that, depending on the customer experience versus outbound marketing.
Tasso Argyros:
So your point is that the profile data is almost biased in some ways, right? In addition to not being probably very accurate.
Aimée Lapic:
In the past it was very biased. It was very aspirational—"These are the type of customers we want to attract." And so it's biased to those people. It takes good data and, honestly, rigor within the marketing team to say, all right, that protocol isn't matching up with who were actually really attracting, and then using that insight inform and change what you're doing.
Tasso Argyros:
That makes sense. So Andrew, when you think about new technologies, what role does behavioral data play? Is this something you think about when you're looking at investment—what data is generated, how it's generated?
Andrew Chen:
I think the kind of consumer data that you get can play a really core part of the investment thesis, when the product category has passed its initial stage. It is crazy to think that in the last five years, the most valuable startups that have created an apps that lets you get into strangers' cars, and then they drive you around. There's an app that lets you sleep at a stranger's home, and it's fine. You just do it. There's an app that lets you watch other people play video games, and that's a $1 billion company. And so I think it's just amazing how for many of these consumer products, the first version of a product is often just kind of a surprise.
Andrew Chen:
And literally that's the only feature it has. It's sort of like, wow, people actually want to do this, this is amazing. It's only later on that a second or third iteration of the product category actually figures out how to use the data and the behavior in a more sophisticated way. When you're lucky, the company that invents the category also figures out how to take it all the way. But it's rare. Let me give you an example. I'm close with the team that cofounded Tinder, and was an advisor to the company off and on over the years. And you know, they talk about how early online dating was this thing that was embarrassing.
Andrew Chen:
It was crazy. You'd go to work and send emails and set up meetings and going to meetings. And then you'd come home and get on match.com or eHarmony. And then you'd spend all your time sending emails back and forth and then going to meetings. It was just like work. Now, take Tinder. It is very fascinating that the team had the insight of the right swipe, left swipe gesture. And from that, they actually figured out that, instead of having people fill out all these forms and enter all this data, why don't we actually figure out how to use Facebook social graph data?
Andrew Chen:
So, if you are right-swiped versus laps left-swiped, you pick up an attractiveness score inside the app, which means that you're only matched with people of similar attractiveness. So they're using data in this really creative way. That's the kind of thing where, as an investor, you might look at a product category that's five years in and say, okay, now that it's been validated that people want to do this, you have to get more sophisticated. You ask, how are you going to use ML to make this better? How are you going to use the data, implicit and explicit, in order to make the experience really break-through?
Andrew Chen:
And so what ends up happening in my world is, I often do not typically take the bat on the first product in a category, especially if it's pre-launch. I like to invest in companies where it's very surprising that it'll work, but you have enough customer data that you can say, this will probably keep going, and you have another frame on it. The other thing that I love is to invest in the version two or version three of a product in a category that's using technology in a really interesting way to 10X the value to the customer. That's very interesting. So five years later, there's an opportunity to reinvent the product by using a lot more data, essentially.
Tasso Argyros:
That's fascinating. I will ask one last question, since we didn't talk about ActionIQ that much. Very briefly, Aimée, we're honored to be your partner and support you. Obviously you're super passionate about the product and about personalization and customer experience. Would you explain to someone why was you engage with ActionIQ? That's off script, by the way.
Aimée Lapic:
I didn't really come to sell, but I will say that ActionIQ has allowed a much simpler way of targeting with granularity. We didn't have the ability to do that in a way that was easy for everyone. For example, we had our analysts writing SQL queries to pull actual customer segment lists and then send them by emails etc. It's just a ton of work, and so they've cut down on the amount of work and made it much easier for anyone in the marketing team to be able to do that. We're also able to get our results really much quicker, in a timely fashion. Instead of having to write the same level of analysis, we are hoping to be able to look at a customer level across multiple touchpoints to paid media, email push, within our own APP, within Pandora house media. We target our own customers with different messages, our own internal media as well. So that's the Holy Grail for us—to be able to literally look at how each of the media performs against individual customers. We're excited about that. That's on our product roadmap with ActionIQ. It's one of the really big things we could get from this partnership that we're hopeful we get to sooner rather than later.
Tasso Argyros:
We're working hard on that. Thank you. One more question. Besides being a great CMO, you're obviously a female leader in a space that doesn't have a lot of female leaders. What advice do you have for the women in the room who would love to sit here one day and have a similar conversation?
Aimée Lapic:
I'm passionate about promoting all kinds of diversity in the workplace. I happen to be a woman, so I'm really passionate about helping other women develop into the phenomenal leaders that they can be. My advice is going to be personal. I am also a mom of three children. And I think if you want to be a strong leader, you're going to have to focus on different things at different times in your life. It's not always about your career. You're not going to have to make those kinds of choices every single time, but every day is going to have to be a choice. Like today, I volunteered in my daughter's classroom this morning and I didn't go into the office. I'm here now. So this is a day I chose family time and then work. At other times it is vice versa.
Aimée Lapic:
I think it's really important to allow yourself to be you as a female first and foremost, before you're think, I've got to be in the C-suite. That's really critical. I would also say, frankly all kinds of people with all types of backgrounds add a lot to the leadership mix. Have your own voice. Be confident in what you think is the answer, back it up with data, but go with your gut, because a lot of times your gut is 90% right. And say it out loud because, what I've seen over time and what I did early in my career is, I held back my opinion, because I wasn't as brash as the guy next to me. Literally the guy next to me, because I worked in consulting which was 17% female. I think it's important to know you're probably right and say it out loud. If you're wrong, just say, I'm sorry I was wrong. I don't even say I'm sorry. I just say, okay, I'll look it up later. But life is long, and there's a ton of opportunity to learn as you go and get more experience and keep going. And I would say, don't hold yourself back. Allow yourself to be the person you can be.
Tasso Argyros:
Thank you. And Andrew, if you had to give one piece of advice about how to use the startup playbook to grow the business, what would that be?
Andrew Chen:
The thing that I love the most about these small teams is just the iteration speed, just how focused they are on this loop of creating insights and executing and analyzing the results and coming up with new hypotheses and executing that loop. Whether you're at a big or a small company, ask how you can create that culture and remove as many obstacles as you can, so that you let the folks that are closest to the problem iterate as quickly as possible. Whether you're at a big company or small company, you can be guided by that philosophy.

Scroll To Top