The Caroline Rose Hypothesis: Absolutely Everybody Should Be Involved In Discovery Research

Caroline is a senior product manager on the Etsy mobile apps team and previously worked at Walmart Labs.

In this episode of the Product Science Podcast, we cover Caroline’s career in product, how she uses story telling to align different teams, how to get buy in for continuous experimentation at companies large and small, and how even a failed experiment can yield positive results.

Subscribe for the full episode on Apple, Google Play, Spotify, Stitcher, and more. Love what you hear? Leave us a review, it means a lot.

Resources

• Follow Caroline on LinkedIn
• Follow Holly on Twitter
• Follow Holly on LinkedIn

Questions We Explore in This Episode

How did Caroline get started in her career in product? How did working for a management consulting firm interest Caroline to pursue a career in product? How early did Caroline know she wanted to be involved in product? How did an MBNA help Caroline’s goal of being a product manager? How many paths are there to becoming a product manager? Is there just 1 path to becoming a product manager? Why can most careers pivot into being product managers? How important is learning to empathize with different teams as a product manager? How can product managers align different functional groups to align on the user’s needs?‍

How did Caroline decide between working at a small startup vs a larger company? What are the benefits of being a product manager at a startup? How is being a product manager at a startup differ from being one at a larger company? How was Caroline able to learn more about product from seasoned product managers at Walmart Labs? Why are large companies like Walmart able to run experiments at large scale? How was Walmart labs like the benefits of both a startup and a large enterprise within Walmart? Why was much of being a product manager at Walmart Labs about managing relationships? How was Caroline able to to translate what mattered most to each stake holder at Walmart Labs?

‍What resources for product managers was Caroline able to utilize along her career? What are influential conferences for product managers to attend? How did the Women in Product Organization help Caroline advance her career? How was Caroline able to use story telling to align different groups? How does story telling help create empathy and understanding in others? How does Caroline explain being a product manager to her family? How do you craft a story that listeners can relate to? How do product managers use data and experiments to illicit emotions out of end users?

‍How does Caroline get buy in for the running experiments? How does being problem focused versus solution driven help to drive better solutions? How does do you keep leadership informed on the results of experiments and progress towards solutions? How can continuous experimentation help break down problems? What is it really like to run experiments? How hard is it to get the resources needed for even small experiments? How can prototype experiments with minimal to no engineering cost produce solutions? What was the result of an experiment Caroline ran at Walmart with regards to the value of brand identity to the customer experience? How did Walmart’s team’s understanding of the customer change with each experiment? How were teams at Walmart able to align their shared mission after experiments revealed new insights into customers?

‍How did Caroline transition to working at Etsy? What is it that sets Etsy apart from other Ecommerce sites? What are the similarities and differences between how different companies approach Ecommerce? How quickly can smaller startup teams move and deliver? What does experimentation look like at Etsy? How was Etsy able to create internal tools that offer great insights into their own metrics? How does the company culture at Etsy promote learning from both successful and failed experiments alike? How is Etsy able to understand the customer journey through its metric data? How are the product teams structured at Etsy? What percentage of the teams are involved in problem solving? How does Etsy’s culture stem from its leadership? How does Etsy align teams to empower everyone to be involved in product discovery? How do you set shared OKR’s among teams? How does this alignment help focus what experiments can be run at Esty? How does Etsy listen to its teams through testing and learning to help promote a healthy working environment?

Quotes from Caroline Rose in This Episode

Don’t promise solutions, promise a commitment to solving a problem.

A win could be we moved a metric positively, or it could also be we moved a metric negatively and therefore learned something about it.

When team goals are not shared, getting everybody on board to then do product discovery together is not always successful because it's not set up that way from the beginning.

Transcription

Holly: Hi, and welcome to the Product Science Podcast, where we're helping startup founders and product leaders build high growth products, teams, and companies through real conversations with people who have tried it, and aren't afraid to share lessons learned from their failures along the way. I'm your host, Holly Hester-Reilly, founder, and CEO of H2R Product Science. Welcome, Caroline, it's so great to have you on the podcast.

Caroline Rose: Thanks, Holly. Good to be here.

Holly: So I love to get started by learning a little bit about people's journey into products. So tell me a little bit about yours.

Caroline Rose: Sure. I guess it begins in my first job after college. So I joined a management consulting firm with kind of the idea that I think most people have, right? They want to get exposed to a bunch of different types of business problems. Admittedly, as a 22 year old, it sounded fun to be able to travel around the US, and work with a bunch of different clients. I worked for a consulting firm within their healthcare practice. So through luck, I ended up on these software implementation projects, and what we were basically doing is we were on site at hospitals trying to launch reporting and workflow tool. So what we would do is we would get on site, we would talk with people in the business office, understand their workflow, their needs, and then, I would turn around and work with our engineering team and say, "Okay, how can we tweak this software tool in order to fit their needs?"

I would then test it with the client and say, "Does this look good to you?" We would launch it, and then we would measure the impact and the success, and really enjoyed that work that I was doing. But ultimately I was working for our management consulting firm and not a tech or truly innovative company, and so they weren't investing in that product. I had some ideas for ways that we could tweak it. We could make the design more user-friendly, and it just, at the end of the day, it wasn't a priority. It was through that reflection and that job, I realized that, "Oh, there's this role called a product manager where I could go work for a tech company. I could do all of that same work and really be set up for success." So around that same time became interested in getting my MBA, and so through that process decided that one of my main goals coming out of business school would be to get a job in product management at ideally a large company where I could learn the tricks of the trade and become the best PM that I could.

Holly: Okay. So you went to business school with the goal of using that to help you get a job in product at a bigger company?

Caroline Rose: I did. I did. The more research I did back when I was at the consulting firm and I thought, "Okay, I think I want to be a PM. How might I transition into that role?" I realized that there isn't really one true path that most people take. I gathered that sometimes it would be a software engineer who could transition. Sometimes it could be somebody in marketing in a tech company could make a transition, and other times, it involved going to get your masters, either going to business school or getting an advanced analytical degree. So MBA is the path that I took and specifically chose a generalist program, one where I would get exposure to operations, marketing, finance, data science. Basically, in a practice of empathy in all those different functional areas.

Holly: Awesome. How did that work out for you? What was the program like, and did it do a good job of preparing you for being a product manager?

Caroline Rose: I think it did. So I attended the Darden School of Business at the University of Virginia. It's a unique program, I think, where they do require you to go through what's called the core curriculum and getting exposure to all those different functional areas. In my mind, in retrospect, it really was a practice in empathy for all of those different functional areas. Eventually as a PM, you have to be able to speak the language of a data scientist, speak the language of an engineer, speak the language of somebody who works in operations, and understand what they care about, what their goals are, and obviously understand your end user as well, and sort of bring all of those things together to launch a successful product.

So in my two years at business school, we ran through tons of examples of, "How would you think about a problem from an operation standpoint and how would you think about a problem from a finance standpoint?" That was great practice for me to be able to then to go be a PM at first at Walmart and now at Etsy, because it basically gave me that practice and that ability to think about a problem from all angles.

Holly: Great. Tell me a little more about that first product job after business school?

Caroline Rose: Yeah, I put a lot of thought into, "Okay, sure, I know I want to be a PM, but at what type of company do I want to get my first exposure and first experience?" The way that I sort of thought about it was, you've got this spectrum where on the left hand side, I could go work at a startup. I could work on a product that maybe we're not even sure if it has product market fit yet. I could wear a ton of different hats. I could be really head to the ground, learning from our customers, but at the end of the day, maybe we never get traction and maybe we don't reach a ton of saturation and scale. Now, on the right hand side of the spectrum, I could also go work for a really large company that's established that has the scale, has the resources, but perhaps my level of ownership would be much, much narrower.

So instead of owning one whole area of a customer journey, maybe I just own a page or a sliver of that journey. So I put a lot of thought into where on that spectrum do I want to learn? Because I was transitioning from a non-technical background and from not a true product background. I landed on, "I really want to be at a larger company where I can have seasoned product leaders. I could launch things ideally at scale, and I was okay to compromise on that level of ownership," I guess. So I ended up at Walmart Labs, which is the technology arm within Walmart, and I was specifically working on the international team on the Mexico website and app.

Holly: Cool. So it sounds like you were very intentional about what type of company you wanted to go to and what kind of experience you wanted to gain. What was the product leadership like when you got to Walmart Labs?

Caroline Rose: I was, I think, intentional. The way that I ended up there was through an alum of my MBA program and the way that he had sort of pitched the organization was that we have the backing of the largest retail company in the world, and yet we operate as a startup, a technology startup within that organization. So you really get sort of the best of both worlds. You get the ability to experiment, learn quickly, and you have the millions of users that are on Walmart's platform every day in order to learn from. The leadership there was really great. I leaned on them heavily in order to learn how to be a good PM. They really invested in the training that they put into the team and also gave us a fair amount of autonomy in order to drive our own roadmaps.

Holly: That's great. So, I mean, you say you leaned on them quite a bit. Are you able to tell me any lessons that you learned from them?

Caroline Rose: I think one of the key lessons I learned, something that I didn't really think about as much when I was deciding where to go. So Walmart, of course, is a retail and a merchandising-based company and not a tech company. So one thing that I noticed right off the bat is we had really strong product and engineering leaders on my team, so within Walmart Labs. But it required a really strong partnership with the P&L owners or the merchandising teams that were the ones actually deciding, what products do we sell and from where? So do we sell them only online? Do we sell them within the physical stores, and how does that then hook up to our e-commerce network? So a lot of alignment happened between the leaders on my team. So from product engineering and those merchandising leaders.

I learned firsthand that a lot of the times, they weren't speaking the same language. So we would talk in terms of product development, life cycle, or completing discovery. These are our OKRs, and I would sort of look at the faces of the folks in Mexico, they didn't speak that language. I think I learned through that process, that breaking larger problems in larger focus areas into smaller building blocks and making it very clear how those problems that you're solving ladder up to what those groups care about. In a company like Walmart, the idea of storytelling and getting buy-in was a really important skill, and that's something that I've taken with me since I've left.

Holly: Yeah. That's a really important skill at a lot of companies, but I think especially so at a big company where you have to really build alignment with a lot of people. Do you have any favorite ways to develop a story?

Caroline Rose: Oh, gosh, I'm definitely still working on that. I like to start with putting yourself in the shoes of whoever the end user is, or starting with an example of something that the person you're talking with would definitely understand. So the way that I actually best think about this is this is top of mind, coming out of the holidays when you're talking with family members that you haven't seen in a while, and inevitably they always sort of ask you, "Oh, what is a PM? What is a product manager? What do you do?" The best way that I like to describe that is I say, "Oh, think about your favorite app that you like to use," and inevitably, maybe it's Instagram, maybe it's Spotify, maybe it's Uber, and you say, "Oh, think about what is the feature in that app that you really enjoy?"

They'll give you an example, and then you sort of talk through, "Oh, a product manager is the person who identifies what that unmet need is and brings all the teams together in order to build it and make it successful." So I sort of take that same methodology when I'm trying to tell a story at work. I get asked a question, and I try to say, "Oh, well, think about an example of an app that you've used and what did it do for you? Well, here's exactly what we're trying to do the same thing here."

Holly: Okay. That's interesting. So you're able to sort of get them to think about something similar in their own life that helps them understand what you're trying to do?

Caroline Rose: Yeah. I think that's it. It's definitely, this is a hard skill to learn, especially in a business where you're working with highly technical people. Ultimately, what you're trying to do with a story is illicit emotion, and the job of a PM is, "Yes, it's a mix of trying to use data and logic, but on the other hand, you're also trying to elicit emotion."

Holly: Yeah. I think we deal with emotion a lot as PMs, whether we're dealing with the emotion of the end user or our colleagues or our leadership team. Emotion comes up all the time. So I'm interested as well to hear a little bit about how you've experienced the products community. I spend personally a lot of time in startups, and so talking to people who've been in big tech companies, it can be really interesting to hear sort of what their experience is like. So I'm curious to hear what are the ways that you like to engage with the product community, and do you find that a lot of the stuff that's out there applies to the big company experience?

Caroline Rose: I definitely have leaned heavily on things like conferences, product blogs, the women in product organization in general, especially as I was getting started in business school, they had tons and tons of resources where I tried to use them to hone my product thinking and be able to exhibit that in interviews. When I was at Walmart, I definitely used the organization to get that startup mindset. So being in a large company, the typical pitfalls that I think that you find are not being able to move fast enough or having to just check the boxes in terms of getting things done. A lot of the tips that I got in terms of running sort of low lift experiments and getting the data in order to get buy-in, those were ideas that I got from conferences that I attended and sessions that I would dial into from product startup leaders. I've heavily leaned on communities like that.

Holly: Interesting. So tell me a little bit more about how you get buy-in for running experiments?

Caroline Rose: Yeah, sure. Both at Walmart, and at Etsy, we strived and strived to be problem-focused and not solution-focused, and so as a product manager, that's very motivating. You dig into customer insights, you talk to them in focus groups, you read app store reviews, and you feel like you get a good sense of, "Okay, yeah, this is the main problem that I want to solve." But you turn around and you tell that to leadership, and they say, "Yeah, sure. We know that. What are we going to do to fix it?" And you say, "Yep. Don't worry. We'll figure it out. We're going to test and learn. We're going to test and learn."

So I think the one thing is going to them and not promising solutions, promising a commitment to solving a problem is part of it. But at the same time, you need to then follow-up later and say, "Okay, remember when I told you that we're going to solve problem A. We just tested these five things and here's what we've learned." So figuring out a way to break that potential problem into smaller problems, and then what are five small, low lift experiments that I can run to see which of these actually have legs. So it's the problem focus at the beginning, and then it's the following up with those stakeholders to show them the progress that you've made.

Holly: Awesome. Yeah. That's a very similar process to what I tend to do as well. I'd love to hear a little more about the types of experiments that you end up running. I think being at companies with such large user bases, I think that you can really run different kinds of experiments than you can if you're at a company with a small user base. So tell me a little more about what kinds of experiments you like to run?

Caroline Rose: Sure. I can talk about two themes that come to mind. A lot of the times, I would like to say, "Oh, yeah, we can build this low lift experiment. We can test it, and if we get a strong signal, we actually build the full-fledged solution." That would be sort of the ideal scenario. But a lot of the times, you don't even have the resources or the buy-in to run that low lift experiment. So we in the past have done a lot of, "Let's use design and build a clickable prototype that then we can put in front of users in focus groups or remote user testing and get their feedback that way."

So we've used very little engineering resources at that point. It's really just been designs help. They've stitched together something clickable that a user can actually interact with, and we have them look at it on their phone to actually simulate the idea of clicking through it in the actual app. The insights that we've gotten out of those experiments have been incredible without ever having to touch engineering capacity.

Holly: Do you have any more specific examples of a time when you did that kind of experiment and generated some insights?

Caroline Rose: Yeah, I've got one example from when I was working at Walmart, the main problem that we were trying to solve was increasing customer lifetime value. So I worked for the international business, and I was specifically on the Mexico market. A lot of people don't know this, but Walmart actually owns and operates a number of different brands of stores in addition to the Walmart brand name. So in Mexico, there was a higher end store, a lower end store, and then one that was sort of middle of the road and the middle of the road is the Walmart brand. So we had this hunch that all these different brands ran different websites and different apps, which were running on separate stacks on the back end. So we had this hunch, "Can we bring the customer experience together for all these different brands and have the user shop all these different brands in a single app?" Sort of similar to an Instacart type of experience. "Can we measure then the impact on frequency and basket size, and eventually users, we would hope would spend more, if we can make it easier for them?"

We built out the business case and got buy-in on the business side that, "Yeah, this is a problem that's worth solving." But we had a number of different problems to solve like, "Okay, do we want to build something that looks like an Instacart where you download the app, you choose which brand you want to shop, and then you could only shop from one store at a time and check out, or do we want to build something more like an Amazon experience where you can shop Amazon Prime, Amazon Marketplace, Amazon Wardrobe, all in a single basket and check out at once?" That may sound minor, but the implications across the customer journey are huge, depending on which direction you decide to take.

This was a big decision for us, which one we wanted to build out. So in this case, an MDP would actually have been very difficult to build, and so we worked with design, I think it ended up taking maybe three or four sprints. So again, not trivial, but comparing it to the time it would've taken to actually build. This was actually pretty efficient. We built out two separate prototypes, one, using the Amazon model and the other using the Instacart model. I think we gathered 8 or 10 customers from the Mexico market. We put the clickable prototype in front of them on their phones, and we just sort of had them click around and play with it. We had a couple of leading questions prepared. We had a couple of open-ended questions prepared, but really we just wanted to see how they would interact with these two prototypes. From that user group, we were able to determine the path that we wanted to take moving forward, and we hadn't used any engineering capacity.

Holly: Mm-hmm. So what path did you end up taking?

Caroline Rose: We ended up going the Instacart route, and it really came from the insight that the brand name was meaningful and strong to the customer. So we found that Walmart Express, which was the higher end brand, meant something to users. It meant that they could pay higher prices because they expected higher quality, and the Walmart brand name also meant something. It meant convenience, it meant reliability, it meant high assortment, and that came from years of brand awareness and brand equity that had been built by the organization. So something like an Amazon model wouldn't have worked where you see basically just all the products mixed together, and the brand name is sort of secondary.

Holly: That makes sense. So it sounds like the customers themselves had a good idea when they were going for a shopping session of what level of product they were looking for?

Caroline Rose: They did. They had very clear expectations and trade-offs that they were expected to make based on price, quality and assortment.

Holly: So I think it's really interesting because this is also an element that plays out a little differently when you're in smaller companies, you don't have that level of brand equity that a company like Walmart has. So I'm curious to hear what did the inner workings of Walmart look like in terms of understanding and agreeing on what that brand equity meant?

Caroline Rose: That's a good question. I think we worked closely with the business owners, the P&L owners, and we also worked closely with marketing, and their understanding of what the brand name meant would change as their understanding of our customer changed. So that was a challenge then from a product team perspective, because everything that we built basically had to be meeting the expectations of the Walmart customer and had to be sort of one step ahead in terms of what was expected from the marketing and the business team.

So very, very close alignment was required between all of our teams and making sure that, "Okay, we agreed three months ago that this was the important problem to solve, but based on our latest data is that still true?" So that was something that we would all do quite frequently is all kind of come to the table together, bring the latest data that we had and made sure that our understanding of the customer was the same.

Holly: Awesome. That's a really good practice, and it's harder in a big organization because you've got so many more stakeholders who need to be agreeing on that.

Caroline Rose: Totally.

Holly: So I'd love to hear a little bit more about what happened next in your journey. So you worked at Walmart Labs, working on this Mexico store. What happened after that?

Caroline Rose: I got great experience at Walmart, and I spent around two years there. At that point I decided I wanted to start flexing different muscles. I wanted to stay in e-commerce. The business was still growing and there was still a lot of room for improvement in terms of things that I could build. But at the same time, I wanted to work at a smaller, more tech-focused company with the idea that ideally it will be a little bit more challenging in terms of being able to move more quickly and the expectations would be higher. I'd sort of built a certain level of confidence in the sense that I'd gotten through my first formal role as a PM at a large established company, and so I felt like I was ready to take that next step. So I ended up at Etsy as a senior PM, where I work specifically on the mobile apps. So at Walmart I was working mainly on apps, but also on the website, and so this opportunity also allowed me to have a little bit more depth in the mobile space.

Holly: Cool. So you made the transition over to Etsy. How long ago was that?

Caroline Rose: I joined five or six months ago.

Holly: Okay, cool. So what has it been like there?

Caroline Rose: It's been really great. It's funny, I think e-commerce is not rocket science. It's a fairly established and mature product, but at the same time, I think I've been surprised by the fact that Etsy is a marketplace. We don't own our own inventory, and so our product really is a platform, which is no different than a lot of the other e-commerce platforms that are run. So the way that we think about how to optimize for both buyers and sellers and the way that we listen to our customer, is actually quite nuanced. So that's been fun to tackle an existing industry, e-commerce, but with a different mindset than I did at Walmart.

Holly: Yeah. So what have you learned in the first five or six months there?

Caroline Rose: So I think at Walmart, I think I learned a lot about stakeholder management. I learned about all the different tools that are in my tool belt in terms of how to build a good product. So basically how to work with research, how to work with analytics, engineering, design, and so on. I think one thing that I am learning at Etsy is how to better plan. So I'd mentioned before that my hope was that moving to a smaller tech-focused company, I'd be able to move faster, and that's absolutely true. I think I am able to move faster here just due to the size of the organization, and that has then caused me to have to be better at planning. So things that I used to think would take a couple of sprints are now taking a sprint or less. So it's more of a challenge to have, I guess, you could say, a continuous backlog of ideas and problems to solve, and making sure that I can work with engineering in advance and design and advance to have those things ready to go sooner than I would've expected to.

Holly: Mm-hmm. So give me some insight into what your experimentation looks like at Etsy?

Caroline Rose: I feel so lucky to be working with the experimentation team here at Etsy. We've got this great internal tool called catapult, and this was built by an entirely self-sufficient squad here at the company. So we don't rely on external tools, and it allows us to very quickly get experiments up and running on either the site or the app and track every single metric that you could imagine within those experiments. So some of the heavy lifting that I had to do in the past in terms of getting A/B tests up and running. So thinking about, "Oh, what events and what tagging do we need in place to make sure we can track X, Y, Z metric?" That's sort of a given here. So we're able to very quickly, within seven days, often figure out, "Okay, we launched this thing. What was the impact to conversion rate? What was the impact to clicks on this one button?" Those learnings, like being able to get those in seven days, allows us then to very quickly iterate and decide what we want to do next.

Holly: That's awesome. One of the things that brought to mind for me is systems that I've seen in the past for internal experimentation tools. I'm wondering, how are the lessons from the experimentation shared within the company?

Caroline Rose: That's a good question, and that's something I'm still getting a handle on. So this tool is accessible by anyone within the company. So you can click into it and say, "Oh, I wonder what the user acquisition team is working on." You can see what the latest experiments that they ran are, what the results were, and there's a little kind of free text field that you enter whenever you end an experiment, talk about what you learned and what your next steps are. So that's one way, but we also leverage Slack channels. So whenever your team has what's considered a win. So a win could be we moved a metric positively, or it could also be we moved a metric negatively and therefore learned something about it. We'll post those learnings in Slack.

We have a highly collaborative culture where you'll see that maybe I'm not quite currently working on anything related to user acquisition, but I'll join that Slack channel just to see what learnings they post about, and often their learnings that they have and their next steps will spark an idea for me that, "Oh, maybe on my team were working on something completely different." We could use those same learnings.

Holly: Mm-hmm. Cool. So it sounds like there's a lot of sharing going on. Does the tool itself have any context stored about the experiments that you're running, or is it sort of your responsibility to know the context?

Caroline Rose: You can share screenshots of what the control and the variant look like, and you're meant to sort of describe what exactly went into the variant experience. So I've been here five or six months, and I spent a lot of time trying to ramp up on all the context behind the current problem that I'm trying to solve, and what other teams have tried to tackle something similar, and so I've been looking a lot at past experiments. For the most part, you're able to look at it and know almost exactly what went into it, and it's very user-friendly in that way.

Holly: That's great. That's really helpful. Are you able to tell us what problem you're working on?

Caroline Rose: Sure. I'll give it a shot. So again, I'm working on the Etsy mobile app, mainly on iOS right now, just since the majority of our users are on iOS, and we sell millions of unique products that you aren't able to go to a store and touch and feel, or see in person. So we are trying to help our users better visualize the product in order to better gain confidence for it and buy it online.

Holly: Cool. Is that something that you can tell us a little more about how you know that that's a problem for users?

Caroline Rose: So we see the average Etsy user being very active on social media and frequently shopping on other apps like Wayfair, Target, Walmart, Amazon. I guess we basically take learnings from what our competitors are doing around social commerce, around AR, around video, and we combine that with learnings, from focus groups that we've done around, "Sure, maybe you find the perfect gift or perfect item on Etsy, but what prevents you from wanting to buy it?" Consistently the theme that comes out is, "I like the item, but I just can't trust it or because I can't see it, because I can't touch it, I don't feel like I can pull the trigger." So we're sort of bringing all those learnings together and try to figure out how we can solve that problem.

Holly: Got it. As you're going through that process of going through all of the learnings about that and ideating, what does the team look like?

Caroline Rose: We've got a squad of, I think it's around eight or nine right now. The leads are an engineering manager, a tech lead or tech architect, a designer, and myself. We've got a handful of iOS engineers, a handful of API engineers, and we have a couple of shared resources as well with another squad. So that's an analytics lead and a research lead and a QA lead.

Holly: Got it. So what parts of the team are involved in the discovery work or the problem space understanding?

Caroline Rose: Absolutely everybody.

Holly: Awesome. I love that answer.

Caroline Rose: That's an easy answer, right? I think it's easy to say that it's everybody. But what I've learned from working at a larger company versus working at a smaller company is that a lot of that comes from the culture that's set from leadership. So when we all get together as a team and we set our OKRs for the quarter. This is another plug that OKRs are a great tool to use for teams. This is a cross-functional effort. So this is something that marketing, research, design engineering is all bought in to one set of OKRs.

So in my experience, then, that flows very nicely into product discovery. So because we've all gone together and set our OKRs for the quarter together, it of course makes sense that when we go into product discovery we're all doing it together from our own functional points of view. Whereas I've seen in the past, when team goals or team incentives are not the same or not even laddering up to the same thing, getting everybody on board to then do product discovery together is not always successful because it's not set up that way from the beginning.

Holly: Yeah. You said it so simply it made it sound really easy, but I know that it's not. How do you set OKRs together?

Caroline Rose: I think the first step that we took was we first looked at what our company-wide OKRs are, and those were published for everybody to see, and because we are a public company, a lot of them don't come as a surprise. It's the same information that's given to the general public. So after we take a look at what our company OKRs are, we then turn around, and we look at the latest customer data and insights that we have. So usually from the prior quarter, the problems that we're trying to solve haven't changed too much. So in this case, the problem that we've been working on, we've been working on for a couple of quarters now. So we do another gut check and say, "Yep, okay, we think this is the most important thing that we should be working on."

Then from that point, it's coming together, I think, particularly with engineering to say, "What is reasonable for us to expect that we could get done this quarter?" Sometimes in one case, it is, "Let's launch this experiment and get learnings from it." But if it's a multi-quarter project, sometimes it will be, "Let's finalize the designs for this one phase of the project, or let's run this one set of user testing for the project." So it's helpful once we have a good sense of the problem that we want to solve, and then it's a high estimation process of, "What do we think is realistic to get done in the next three months?"

Holly: Mm-hmm. Well, it sounds like a very collaborative environment.

Caroline Rose: We try to be. Again, I think it comes from the culture at the top, too. The way that our teams are set up, the fact that we are all dedicated towards this one squad and this one set of problems, it makes it a very natural process.

Holly: Yeah, absolutely. You mentioned squad, are you following the Spotify framework?

Caroline Rose: I'm actually not familiar with the Spotify framework. What is that?

Holly: Well, it's squads and tribes. Basically, it's a particular framework for setting up your teams and saying, "Okay, there's essentially cross-functional teams that have everything they need to get something done, but then there's also horizontal teams where there's some responsibility to communicate and collaborate." Say, there's a UI engineer on multiple teams, and then there's the group of UI engineers as well, that are connected to each other in another way.

Caroline Rose: Yeah, that sounds similar to what we do. It's basically that a product manager, an engineer and a designer will always be on one self-sufficient squad, but then under us, there are platform squads. So there might be, for example, when we need content help, there's a content designer who helps all the different squads with their content needs, and for analytics, we have a shared analytics resource with one other squad. So that sounds about right.

Holly: Got it. Cool. All right. Well, I think we're almost out of time. So my last question to you is just what advice do you have for other aspiring product leaders who are working through their career and product to help them reach their next step of their journey?

Caroline Rose: I think, within the last year or two, we've seen a lot of turnover at a lot of companies. People getting burned out with working from home and whatnot. I think Etsy has seen a surprising level of people stay at the company. So I've been trying to reflect on what is it about the company culture that's keeping people here? The best answer I can think about right now is that I think leadership has a really good handle on how employees are feeling, what their biggest problems or roadblocks are with their job, and they have a good handle on this by frequently giving us surveys to fill out, and they take that data and do something with it.

So they will frequently tell us, "We heard this from you, and so therefore, we're going to try doing this new process or this new thing." So I think first, piece of advice would be listening to your teams and caring about what their roadblocks are and doing something to try to fix it, and the second thing would be using a test-and-learn approach. So it hasn't been this wide-reaching edict saying, "We are going to do this, no ifs, ands, or buts." It's, "We're going to try this and then reach back out to you in three months and see how it's going."

Holly: Yeah, that's awesome. I've always liked working at companies that gather data and then actually do something with it.

Caroline Rose: Practicing what you preach, right? With being at PM.

Holly: Yes, exactly. Well, it's been a pleasure to talk to you today. How can people find you, if they want to learn more.

Caroline Rose: I'm on LinkedIn. If you'd like to connect with me, I'd be happy to.

Holly: All right. Awesome. Well, thank you so much for your time, Caroline.

Caroline Rose: Thank you, Holly. Have a good one.

Holly: The Product Science Podcast is brought to you by H2R Product Science. We teach startup founders and product leaders how to use the product science method to discover the strongest product opportunities and lay the foundations for high growth products, teams and businesses, learn more at hqrproductscience.com. Enjoying this episode, don't forget to subscribe so you don't miss next week's episode. I also encourage you to visit us at productsciencepodcast.com, to sign up for more information and resources from me and our guests. If you like the show, a rating and review would be greatly appreciated. Thank you.

‍