The Matt Wallaert Hypothesis: Great Product Teams Use Behavioral Science to Build Products That Create Change

Matt Wallaert is a behavioral scientist working at the intersection of technology and human behavior. He headed product at two successful startups, then Microsoft and Microsoft Ventures, and is now the Chief Behavioral Officer at Clover and author of Start at the End: How to build products that create change, with a slate of pro-social side projects including GetRaised, SalaryOrEquity, and IAskedHer. In this episode of the Product Science Podcast, we talk about how Matt gets his team to focus on the behavior they’re trying to change, and his unique research process.

Subscribe for the full episode on Apple, Google Play, Spotify, Stitcher, and more. Love what you hear? Leave us a review, it means a lot.

Resources

• Start at the End: How to build products that create change by Matt Wallaert
• MattWallaert.com
• @MattWallaert on Twitter

Questions We Explore in This Episode

What carries over from Matt’s experience in product management into his role as Chief Behavioral Officer? What has he learned from talking to his company’s most frequent customer service callers? What changes in customer behavior does he look at? How can supporting and helping other people serve as an effective motivator to drive behavior? What are the shortcomings of looking at product from a purely transactional model?

What are the inhibitors people consider when deciding not to use a product? What are the limitations of trying to maximize value? Why does Matt encourage product teams to think about how they can create things that require less investment? How do you approach the problem of limited customer attention? Why did Matt build a script to automatically buy clothes on eBay?

What does Matt do in his role as a Chief Behavioral Officer? What is his team’s research process like? How do they move from a pilot program to implementing a solution at scale? What question does he ask when he’s hiring quantitative scientists? Why does Matt encourage teams to take a closer look at data they might otherwise throw out? Why do companies hire behavioral scientists, and why is it hard to find people with experience?

How did Matt approach writing his book to help organizations develop a behavioral science research process? Why does he emphasize building a process and methodology? Why is it so important to help understand what it is that you’re building? What are the unique problems Matt encounters building a behavioral design process?

Why does Matt emphasize laying out what you’re building in terms of how you’re trying to change behavior? What is a behavioral statement and how do you write one? What was it like doing the audiobook for his own writing? What advice does Matt give to behavioral scientists who are just starting out? Why is finding friends so important in your career?

Quotes From Matt Wallaert This Episode

When I watch people build product these days, it looks like a rational economic transaction. The user gets this from us and they give us this, they trade us data for music listening or whatever, and that's such an impoverished way to think about how humans are actually motivated.

What we do in behavioral science at Clover is we take strategic priorities for the company that are behaviorally based and then find interventions that change those behaviors.

You want to scale your certainty with the resources that you're committing and with the size of the decision. If it's a big decision you want to be really certain.

Picking a process or framework to follow will make it easier for other people to understand what your behavioral science team does.

‍

Transcription

‍

Holly Hester-Reilly:

Hi and welcome to the Product Science Podcast where we're helping startup founders and product leaders build high growth products, teams and companies through real conversations with people who have tried it and aren't afraid to share lessons learned from their failures along the way. I'm your host Holly-Hester Reilly, founder and CEO of H2R Product Science. This week on the product science podcast. I'm super excited to share our conversation with Matt Wallaert. Matt and I started chatting on probably Twitter because Twitter is amazing. It's like a salon, you can just walk in and talk to people. And I've been following his writing and podcasts and such since then. He's recently put out a book and I'm super excited to have him here. Matt, welcome.

Matt Wallaert:

Thanks for having me. I don't know, Twitter is sometimes like a salon. Sometimes it's like the streets in Harlem where someone just yells at you, sort of [inaudible 00:00:58], I don't even know what you're saying.

Holly Hester-Reilly:

Yes. And there's that crazy person who's just yelling stuff off in the corner to themselves and you're like, well.

Matt Wallaert:

I actually, I'm a social psychologist by training and I have a sort of list of things that if I ever have a lot of time I'd really like to look into. And one of the things that is most fascinating to me is I would love to have conversations with people who have 100 followers and 100,000 tweets. You're saying a lot to very small number of people. What is going on for you? What part does that play in your life? Is it like you're talking to yourself? Is it an emotional thing? I don't need it in a mocking way, I need it in a very serious sociologically, what is happening in this really micro community that you are like, 10,000 tweets or something to, it's just so fascinating to me.

Holly Hester-Reilly:

Yeah. That would be really fascinating. When you do that, let me know.

Matt Wallaert:

I think that in the product sense, I'm always fascinated by outliers, I'm always interested in who's using my products in unconventional ways? What is going on that you're logging in every 20 minutes, you're doing real things, you're not a bot, but what is going on? How are you using this? And I actually over the years I've had a chance to talk to, I try and find those outliers and have conversations with them. And I think it's always really a lucid area and you find out really interesting things about why they're using your product in totally interesting ways. And sometimes it's [inaudible] hearing it and sometimes it doesn't take you anywhere, but sometimes it can really push you in really startling directions where you're like, Oh, I never in a thousand years would have thought of my product that way, but I can understand what's going on for you as you do that. And sometimes it pushes me to build new things or thinking different ways, and I love that.

Holly Hester-Reilly:

Can you tell us about a time when you discovered something that was very surprising from one of these outliers?

Matt Wallaert:

Wow. I'd have to think about it. It's been a long time since I've managed product. I always manage from a behavioral science perspective, but as I increasingly became less of a product leader and really fulfill the chief product or chief behavioral officer sort of role, but sort of Microsoft and here at Clover we have a chief product person who does great product that, it's far away from me. It's an interesting question. One of the things that we still do at Clover are not related to product but we still look at those hours like who calls our call center if yay and what is going on from them. And it's interesting we're looking at a particular person and I'm going to try and do this in a way that doesn't violate PHI.

Matt Wallaert:

We were looking at someone who was calling us a lot, not every day but on the order of 100 times a year, 150 times a year, every other day or very close to and often relatively angry about relatively minor things. And we had the impression that he might be sort of not inventing consciously, but finding reasons to be angry at Clover. And we delved into it, he was definitely a little on the lonely side. He lived alone. His wife had died a couple of years ago and it was interesting to sort of sit down with him and talk about how he sort of related to Clover in his life. He was cantankerous, he wasn't mean spirited. He would call us in a sort of cantankerous way. We actually have an experiment running right now where we have put a consumer device that you can talk to of one of the larger tech companies into people's homes.

Holly Hester-Reilly:

[crosstalk] that be?

Matt Wallaert:

There's more than one, you don't know.

Holly Hester-Reilly:

I don't know. It's true. Okay. You put something that listens into people's phones?

Matt Wallaert:

Listens and talked back. And we're [inaudible] actually at how people engage with those. We've built a small program that basically once a day they check in with us. You say device check in with Clover and it says, "How are you feeling today?" Sort of like awful, okay, good, great. And then we track that data and then we actually reach out. If you sort of say enough days in a row that you're not feeling super well and we're looking at are there ways to give people the social satisfaction that they felt from calling in without literally calling us in all the time because then that jams the phone lines and we do have actual medical things we need to process and it's a fun sort of thing. We look a lot at outliers. We look a lot at switchers.

Matt Wallaert:

This is something I think people don't do enough. I always look at when you think about my group, the behavioral science team, our job, we are given behavioral outcomes that the company needs to accomplish. And then we then let's say everybody needs to get a flu shot. Great, let's take that as a behavioral outcome. Let's figure out why it's not already happening. Let's figure out what we need to do to create, to make that happen. And to what two of the groups, we look at outliers a lot, but specifically we go to two groups who are people who consistently did it, but all of a sudden stopped?

Matt Wallaert:

You were our member for five years you got flu shot, flu shot, flu shot, flu shot. All of a sudden last year you didn't get your flu shot. We interviewed them to understand what changed that then caused you to no longer do what had been a steady state behavior. And then vice versa, you've been our member five years, you never got the flu shot, you never got the flu shot, you never got the flu shot, you never got the flu shot. All of a sudden you start getting the flu shot, what happened? And we've found really interesting things. For example, people who get the flu shot for the first time, very often it's related to the birth of a grandchild. Grandkids happens, parents say you cannot see the kid if you do not get a flu shot.

Matt Wallaert:

This person who would never get a flu shot for themselves, we could tell them it will prevent the flu, they could get the flu, they could be really sick with the flu, they just don't get the flu shot that day. I believe that is an important part of their health routine, will do it on behalf of someone else. And that actually led us to a really interesting place where we created at Clover, we track people's sort of personal health motivations, why you want to be healthy. And we started actually last year a flu campaign that was personalized to your personal health goal. And by and large, it usually never says, oh, you should get it for you instead, it is almost always some other exogenous reason you should get it so you don't get your grandkids sick. You should get it so you don't get your wife sick, you should get it so you don't get your church congregations sick. And that's really highly effective. Things I would never ever, ever do for me. I would easily do for someone else on behalf of someone else.

Holly Hester-Reilly:

That's amazing. Can you tell us more about what's going on there or how you understand that?

Matt Wallaert:

Yeah, I mean I think that, as a psychologist, you start to look at motivation and in a sort of like rational economics, I mean this is why there is no Nobel Prize for psychology. But over the last several years we've won the economics, Nobel Prize has been given to psychologists and the reason is we've proven that there's a departure from this sort of homo logic version of person, who is self interested and does what maximizes self interest. It turns out, self is often not a particularly good motivator for many, many people, they've been socialized that into collectivist sort of things.

Matt Wallaert:

And that's, it can be a really interesting, I don't think we use this enough particularly in tech products, is often viewed very much from a sort of personal lens. You take it something like Spotify, and the model in mind that you have of a person who listens to Spotify is about them. Maybe you expand it and you say, well it's social because they get to look at other people's lists and share what they're listening to and it's a self presentational. Okay. That's getting a little more away from this rational, music brings me value, economic value. But I think there actually can be a really interesting things that come out where people ... you and I were joking before before the show about last names.

Matt Wallaert:

And we talked about how your last name could sound like a really awesome band name, Hester Reilly's like a really awesome band somewhere in the Pacific Northwest. I'm sure. Think about the number of times people go out to a show to support the band or to support their friends. They would never individually choose to go do that on my own. I would never go see Hester Reilly, the band on my own, but my friends, the bassist or her girlfriend's the bassist or whatever, there's some really tenuous thing where now is important to someone else that I show up and that's an incredibly effective motivator. And I think expanding our sort of motivational set to look at people's willingness to do something on behalf of another person, on behalf of a culture, on behalf of an identity statement is really interesting and really underdeveloped.

Matt Wallaert:

I think in modern product management, when I watch people build product these days, it often is very much, looks like a rational economic transaction. The user gets this from us and they give us this, they trade us data for music listening or whatever, the sort of economic trade is, and that's such an impoverished way to think about like how humans are actually motivated.

Holly Hester-Reilly:

Yeah, I love that. And I think it has a lot of resonance for some of the even recent conversations I've been having with people that we coach on product discovery and what do you take away from the conversation you have with the person about their stories and behavior. Because people are so often, especially with the early stage startups or the new product launch that someone's doing, they're just like, "But this would make their life better. It clearly would, it'll win. Right?" And we have to be like, that doesn't mean that they'll use it.

Matt Wallaert:

That means actually nothing. There's lots of things that would make our life better that we just don't do for lots and lots of reasons. If that was the right model, we all would go to the gym. None of us would pollute. There are all sorts of other notions in play and you and I have talked about the book before and the notion of promoting and inhibiting pressures and this idea that they compete. You and I just talked a lot about motivation, but there's stuff on the inhibiting pressure side too where you're like, yeah, but it would bring value to their life. I'm like, yeah, but it's a pain in the ass and it's just not worth it.

Matt Wallaert:

There is a number one product Hobgoblin of early founders and first time founders is their belief that the pros outweigh the cons, they're like massively overweight, the pros and massively underweight the cons. You and I both have kids, I'm busy. I’ve got a lot to do. I'm very thoughtful and careful about how I use my mental attention and energy. Yeah, sure. I'm sure your thing would be great, but I'm not spending mental time on that. I think what's interesting is that has given rise to a whole interesting host of startups in recent years that are not about providing additional value, but simply about minimizing costs. And I don't mean economic cost, but I mean mental attention cost. Taking an Uber X is not better than taking a cab, it's arguably worse, but it is infinitely more convenient.

Matt Wallaert:

And that infinite convenience factor, I just think this is a very different model of how people build product. You go to someplace like American Express and finance companies talk about their products as share of wallet, of the financial transactions that Matt is doing and of the money Matt spends, how much is going on his American Express card? And their job is to maximize the share of wallet, which is very much our product is traditionally been built. I mean, I think one of the crises of Facebook is they were like, our job is to get as much of Matt's attention as possible, here's the pie that is Matt brain, I'm going to try and take up as much of that as I can. And that's a really impoverished view of product building.

Matt Wallaert:

Because it is perfectly viable to also say, I will hold value constant. I will give Matt no additional value. I'll even make it in the case of Uber's slightly worse, but I will make it just require less resources rather that's time or money or mental attention or what have you. And I think there are often, there are a ton of companies that are leaving massive value on the table because they're only fixated on how do I provide more value? How do I provide more value? How do I provide more value? And they're leaving off the sort of I portion of return on investment. They're just like more return, more return. I'm like, what about less investment? Can we do that? Less investment?

Holly Hester-Reilly:

Yeah. And I think we're seeing more and more of that these days because everyone's attention is so saturated all the calls, everyone's just like, Oh my God, everyone's starting to realize, well maybe not everyone, but I feel like there's a lot of realization in the past five years about how our attention budget is limited and we have to actually be, it's better if we can be intentional about how we spend it.

Matt Wallaert:

Yeah. There's no more illustrative example for me than alerts on phones. I recently had to wear an Apple watch for an experiment that we were doing here. I was trying to get familiar with the device and it was virtually useless to me. I was like, why does everybody love this thing? And I don't use it at all. And what I discovered is everyone else has their alerts on and I don't let anything, nothing gives me alerts, text messages don't, my phone vibrates when it rings. Otherwise, I check it when I check it. Everything is allowed to put something on the badge. You can tell me there's something there that I'm supposed to be looking at and whenever I have phone time I'll look at it, but I'm not going to let you like interrupt me in whatever else I'm doing.

Matt Wallaert:

And I just realized this is massively different than how other people are treating their devices. But I think if everybody did it this way, we probably would have a lot better net effect, because one of the problems of the American Express, shared of wallet version of the world does, it means everyone is in competition with everyone else. Whatever you're building and whatever I'm building, even if they seem like they are so far apart, you're like, I don't know, often the exercise space somewhere and I'm over here in games or something, just really far apart. No. Still limited pile of mental attention that I can spend. And we're just in a nuclear arms race for who can land grab the most of my attention.

Matt Wallaert:

And that just, I think result in a really impoverished world. I think people are going to start waking up and getting smarter and saying, you know what, I want to spend my mental attention in the places where I care. I care about my son, I want to spend mental energy there. I like computers and so I want to read about and spend time thinking about computer hardware. I don't care about clothes at all, and I literally have automated that. I built a little script on eBay, it buys the same clothes. I wear those clothes, I haven't done shopping in years.

Holly Hester-Reilly:

Wait, is that real? You've built a scripter on eBay or are you making that up?

Matt Wallaert:

No, that's real.

Holly Hester-Reilly:

Wow.

Matt Wallaert:

I can guarantee you that if a John Varvatos 40 yard blazer sold for less than $20 it's in my closet and I wear the same.

Holly Hester-Reilly:

Wow. That [inaudible 00:17:21].

Matt Wallaert:

And I wear underwear from Costco, Kirkland man underwear and socks from Darn Tough socks. But other than those two, everything else is used. They're used black cowboy boots, used John Varvatos jeans, used Nordstrom trim fit shirt, everything comes from eBay. It's all green. It's all very cheap. It's all perfectly environmentally friendly because it's all reused, if doesn't fit, I feel no guilt. New jacket shows up and doesn't fit. I spent $20 on it. I don't care. I just read donated to the Goodwill and it'll go to someone else like jacket, if you come up and throw red wine all over me, I'll just shrug and be like, okay, nothing I'm wearing cost more than $20, so sure, okay.

Holly Hester-Reilly:

Oh, that sounds really cool. I try not to put much energy into clothes, but I haven't automated it that well.

Matt Wallaert:

I think that you will increasingly see, and I think there's a whole new product class that will come out around automating the things we just don't care about. The rise and fall of Blue Apron I think was a dramatic characterization of the misunderstanding of automation. Blue Apron sold the world, sold their advertisements, sold their investors, sold the world on automated food and that is not what Blue Apron is in any way, shape or form. It takes an hour-

Holly Hester-Reilly:

100% agree.

Matt Wallaert:

I am a decent cook. I grew up cooking. I don't know what this word means. You're like, do this thing. I'm like, I don't know what I have to Google that cooking term, that's not automated food. That's for people who want to spend all their time thinking about food.

Holly Hester-Reilly:

There were many nights where I was the only one who wanted to eat it and everyone else was like, "Mom, just make us Mac and cheese." I'm not using it right now. But it definitely like many people, I started with Blue Apron and then I was like, Oh my God, this takes me like an hour. This is not better. I have little kids I can't, it takes me even longer. They have the thing, I'd be like, Oh, in 40 minutes. And I'm like, yeah, but not 40 minutes when there's two kids hanging off your feet.

Matt Wallaert:

Even if it was 40 minutes, I have no desire to spend 40 minutes ... I have a behavioral outcome I want to feed my kid, things that broaden his palette and that are reasonably healthy and I want to eat things that broaden my palette and are reasonably healthy. I don't care if that's a frozen thing that I put in the, I don't care. I don't need to cook. I don't have anything special. ~The only reason people like cooking things is because they feel like they're less processed and they know stuff about them. Fine. Just give me less processed stuff that I know more about.

Holly Hester-Reilly:

Yeah. Totally.

Matt Wallaert:

Years ago New York city was doing, they were talking about what the apartment building in the future could look like and they weren't actually going to build it. They were just sort of musing about it. And they came and talked to me and I was like no kitchens or one kitchen per floor like a dorm and then subsidized cafeteria in the basement and not shitty cafeteria food that will make you sick but healthy mass produced but healthy, fresh, at a decent price point food in the basement and everybody goes down there and it's commutable. Put it on the top floor.

Matt Wallaert:

No penthouse. The most beautiful view in the thing is the cafeteria. Everyone goes there, kitchens are a massive waste. That means you have to run gas lines to every apartment, which has its own risk factors and things and it's very expensive. You have to have a serious electrical in there. People have to have storage for plates and knives and cups and forks and things, why? Why any of those things? That is a massive, massive waste. And I think that what we will start to see at a structural level in society is many more places where we say, hey, if you just sketch this, no one would build it this way.

Holly Hester-Reilly:

Yeah, I agree. I've seen a lot of the evidence of that starting to happen and I haven't looked into it much, but I guess is that a little bit what we live is like, have you looked into that at all?

Matt Wallaert:

Well, I don't know much about we live, I know a fair amount about common, which is Brad's thing and they have automated a bunch of things. What would be fascinating to me is trying this with existing hotels, which have a lot of this infrastructure. Right. But actually did a sort of thought experiment once were where I looked at what would it take to buy a 500 room hotel in Boise, Idaho and then convert that to something where engineers could come and work and live and have a decent sort of, I think dorm style living will make a comeback. [inaudible] biggest problem is actually just trying to find real estate that fits its needs and templates and bolts at a price that works.

Matt Wallaert:

Common, we work, all those things are actually real estate plays as I understand it from the last time I talked to her head, the demand at Common is very, very, very high. They have more demand than they have supply and that doesn't surprise me at all.

Holly Hester-Reilly:

Tell us a little more about what Common is?

Matt Wallaert:

Yeah. Common is like WeLive, the idea is, it's a standardized sort of lay out apartment type communal living thing. Mainly I think for millennials and Gen Z tend to sort of be the people who gravitate towards there. And I can understand why there are other versions, but you and I both have kids. Wouldn't it be amazing to live in a couple of style buildings? Sort of like you didn't have to lock the doors. The kids run everywhere, school's in the basement.

Holly Hester-Reilly:

I've been fantasizing about having an actual tribe and in modern America, you wouldn't believe, if we could actually live with people who all wanted to help each other with the things that are involved in living, that'd be amazing.

Matt Wallaert:

Well, and it creates, there's massive economic potential there, the need for you and I and everyone else to employ separate nannies, we don't want New York school public. We don't want New York public schools, where it's like one teacher to 25 kids, that's too much. But certainly a nanny can handle three kids or four kids. Imagine if we just took literally the national expenditure on solo childcare and we said, No, all of the nannies are going to have four kids.

Matt Wallaert:

Totally manageable size. It's actually not that hard to manage four kids. We could quadruple the salaries of nannies across the country, lifting them out of poverty, doing many things that are very, very good for the country and giving them a living wage or triple their salaries and everybody childcare costs come down a little bit for the parents who are already paying, which is great. And nannies get a living wage now, wouldn't that be amazing? And it's those kinds of economic inefficiencies that drive me just a little bit nuts. You start to see it around car sharing, tools sharing, some of these other things that have started to come out. But the problem them has always been there're software solutions.

Matt Wallaert:

Everybody's like, I'll build a software solution so that you don't have to get your own weed wacker and you can borrow my weed wacker, but that's actually a very good model. Good model is there's a shed that everybody has keys to and the weed wackers community owned. I think what we work in Common and other people have figured out about real estate is it's not Airbnb. It's not a software platform to unlock some vacant apartments somewhere. What has really made Airbnb work is people running Airbnb apartments, the Airbnb, it started with the idea of I have this unused property some of the time and therefore other people can use it some of the time. That was Uber too. You have an existing car and then you will use that car some of the time.

Matt Wallaert:

That has largely gone away. What Airbnb is actually powered by is, I will buy, refurbish, rent out and run as a small hotel room, essentially an Airbnb. I will buy a car specifically and only for Uber and then we will get we'll get four different drivers to drive the same car in rotation. That's the model that works. It's not borrow my weed wacker, it's an owned and operated weed wacker that I can pay a smaller amount of money to access on a regularized thing. If we go back to nannies, right, it's not, well you can come drop your kids off with my nanny for a little bit. It's a nanny company that has a three to one ratio or whatever and has its own space and blah blah blah blah blah blah.

Holly Hester-Reilly:

Yeah. And if everyone's living in the same building or something, then you get more of those benefits too. Because I've also done childcare, we have to drop them off and it has definitely like, Oh it's so much easier in the morning when the nanny comes to your house.

Matt Wallaert:

Yes. Wait, we're in the same building, I think building are a really interesting place to this too, to your point. Companies like Alfred where one Butler for the buildings, sort of versions of that. That's technology enabled, but it's not a technology business, it's technology enabled business. But like some smart systems coupled with a human. Why couldn't you have the Alfred of childcare where sort of there's one nanny for the building, she or he lives on the first floor, knows all the families in the building. There's a small communal space run. That to me, it's tech enabled within the building, that stuff makes a lot of sense to me. I think buildings will become the new unit of sales very, very quickly. I love Alfred's model precisely for this reason, I think you'll start to see people paying escalating amounts of money for a package of services that include a bunch of things.

Holly Hester-Reilly:

Yeah. Interesting. I'm thinking I'd love to hear a little more about what your role is today. Because I know you were aiming to become a chief behavioral officer and you are and there's not so many of those in the world that everyone knows what that is. Tell us a little more about what that is.

Matt Wallaert:

Yeah, sure. I run a team of about 10, it's a mix of quantitative scientists, qualitative scientists and project managers. And basically what we do at Clover is we take strategic priorities for the company that are behaviorally based and then find solutions, interventions that change those behaviors where they have not previously existed. Flu shots as we talked about earlier, it's actually a pretty decent example. Nobody has done a very good job of getting flu shot penetration to be all that high. That's why flu shot penetration in the United States is not amazing given that it has no side effects and prevents the flu, which is horrifying and kills people, flu shots who shot penetrations are still not all that high. And you can take something like that, which the industry hasn't figured out. And then we go through a really regularized 10 week process.

Matt Wallaert:

And so we spend the first week really flushing out what is the behavior we're trying to get to? Then you spend two weeks doing qualitative and quantitative research. And really mapping the territory of the world as it exists today. We often use it in my team, this sort of sci-fisc analogy of right, trying to create an alternate universe. The universe that is exist today, we want an alternate universe in which this behavior performs this way. How do we create a bridge between those two universes? And yeah, first sort of three weeks, we're really trying to describe the universe as it exists today. And the quants go off on their own, the quals go off on their own and then at the end of week two they switch.

Matt Wallaert:

And the quants say, here's what we found, the quals say, here's what we found. And then they spend the next week trying to validate or invalidate that, can I find data-based evidence that out the things that I'm hearing from members? Do members say things that are reflective of what I'm seeing in the data? By the end of week three, you've got a good handle on what you're trying to do and you have a good handle on why it's not already happening and why anyone would want to do it in the first place. Those are two dominant questions, why would somebody want to do this and why are they not already doing this? Then you go into a two week sort of design sprint where week one, you're really looking at those pressures, trying to map them out, trying to sort of get your hands around that data and understand what's going on.

Matt Wallaert:

And then week two, you're really flushing out the details of the interventions that you're going to try. And then you spend the next several weeks running those interventions. We actually bring three to five pilots to market and then measure the outcome and they're designed to be ultra light. You want it to be just heavy enough to get the behavior but they don't need to be statistically significant. I'm not trying to run 500 people, I'm not trying to do all that. I'm just trying to understand does it sort of seem to work in the world? And then once we have traction at that, then we go run bigger stuff, then we start to bring stuff to test, start to look at how would we scale this? And when we did it in the pilot we did it with duct tape and spit and how would we make the version that is a little bigger?

Matt Wallaert:

And then at scale we usually hand it to sort of more the operational side of the business. And we sometimes use the engine building analogy here or the machine building analogy. The pilot for us is very much like, what the heck are we trying to build? It's like the Rube Goldberg version of it, it's made impossible. It's not very good. The test version is much more like, okay, now stick a crank in it, make it out of aluminum, write an instruction manual that someone who didn't go through all the work that we just went through could have access intellectually to what's going on and why it is the way that it is and knows how to turn the crank and then make it out of steel. Put a real crank in it, write a really good instruction manual and hand it to the operators to say like, okay, turn the crank, here's the instruction manual, let us know if it breaks and then we do some instrumenting to make sure that we get continuous monitoring, but then we move on to the next, next project.

Holly Hester-Reilly:

That's awesome. One of the things I loved about what you said there is the that the pilot isn't about statistical significance because I feel like I come across that so often in people who are early in their lean and agile, experimentation focused approaches. And then you go and do something and they're just like, "But how could you make a decision based on that? It's not enough." And it's like, well I'm not putting it out in a journal for the next a hundred years of scientists to take as gospel. I'm just going to accelerate to the next phase of research.

Matt Wallaert:

Yeah. I think this is what happens when people don't actually have a core understanding of the tools that they're using, people are using the tool of statistics without actually understanding what the hell anything means. And I have a funny story that I tell that happened here at Clover where the base rate for pharmacy by mail getting your drugs in the mail is about 2% in the Medicare advantage population, it's very unusual. And we found an intervention that moved it to 50% so 50% of people would do the thing-

Holly Hester-Reilly:

That sounds huge. Wow.

Matt Wallaert:

Chemi is massive. And we did it with 80 people and so I was talking to some executives about this research and one of them was like, "Wow, 80 doesn't sound like very many given the total size of our population. Is that statistically significant?" And I was like, I'm sorry, is the difference between three and 50% statistically significant? That's statistically significant in 20 people.

Holly Hester-Reilly:

Wow.

Matt Wallaert:

The expected outcome for 20 people if the base rate is 3% is zero, no one will uptake. Or it's like hovering the line of maybe one, we got 10, 50% then is the difference between one and 10 real?

Holly Hester-Reilly:

Yeah.

Matt Wallaert:

Yes, the difference between one and 10 is statistically significant. People just throw things out I think without really understanding what they mean. And he was well spirited and asking the question. He just like, yes I don't even have to do the math. Yes. Then on the flip side of that, to your point you want to scale your certainty with the resources that you're committing, with the size of the decision. If it's a big decision you want to be really certain. You and I were talking about does one have a second child? That's a big decision. You should be really certain before making that decision.

Matt Wallaert:

Does one have a hamburger for lunch? Right? If you're wrong, the penalties are low, you have indigestion in the afternoon, it's not like the end of the world. Because you have frequently give him, we're hiring quantitative scientists as opposed to following sort of question that tries to get them to think. I'll be like, imagine I have an intervention where P equals 0.2, significance is 0.2. A brief history of statistics for people in the room or listening, significance P, the P value tells you how likely it is that the effect that you're measuring is not due to random chance.

Matt Wallaert:

That is in fact a real and genuine thing that results from your intervention. And 0.2 means there is a 20% chance that you cannot reject the null hypothesis. Meaning there's a 20% chance that what you are observing is due to random chance. As a contrast, the statistical norm used for publication, generally speaking in psychology is P equals 0.5 in other fields is often P equals 0.1, you need to be 99% confident is real or at least 95% confident is real. Psychology has a somewhat lower benchmark because people are so complicated that there's so much variance, it's very hard to get that specific. In traditional academic senses, P 0.2 would be nothing. People are like not is functionally nothing, even though you're 80% certain that something real has happened, that is not enough.

Matt Wallaert:

And most quants will say, "Hey, you got nothing." I say, well, let me tell you about this particular intervention. It's a tiny pill. It's very easy to swallow. It dissolves in your mouth, has no side effects of any kind, costs a penny to produce, tastes like fucking unicorns and rainbows and instantly cures all forms of cancer forever. Think maybe we should not-

Holly Hester-Reilly:

I [crosstalk] find that.

Matt Wallaert:

We're only 80% certain it does that. We might want to move that to some further investigation before we [inaudible] on the public. Sure. We should get absolutely 99.999% sure before we give it to the public. But we probably don't want to abandon the project. And I think it's like mining for gold. Sure. Some of the veins are going to trickle out, you're going to bet on something that's P 0.2 and you're going to find out it was the one in five chances that you are wrong, but pursue that gold, because there might be a motherload underneath there, you might cure cancer. And so I think that we just, people abandoned ship way too early. Oh, that didn't work on the first revision of the thing that we did didn't work, [inaudible 00:37:25]. I don't understand why people are super revisionist and about product, they're like, Oh I get that. It's about iteration. But they're like abandon that altogether when it comes to experimentation. Oh, the first version didn't work. Guessed that whole line of research is useless and I should go away.

Holly Hester-Reilly:

Yeah. It shocked me the first time I came across that, I see it a lot in companies that do a ton of AB testing and maybe they do a little bit of research to figure out what they should AB tests, but they just throw a lot of stuff at the wall. And then they run that AB test and then they're like, "No, didn't work. I'll just throw the whole thing out." I'm like, wait, what?

Matt Wallaert:

And their criteria for didn't work. I'm like, Whoa. It was real close to working thing. Maybe you should iterate on, you were in the neighborhood of right.

Holly Hester-Reilly:

Yeah. people definitely do that. It's amazing. Do you see more growth in this behavioral, I don't know, chief behavioral officer and behavioral teams? How's the uptake on that in the business community?

Matt Wallaert:

I think it's hard to know. I think there have been swings up and swings down. There are companies, I'll try and draw a graph in my mind, the overall trend is up into the right, there are absolutely uncertainly more nonacademic behavioral scientists, applied behavioral scientists today than there were a year ago in absolute relative numbers. Now, individual companies sometimes go up and down. Often having to do with the loss of a leader because there's only so few sort of more senior behavioral scientist. And so if you sort of lose your behavioral scientists, the team goes away and, or if you're able serendipitously to hire one, you might make a team where one didn't exist. I don't know this for a fact, but I'm going to conjecture that Maya Shankar, who's now at Google previously was at the White House.

Matt Wallaert:

I don't think Google was like, "Let's shop for a behavioral scientist." I think they were like, "Yes, this person is on the market and we could build something around this person." And I think we're still a little in that phenomenological. One of the big problems in larger corporations is the expectations are incredibly unrealistic for the level of experience and salary and things. I vividly remember a company, an insurance company, calling me and they weren't trying to recruit me. They knew I was too senior, they at least knew, okay, I'm not trying to recruit Matt Wallaert, I get that. But it was like a mid management position and let's call it like 150 in a suburb of Chicago. It's like an okay salary, okay position. It's not blowing anybody's socks off.

Matt Wallaert:

And it was meant to be in the data science org is going to be first behavioral science hire is going to be in the data science org and they're sort of shopping for people and I talked to them for a while and they said, "Do you know anybody?" And said they'll send me the JD. And they sent me the JD. And the first bullet point was 10 years of applied behavioral science experience. Okay, that's me and maybe five other people maybe and maybe not even that. Now you've got one circle that's five. Then the second bullet point was 10 years of insurance experience and I called them back and I said, you've created a Venn diagram with zero overlap. Precisely, zero behavioral scientists who have spent 10 years in the insurance industry.

Matt Wallaert:

I think probably even if you had said one year, it would have been a zero overlap. And so many companies I see that go on this quest start at a place with really unreasonable expectations and then we have to help reel them back in where we're like, okay, but it's probably not actually going to look like that. And I think it's up to the industry to prove its value, to train the right people, to demonstrate stuff. I don't think you can blame companies who have fiduciary responsibilities and things for not taking a flyer on behavioral science. I think it's up to be able to science to prove that it can be valuable.

Holly Hester-Reilly:

Yeah, that's fair. That makes sense. You mentioned earlier on about economics and the Nobel Prize for economics and I've actually found that in some of my pitches for certain work, I end up talking about Richard Taylor and nudge and things like that. And I find that it feels like there's a whole generation of business executives who haven't yet come to realize that the academic community has moved beyond assuming that everybody is just a perfectly rational actor for themselves. And sometimes I just feel like we're just trying to educate them on this is where knowledge is now. Do you come across that?

Matt Wallaert:

Yeah. Absolutely. I mean, it's easier that I'm old enough to feel the difference. Meaning once upon a time, that was always the case. I would always have to explain what behavioral science was to absolutely everyone. I now think enough people have read Danny Concannon's book or Dan Ariely's book or somebody's book, Adam Grant or somebody like, they've started to get more familiar. Ted has helped but you put Angela Duckworth on stage, she's incredibly compelling. I think you could do things to sort of start to, new distribution channels of knowledge I think has started to do the education for me. Where I actually find the more frequent barrier. And again, I tend to play in the larger companies face, not so much the startup space these days, is people will be like, "Yeah, I want that nudge stuff, but nobody has any idea how to do that."

Matt Wallaert:

There's a reason that my book is very much not nudge. It's very much like this is how you create a process and a team and a structure and an organization around this concept and around this approach. Because I think there have been very few of those. Even the teams that exist in the world, I don't know anybody that runs as strict of a 10 week process as I do. Right. Other people I think approach it more from a sort of flexible timeline, more like a research organization. It's not a factory, mine's a factory. And the reason is because I think it's easier for my executive to understand how the factory works, it's not research organization. It's understanding how the factory works and turning out the widgets. And I think there's a lot more of that.

Matt Wallaert:

I have a lot of friends at different tech companies who are earlier in their career or the first behavioral science hire or whatever and are talking to me about how it sort of plays out in their company. And one of the first things that I sort of always drilled them on is find a process, find a framework, find a methodology, I mean, I actually think it will make you a better behavioral scientist, but it will make it easier for other people to understand what it is you are building, how to staff it, how to resource it, what the expected outcomes are, what you can ask of it. I just think that standardization and is so key.

Holly Hester-Reilly:

Yeah. I'm a big fan of that myself. I always tell people I had studied chemical engineering and it's both product engineering and process engineering and I've never shied away from, no way, process can be good, we need some process. You mentioned your book, I think it would be great to hear a little bit more. Listeners who haven't heard about it, know more about what that's about. Tell us a little.

Matt Wallaert:

Just go read it folks. If you like this conversation, it's more of this conversation, I don't need to tell you about it. It is very much an applied book. It is written with lots of cursing and fun examples and is not academic and is really meant to be as close to a handbook of how does this play out in the real world as I can get. It's broken up into two halves, probably two sections. The first one's about two thirds of the book is sort of how do you actually go about building the intervention design process and sort of make that happen in your organization. And then the last third is more advanced topics. You can go into about individual behavioral science challenges, but it's a fun to read.

Matt Wallaert:

It's interesting to me, it's starting to be taught in a lot of college courses in really disparate kinds of programs. It's in an MBA program, there's a marketing course that's using it. There's an implementation science. I don't know if you know this part of the world. This is sort of a new thing, implementation science, which very much to me it looks like apply behavioral science to process science. There's an implementation science course that's using it and it's interesting to watch different people engage with it. I was joking on Twitter the other day that I inadvertently wrote a self help book because, one of the things in the book that we talked about is how to write a behavioral statement, how to lay out, I fundamentally believe everything we build, we build to change behavior.

Matt Wallaert:

Yet we almost never talk about it that way. We almost never lay out like whose behavior are we changing in what conditions, what is the behavior we want, how are we measuring that? And you write a behavioral statements do that. And someone read the book and is now has started tweeting at me or texting at me like behavioral statements. Literally what their behavioral statement for the day is. And I'm like, well that is not at all what I had in mind when I wrote the book. But it can be viewed that day. There are behaviors you're trying to create that day you're going to go try and create them.

Matt Wallaert:

And it's hopefully a very broad generalizable model, which has its weaknesses. If you have a PhD in behavioral science or in social psychology or something, you're going to read this and think like, wow, this is really reductive. But that's how you apply things, not everyone it turns out has a PhD. And so you need to sort of find a way that gets them to act in the way that you want, not necessarily having eight years of a PhD.

Holly Hester-Reilly:

Yeah, absolutely. And for listeners who don't check out the show notes, the book is called Start at the End. How [inaudible] products-

Matt Wallaert:

[inaudible] name of things, I have a terrible author, somewhere penguin random houses in the background listening to this podcast going, Matt Waller, what are you doing? You're not talking about the book at all. You don't say the name? Yes, available on Amazon and Barnes & Noble sellers near you.

Holly Hester-Reilly:

Oh God. People know how to find a book but they need the name or the author's name spelled for them.

Matt Wallaert:

Start at the End. Matt Waller, M-A-T-T W-A-L-L-E-R-T because I have an extra vowel for no apparent reason because the Belgians are weird. Google a book and it has an audio book. You could do the audio book instead. I often recommend the audio book because then the cursing is actually me cursing and-

Holly Hester-Reilly:

I was just going to ask, did you do the recording for the audio book?

Matt Wallaert:

I did. I got to apparently a nonfiction books. Normally the author gets to do the recording and I am taking advantage of this and so you get a three hours of me going, what the fuck is going on here?

Holly Hester-Reilly:

Oh that's so awesome. Anybody who really enjoyed this conversation and wants the full flavor should definitely download the audio book.

Matt Wallaert:

But first you should subscribe to this podcast. That is the first thing you should do. If you enjoyed this conversation, you should listen to more conversations like it with our wonderful host. Then may be as a secondary action may be, you should buy a book. Maybe buy a book there, that's more up to you, but you should subscribe to the podcast so that you can listen to more conversations like this one with talent leaders from across the [inaudible 00:49:38].

Holly Hester-Reilly:

Yes. Thank you for that. That is wonderful. The last two things I always like to ask people is, the first one is, if you could sum up what advice you'd give to someone who's, what advice do you give? You already worked with people who are much earlier in their journey as behavior scientists. What is the core of what you tell them?

Matt Wallaert:

Find friends. No. Look, I'm also as you know, a massive feminist and with anything that is a struggle, in the way that behavioral sciences struggle, feminism is a struggle, the sorts of things are struggles. One of the most important thing that you can do is find the emotional support that you need to get through that struggle because it will be hard on some days. I take great joy in my career. I love my career. I've had a great one and I really enjoy the work I do. I've given [inaudible] talks and other things about the importance of work. I do believe very strongly in the meaning of the work that I do. But there are some shitty days and find people who understand you, understand what you're trying to do, get you, support you, validate you. And if you can't find anybody, then come find me. But find somebody.

Holly Hester-Reilly:

Yeah. That's awesome. I love that. And then the last thing is super tactical. How do people find you if they want to follow you online or on Twitter, et cetera?

Matt Wallaert:

Well, I'm one of those people gifted with an incredibly unusual name, hence that extra vowel sometimes pays off. And it's mattwallaert.com or Matt Wallaert on Twitter or matt@mattwallaert.com it's a very easy, I try and be very open. I have these published links for people to sit down and have a meeting with me or have a phone call. I spend, I figured it out, something like 15 hours a week doing mentoring of one kind or another. I don't consult. I don't charge speaking fees. I believe very passionately that the world gets better when more people think about behavior and think about behavioral science. And I choose to spend my time focusing on the spread of that. And if there's something I can do to help you, please just reach out and I'll do what I can. And once again subscribe to this podcast

Holly Hester-Reilly:

And buy the book.

Matt Wallaert:

And may be buy the book. But [crosstalk] subscribe to the podcast.

Holly Hester-Reilly:

Yeah. Awesome. Well thank you so much for your time. This was so much fun.

Matt Wallaert:

It's always so much fun. And I will go for a walk soon since we live, here you go listeners, we live in the same neighborhood.

Holly Hester-Reilly:

Yeah. We definitely live pretty close. I was like, Oh yeah. I found myself wondering if we may have actually had our kids run into each other at my gym because I'm ...

Matt Wallaert:

Oh that would seem totally reasonable. Or playing in one of the many playgrounds in our neighborhood. We're recruiting. Please move to Harlem. It's a lovely place to have kids. Please join us in the kid friendly neighborhood, that is Harlem, you will enjoy it greatly.

Holly Hester-Reilly:

Yes. I call my part upstate Manhattan because I am up by the GW Bridge. I tell people, I live in upstate Manhattan.

Matt Wallaert:

It's amazing to me living in Harlem, how far people think it is, right there. They act like it's another country. I'm like, it's not any farther than going into Brooklyn. There are express trains, the two, three and the AD are a long way and not a very long, 59 to 125th.

Holly Hester-Reilly:

That's so fast. Yeah.

Matt Wallaert:

So fast. I don't know, these people are like, "Oh, you're so far away." I'm like, okay man.

Holly Hester-Reilly:

Yeah, I know. It's just mentally far for people. Just mentally.

Matt Wallaert:

I guess.

Holly Hester-Reilly:

Yeah. Awesome. Well, thank you so much. Have a wonderful day.

Matt Wallaert:

Thank you. You too.

Holly Hester-Reilly:

Product Science Podcast is brought to you by H2R Product Science. We teach startup founders and products leaders how do you use the product science method to discover the strongest product opportunities and lay the foundations for high-growth products, teams, and businesses. Learn more @h2rproductscience.com. Enjoying this episode, don't forget to subscribe so you don't miss next week's episode. I also encourage you to visit us @productsciencepodcast.com to sign up for more information and resources from me and our guests. If you love the show, a rating and review would be greatly appreciated. Thank you.

‍