The Andrew Michael Hypothesis: Mixing Qualitative and Quantitative Research Methods Leads to Better Product Decisions
In this episode of the Product Science Podcast, we cover the mistakes Andrew made early on in his startup career and at what points in the development process he focuses on quantitative vs qualitative research.
Holly Hester-Reilly
Andrew Michael is an entrepreneur with 14 years of experience in digital growth companies as a founder or senior manager with a focus on customer retention, data analytics, product, and growth. He is currently the founder and CEO at Avrio, a collaborative research platform.
In this episode of the Product Science Podcast, we cover the mistakes Andrew made early on in his startup career and at what points in the development process he focuses on quantitative vs qualitative research.
Subscribe for the full episode on Apple, Google Play, Spotify, Stitcher, YouTube and more. Love what you hear? Leave us a review, it means a lot.
Resource Links
Questions we explore in this episode
How did Andrew look at the decision of whether to find cofounders, build a team, or be a solopreneur?
- Andrew originally steered clear of cofounders because of stories he’d heard about difficult partnerships.
- Andrew tried at first to build with consultants and hire people here and there.
- Andrew realized through his experiences that having a team of people to push you and provide feedback is invaluable.
How does Andrew use qualitative and quantitative evidence to make better product decisions?
- Early on in a startup’s life, the data is a lot more noise than signal, so he uses more qualitative evidence like user interviews in that stage.
- He says you need to focus on pain points and talk to customers continuously.
- When you hit a certain size and scale, data becomes extremely useful, so you need a good data stack. At Andrew’s startup, they currently use Segment, Mixpanel, Amplitude, Redshift, Intercom, and HotJar.
- When the data tells you something interesting is happening, conduct user interviews to understand why users are behaving in that way.
How does Andrew build his team to be able to compete in the modern economy?
- He concentrates his hiring within selected time zones to make collaboration across the distributed team easier.
- He has founders and team members involved in the user interviews so that everyone can empathize with the user.
- They use weekly active users as their North Star Metric and they focus on the activities that lead to improving that.
Quotes from this episode
I had been told these bad stories around having partners and how that was a negative experience for certain individuals. And then that led me to think, okay, I don't need anybody else. I'm going to do this by myself. I'm going to just hire people where I can or work with consultants here and there. But ultimately, if you're not really surrounded by a really strong team that cares deeply about the product or the problem that you're solving, you never really get any good pushback from anybody. So you can end up going down the wrong route very easily, very fast.
So at an early stage, you're getting more noise than signal from data. The focus should be on just speaking to customers continuously and really trying to dig into the pain points and understanding. But then you do hit a certain size and scale when you absolutely need to have data.
The Product Science Podcast Season 5 is brought to you by Productboard
Check out their eBook: 10 Dysfunctions of Product Management
If your product team is struggling, it could be one of the 10 dysfunctions of product management. Learn more about each dysfunction and how a product management system can help you address and avoid them.
Transcription
Holly Hester-Reilly:
Hi, and welcome to The Product Science Podcast, where we're helping startup founders and product leaders build high-growth products, teams, and companies, through real conversations with people who have tried it and aren't afraid to share lessons learned from their failures along the way. I'm your host, Holly Hester-Reilly, Founder and CEO of H2R Product Science.This week on The Product Science podcast, I'm excited to share a conversation with Andrew Michael. Andrew is an entrepreneur with 14 years of experience in digital growth companies as a founder or senior manager, with a focus on customer retention, data analytics, product and growth. Welcome, Andrew.
Andrew Michael:
Thanks very much for having me, Holly. It's great to be here.
Holly Hester-Reilly:
Yeah, I'm excited to have you. You have a pretty interesting background. It sounds like you've been a part of several startups and you're leading one right now. I'd love to hear a little bit about your journey. What got you into startups in the first place?
Andrew Michael:
What got me into startups in the first place? It's a good one. I haven't actually thought about the specifics behind it, but I think for myself, I've always had that entrepreneurial itch. Growing up, just even in school, and I came from a family of entrepreneurs as well. I think it was ingrained in me. The first business I started was just selling sweets and ice creams at the school until the school shut it down because the school shop wasn't doing as well as it normally did.I think that was always in my nature was to start businesses. And then I actually, at school, was lucky. Fortunate to a school where we were taught to program at an early age and I got into that side of things, and then I actually moved. I grew up in South Africa, I moved to Cyprus and I did a whole bunch of different jobs. If I tell you, I tried everything and anything. From digging, driving a digger to dig holes for the electricity company, to pool boy to barman, literally everything.And then just decided one day wait, "I have this really good skill that I used to have. Why not pick that up again and see what you can do with and see make useful." And then just really fell in love with it again and started from there. Just trying out different things and tinkering to where we are today.
Holly Hester-Reilly:
Sounds like you're a doer through and through.
Andrew Michael:
Yes, I'd say that.
Holly Hester-Reilly:
Yeah. What was your first experience inside a tech startup?
Andrew Michael:
Yeah, so the first startup I built for myself was actually a company called Mega Mall Cyprus. It was a tech company, and this was sure quite a long time ago now, 10 to 12 years ago. And the initial premise was we saw everything was slowly starting to move online more and more. And I thought, okay, it was only natural that online shopping was going to become a bigger thing and people would want to be able to start selling products online. So Mega Mall Cyprus was an attempt to essentially what Shopify or others have become today, is to allow people to be able to create their own online store in a matter minutes to accept payments and start selling their products, which is what we did.But I think back then I was still very young, very naive, and I felt that Cyprus was a good enough market to focus on when in retrospect you cannot build any business of substance in a place that small. It's a good testing ground to figure things out, but you really need to be thinking globally from the start. That was the first deep dive into it and a lot of really valuable lessons learned along the way.
Holly Hester-Reilly:
What were some of those lessons?
Andrew Michael:
I think one of the early ones was really how critical the team is. I had been told these bad stories around having partners and how that was a negative experience for certain individuals. And I think these issues that were fed to me maybe don't really apply as much in the startup world if I see in retrospect now the differences in that. And then that led me to think, okay, I don't need anybody else. I'm going to do this by myself. I'm going to just hire people where I can or work with consultants here and there. But ultimately, if you're not really surrounded by a really strong team that cares deeply about the product or the problem that you're solving, you never really get any good pushback from anybody. So you can end up going down the wrong route very easily, very fast.You run out of steam I think at some point as well when you don't really have that good backing and real energy from the rest of the team as well. So there was a whole bunch of things like this that just led me to say there's first of all, there's no ways you can really build a good company without the right people around you. And the notion that there's issues with partnerships and things like that. I think that whatever you do, you can always find issues and things, but if you spend the time to get to know and understand the people you're going to be working with, those can lead to some of the best relationships and actually follow on startup. My two closest friends now were ex co-founders, one of them, Chris and my son and the other one is the best man at my wedding. And we're still really great friends today even though we don't work together anymore. So I think proved that fallacy false like there are hunt partnerships, and I think that came from more traditional business and I think that was a big lesson.
Holly Hester-Reilly:
That's interesting. So what was that next startup that you just mentioned?
Andrew Michael:
Yeah, so the next startup we started actually again, this was probably not a great way to start a startup, but I was working with the previous co-founder and we had a startup live weekend in Cyprus, was one of the first things. And we decided, okay, let's go and check it out. Let's see what it's all about and what's happening. And called up another friend of mine who had previously worked together. We said, let's go together. You need teams for this, so let's figure it out when we are there what we're going to do? And we had no idea what we really wanted to do over the weekend, but was actually just a game on the Friday night. It said, pick three words and come up with an idea. And we did that, we pitched it, everybody loved the idea and we decided, okay, let's try build this. And a month later we had all quit our jobs.We were on our way to Copenhagen because we got accepted to start up bootcamp, which is one of the good accelerator programs within Europe at the time. And that was really, it was called Funny farm and the idea behind it was to gamify chores for kids. So to make it a game out of it. In retrospect, none of us were parents, so we didn't really understand the problem clearly. We just fell into this emotion. I think once we had gone deep down the rabbit hole of literally quitting our jobs and me and my co-founder had probably some of the best jobs you could have at our age on the island at that point in time and we were heavily invested there. I think knowing and understanding the problem space and the problem itself is a big challenge when you can't really empathize and relate. I think you can do a lot of user research and understand the problems and pain points and work towards it. But I think you definitely have an edge when you build for yourself and you are the customer and we definitely weren't the customer at that stage.
Holly Hester-Reilly:
Interesting. I guess I would challenge the idea that you have an edge when you are one of the customers, but I know, I guess it depends a bit on what research you were able to do and how you explored the problem space or if you did enough of that in the beginning.
Andrew Michael:
Yeah.
Holly Hester-Reilly:
What do you think about that when you look back on it?
Andrew Michael:
So we definitely didn't do enough research in the beginning in terms of the problem space. And I think one of the biggest learnings was just that we had an issue with the product when it came to graduation churn in the sense that our product did a good job, but that job shelf life was very finite. So once your users had started to use the product, they started to see good results. There was no longer a need for the product because it had solved the problem that that came to you in the end. This was like we're trying to build a subscription business off the back of this. So I think that's something we probably could have picked up a lot earlier from research itself.But yeah, I think the edge, when I say the edge, I agree with you, you can do a lot of good research and that you can still build product, but I think just being able to empathize a little bit better with users and feeling that pain a little bit yourself, it's hard to beat that direct experience and that direct knowledge. It does come with its drawbacks of really bad cognitive biases and you're thinking you understand the problem better than you, so you can't trade one for the other. But I think just having both together is a big help.
Holly Hester-Reilly:
So what ended up happening with that startup?
Andrew Michael:
Yeah, so that company we ended up to about a 100,000 users. We did some really good partnerships with Lego and with Unilever. We ended up getting acquired by a Danish company they were working with at the time.
Holly Hester-Reilly:
Okay.
Andrew Michael:
Not much happened from that company, but out of there we actually launched a new company. We just happened to randomly be a hack on one weekend. We were trying to see how to drive traffic and actually nothing related to the product, but we ended up launching a new company which was called Ad Jelly, which is the essential notion was to make Facebook ads stupidly simple. So this was still six, seven years ago when Facebook was still a little bit complicated for most small businesses to be able to create ads on the platform. And we ended up as well building, which was called the ad size guide, which essentially back then as well, it was very difficult to understand the different ad sizes.Things were changing on a weekly basis to get the right templates and so forth. We built that in the space of a weekend, that first version of just the ad sizes and I think on the Monday we had over 10,000 visitors to the sites and started generating some serious traffic to it. And from that business as well, we ended up getting acquired by a company called Brand Zuka, which is the fossils growing ad network for video ads, for TV ads in the US. And at that time then I think we'd gone off a run of five, six years of full in startup mode, startup life, traveling around the world, sleeping on mattresses. At some point to going all in, I decided as well then David Darmanin do the CEO of Hotjar, if you're familiar with Hotjar analytics and feedback. He was an advisor at the time and he said, okay, I want to come to Hotjar, see how things are being done and when you're ready you can get started again. So I was lucky. I had a sauce masterclass there over three, three and a half years. Yeah.
Holly Hester-Reilly:
Okay. Yeah. So what did you do at Hotjar?
Andrew Michael:
Yeah, so Hotjar, I was very lucky in the sense that when I joined with the entrepreneurial background, I had a lot of freedom and flexibility as well to work on different aspects of the business. I joined in marketing, but pretty much I think whenever I saw a problem and I suggested a plan for the problem and how we go about doing it was almost always, yes, go ahead and let's get this done. Through the time I got to work on some really interesting projects. I worked a lot on the go-to-market strategy of the, in early days, did a lot of the pricing and packaging research for the company of which the end results ended up in the 30% increase in ASP from that initial research and studies along with a few other good product managers.
Holly Hester-Reilly:
Can you tell me more about that?
Andrew Michael:
Yeah.
Holly Hester-Reilly:
Before you move on?
Andrew Michael:
Yeah, of course. So in the early days, Hotjar essentially was charging a flat fee. We had a $29 plan, we had an $89 plan. And the idea was that as traffic scaled, the pricing should scale, which had never, in the beginning we never really enforced it, which was one aspect. But we did quite a lengthy study. We worked with an outside department as all price intelligently to conduct the research itself. And basically the research is broken up into different phases, which is first you dive deeply into really trying to understand who the buyer personas are. And we typically start with a panel study in these studies. And the reason for that is that the customers that you have today are a director of reflection of the product that you've built in the marketing you've done up until now, they may not necessarily be the best reflection and you cannot really get good pricing data and research from customers.How much are you willing to pay for a product to service? They already know what they're paying and that's what they're willing to pay in their mind. So you already said, so I think with the pricing and packaging research, it's done over a series of different panel studies really trying to understand, okay, what are users' willingness to pay, how likely are they to buy? And they also tend to be very useful then to understand for your packaging what features go into different packages and things like this.
Holly Hester-Reilly:
Tell me more about what you mean by a panel study.
Andrew Michael:
So a panel study, what you do is there's different providers. So myself as well, I did consulting for pricing and packaging for a few other companies and we use a company like Cint. So Cint, essentially what they allow you to do is find a specific profile that fits who your target buyer is. So you can set the demographics, what their role is, income company size, and that essentially allows you to reach that audience for the study. And if for this specific one you'd aim for five to 600 respondents for these surveys per study. And essentially the panel is just being able to meet and reach people that match your criteria.
Holly Hester-Reilly:
So what was the mix when you were doing that pricing and packaging research? What was the mix of one-on-one interviews versus surveys versus other forms of gathering data?
Andrew Michael:
So there was quite a bit difference. So the work we did with pricing today was just purely panel study, but ourselves internally as well. We looked at a few other aspects and the first aspects that we started actually with was some retention analysis. Just trying to look at data and see, okay, what makes our most successful customers? At the time, what we did was we took a cohort of users who had been paying us for longer than 12 months that were spending over a certain amount and that also showed some good levels of engagement within the product. So it wasn't just about paying us but also making sure that they're actively using it. And we took that cohort, then we enriched the data. So there's services like Clearbits that allow you to run through domains and run through email addresses and they give you back firmographic and demographic properties about the end user.And then from that what we were able to do was take those properties and then map them back to a retention itself to see, okay, was there any specific firmographic or demographic properties about these users that stood out to be more likely to be healthy retained customer versus not? So that was the first aspect. So that gave us a good indication roughly of who and what size of company or the age of the company or the roles that this person had was most likely to have would be good customer fits. And then we also then took another viewpoint of it and thought, okay, recently acquired customers. So we also wanted to say, okay, what is the demographic and firmographic makeup of those that are resonating with the messaging that we're putting out in the world today? Because obviously the longest retained customers, there was some bias in there as well from the early days and bias as well.So we wanted to understand that. And again, similar process with Clearbit, just looking at those properties and then identifying good forms. And then the other one as well is trying to understand the conversion rates of the various properties. So we took a cohort of users that signed up six months previously. Hotjar was a freemium product at the time, still is. And we wanted to see over time how many of the different of that cohort converted and what was their demographic makeup. Because the same thing applies as well is that we may have been acquiring a lot of a specific segment but not converting them as well, or we may have not been acquiring that many of a specific segment, but the conversion rates were insanely high. So we wanted to understand, okay, maybe marketing could do a better job than of acquiring that specific segment and acquiring less of the other.So that was the third component from a data perspective. And this was purely from an analytics perspective, just going through quantitatively looking at the different properties and then mapping those out on various different graphs. So we could see, and we actually ended up coming up with a good scoring model as well out of this in the early days, which allowed us to then see, okay, a really good correlation between the higher the score was the more likely they were to retain and the higher the net MRR retention net MRR revenue, everything trended nicely with that based on the scoring model that we produced and was a really simple model in its own right.
Holly Hester-Reilly:
What were some of the actual concrete things that you found in terms of what was maybe a customer profile that fit one of those stories that you just told? Either you didn't have a lot but they converted really well or vice versa?
Andrew Michael:
So one of the first, I think lessons in this, the scoring model and what is an ideal customer profile I think is not have this idea, it's a specific thing. So it's not saying, okay, it's a company of a 100 to 200 people that have this many users in this industry and this that's our ideal customer level. I think that's a very narrow view on the world and you end up missing out on a lot of opportunity because if a company has two or three properties, that could still be a really good fit company if they don't meet all the criteria. So there's just a degrees to how much of a good fit they are. So that was one of the first learnings from that perspective. The second thing we learnt as well, and it's something actually that has been phased out now, but Alexa rank was one of the biggest indicators of indication of whether company would be successful using the product or not and if they'd be retained longest.
Holly Hester-Reilly:
Oh, interesting.
Andrew Michael:
Yeah, and the reason for that is Alexa rank is a good indicator of traffic. So it seems quite logical that the sites with the highest traffic would have the most interest in using a service to improve their experience. But I've also seen this translate in other companies that I've since helped with pricing and packaging where they're helping a Shopify store and Shopify also give a rank, I believe it is to the stores themselves. So just like Alexa rank, you get a Shopify rank at Shopify score, which is how much traffic or how busy the store is itself. And that also ended up being one of the biggest indicators of a good fit for the business and for the customer. And yeah, there was a lot of areas where we looked into company size made a difference, company age made a difference, the role of the end user made a difference.So we saw product managers, if they were the ones who set up the accounts initially were a lot more likely to be sticky and roll it out. So there was a lot of these individual properties, but you can't say it was a group of five, six properties, there was a clear outright winner was more just, okay, when you look at just the different roles, we could see okay, there was two, three outliers compared to all the other roles. When you look at the company sizes, there's two or three compared to all the others. And then from that it's like how you develop the scoring model and say, okay, if they're one of these two or three, it's a three points. If one of the next best two or three, it's a 2.9. And that's how we just developed a very rudimentary system, but it ended up working really, really well.
Holly Hester-Reilly:
And what did you do with the information when a potential client had a good score? How did that change their experience or how you spoke to them?
Andrew Michael:
Yeah, so it didn't change the experience that much or how we spoke to them, but what we were starting to do was look at ways from a customer success, from a support standpoint was and from a sales standpoint. So sales was probably the first to implement it and start to work with the score. And really what it allowed them to do was to prioritize the leads because I think I tried really high volume of inbound leads and there was just never going to be enough time for sales team to be able to go through everything. So the first step was just really how could we enable them to prioritize a lot better? We rolled that out with the sales team and then started using their feedback from them to iterate on the model itself and trying to understand, okay, are these good fits? Are there certain properties from there?So sales was the first and then the next one, which is a little bit, I think time of leaving and stuff was going to be around supports. But ultimately I think Hachi also had a really great philosophy around support and that support should be equal and fair and great for everybody. So I think that was less likely to be adopted. And then the next one was in marketing as well. So you could slowly start to see, okay, different campaigns that were bringing in, what would the score from the different campaigns, what was the lead quality that marketing was bringing through the door? What improvements could be made? Are there specific campaigns or channels that tend to bring in higher scoring elites?
Holly Hester-Reilly:
Yeah. And how did this work that you were doing contribute towards the startup that you run today? Because it sounds like there's some connection.
Andrew Michael:
Yes, there is a connection in some way. So the startup I'm doing today started actually out at Hotjar. And as I mentioned, I've been working on a lot of different research for the team. So at some point as well focused on the ICP on pricing and packaging churn research, and I set up the business intelligence team at Hotjar. So we're analytics and feedback company, which is a little bit ironic. We started setting up really good data stack quite late into the setup, although we did have a good setup in the beginning, the team really didn't rely much on it because it ended up getting polluted. It was messy. So Hotjar actually in the early days was heavily driven by qualitative feedback and founder intuition and things like that. And I think it makes a lot of sense and in retrospect is something that I think actually is the only way in the early stage because you don't have the luxury of the data to give you good signals at an early stage.So an early stage, you're getting more noise than signal from data. The focus should be on just speaking to customers continuously and really trying to dig into the pain points and understanding. But then you do hit a certain size and scale when you absolutely need to have data. So that happened, we set up the team and actually it was one quarter, the company was using OKRs and we had started planning. And even though the whole company had the same objectives, everybody was doing a little bit of their own research and going off into their own different directions. And I noticed that one of the product teams did an amazing job where they brought all the research together, they brought some customer feedback together, they had some data and analytics and they created a space in Confluence and they were like, okay, this is how we're going to go and act these objectives and these are the solutions and so forth.But that information pretty much died there and that confluence page and nobody in the rest of the company saw it, even though the insights that they had would've been valuable for customer success, for sales, for marketing. And I just said, okay, we need to fix this. And part of it as well was that in business intelligence, we had done some really great studies that research had been repeated by the product team that they managed to get it on their own and bring it into certain places, but it was clearly a waste of time and resources as well that had happened there. I said, there must be a better way. And just started interviewing different companies and just seeing what was out there, who was doing something and realized, okay, this is a bigger problem than just Hotjars and decided to give it a go.
Holly Hester-Reilly:
So what were some of the things you found when you interviewed other companies? What were their sharing knowledge challenges?
Andrew Michael:
So it was all pretty much the same. I heard it like the continuous pattern that it got frustrating at some point. So everybody decided, okay, we needed us to centralize it. They all decided let's use Notion or Confluence or Airtable. It all ended up breaking because people had to break out of the natural workflow to go and save things. Most of the time, only person that saved something, they knew it existed. It was very difficult to keep fresh and up to date in these places because they're not purpose-built. So all the same problems ended up occurring, but all the same solutions were being repeated to the problem. That was when I said, okay, there is a better way we can rethink how we approach the problem. And I interviewed a lot of different companies and it was always pretty much the same thing over and over again.
Holly Hester-Reilly:
And what were the roles inside the companies that you were talking to?
Andrew Michael:
Marketing, product design, user research. I spoke to customer success, I spoke to everyone within the organization, but predominantly focused in product teams was where I saw the biggest value for the product or service.
Holly Hester-Reilly:
How long ago was it when you did this research?
Andrew Michael:
Two years ago now.
Holly Hester-Reilly:
Cool. And what's happened since then?
Andrew Michael:
What's happened since then? So we actually ended up launching the first version of the product in September last year I think about there. And we had really good initial interest and traction. So we had around five to 700 signups in the first month. We had companies like Adidas and Envision set up accounts with us and then activation and usage was absolute. Nobody used the product.
Holly Hester-Reilly:
It's hard, right?
Andrew Michael:
Yeah. We really tried to scratch their heads trying to understand what went wrong. Obviously we did a lot of research and I think in retrospect there was a couple of things, there's one we tried in the beginning just to be more of an aggregator where you could bring from different sources. But one of the biggest learnings was that for knowledge sharing and for centralize it, creating a repository to share research is that I have a quote from one customer that's stuck in the back of my head is that whoever has my attention for analysis is going to have my attention for the distribution. And people had an expectation to be able to store and synthesize user research that use interviews from a repository. So that was a big key feature. And the second thing I think we made a mistake on was our first version of the product.We tried to change user behavior too much from what they were used to. And I think changing behavior is incredibly difficult and it's something that needs to be done over time through education and activation. And we didn't do that justice I think to begin with of how we went about it because our original product revolved a lot around the browser extension, which it still does today. But that wasn't a familiar way of collecting and storing and sharing knowledge within an organization. It required I think a lot more of an onboarding handled experience than we gave it in the beginning. So yeah, we hit the drawing board again, we reworked the product, we relaunched in May. And then since then we've been seeing really great retention engagement. This month we are going to have, last month we had close to a thousand signups for the product. We launched our paid plans, we're converting it to do 4% from free to paid depending on the cohort. Now the engine's starting to work and we are starting to focus on some of the more interesting and fun challenges now of growing the business.
Holly Hester-Reilly:
That's exciting.
Andrew Michael:
It is.
Holly Hester-Reilly:
Yeah. How did you come to understand this challenge with using the browser extension?
Andrew Michael:
How we understood was that nobody used it.
Holly Hester-Reilly:
But how did you figure out that was what was the problem?
Andrew Michael:
Just use interviews. So we had a lot of signups segment doing 500 or 700 signups in that month and literally just try to speak to everyone that would be willing to speak to us to try to understand going through it. The only way I think, like I said in the early stage is really speaking to customers and trying to gather that from them. And we were lucky that we were able to speak to a lot and get to that point.
Holly Hester-Reilly:
Yeah. So did you just email people and invite them to set up a time or?
Andrew Michael:
Yeah. Email. We had to incentivize. I think that's unfortunate, but it's a necessary evil I think when you're trying to recruit as well, if you want to get to any certain level of volume. So we incentivized, I think at the time we did $25 gift voucher, Amazon gift voucher was the thing. And yeah.
Holly Hester-Reilly:
Were you doing that research yourself as the founder or did you have any UX research experts or what did that look like?
Andrew Michael:
Doing it myself as a founder, I think it's really important, but ultimately what we did try to do is get the whole team actually doing it at some point. So everybody was interviewing different perspectives. Because I think also founders generally also have their own internal biases and wanting to make sure that it wasn't getting lost in your own internal biases. So having others on the team is really critical to be able to bring different perspective to it and different understanding.
Holly Hester-Reilly:
Yeah, I think having several people involved in research really adds to the value of that research because you get the different perspectives and two people can sit in the same conversation and come away with different takeaways from it.
Andrew Michael:
Exactly.
Holly Hester-Reilly:
Was there anything that you found during that stage that was surprising to you where you came away from a conversation and said, wow, I really needed to hear that. I didn't know?
Andrew Michael:
I think the one aspect that was surprising was the quote that I mentioned. And like I said, because that's literally, it's been burnt in the back of my head since then. And because I think there was, after all the initial research and interviews that I had been doing, I'd never really heard something like this before because I was really focused on the problem of distribution and how people are getting access to it and seeing it and less around what was the content that was going to be stored inside it and what would be the use cases for the content, which in retrospect was a very big mistake. But yeah, so I think that was definitely one of those moments. And then just being able to dig into that a lot more and realizing, okay, this is probably the biggest issue that we have today with the product is we need to solve for this.
Holly Hester-Reilly:
And if I can get meta for a moment, did you use your own tool to share the learnings from your own research?
Andrew Michael:
Yes, of course.
Holly Hester-Reilly:
And what was that like?
Andrew Michael:
That's obviously always a really good learning experience because as you're going through it, you're realizing a lot of pain points and what needs to be fixed and you're like, come on, this can't be right. So I think that's definitely a big aid and a big help is always if you can, using your own product or service, use it as much as possible. So for us, we also in the early days, just really tried to think what else could we put in that maybe wasn't a good use case for our product, but would get more of the team using it so they could also feel the pain points going through different aspects. So definitely helped bring polish to the product and to solve some of the more irritating little aspects of it as you're going through it.
Holly Hester-Reilly:
Yeah, that's awesome. It's really great when you're building a tool that is useful for your own team so that you can dog food it.
Andrew Michael:
Yeah, absolutely.
Holly Hester-Reilly:
So what does the team look like today?
Andrew Michael:
So we're six people today, engineering data science predominantly. And we're going to start to look to hire over the next couple of months to grow the marketing team. We're going to start a marketing team, so we'll hire our first person in marketing and we also continue to grow our engineering team as well, hiring a product designer at the same time.
Holly Hester-Reilly:
Awesome. And are you based in Cyprus? Are you remote? What does that look like?
Andrew Michael:
Fully remote business. So company is based out of Cyprus, but following in Hotjar's footsteps, I think we definitely want to be building a fully remote business. Hiring at the moment, mostly in Europe and Middle East time zones. Just something like learnt I think at Hotjar, learnt the hard way is that even though you want to work and create a distributed workforce. I think if you don't have a good number and a good concentration and a specific demographic, it's very difficult I think to work across different time zones.
Holly Hester-Reilly:
Yeah, I hear that. Definitely. I've worked with some clients that had a really broad distribution of time zones and man, did it take a toll on the workers to try to coordinate with each other across that?
Andrew Michael:
Yeah, and I think, at Hotjar obviously it got to certain size and scale where it was inevitable and necessary, but then it was really what specific team, so it was like a sales team needed to be in the US because that's where they were servicing the customers. But that's pretty independent on its own. You can have a sales team operating in a specific region that doesn't need to necessarily always be communicating with the rest of the team. But a product team where you're working on a day-to-day basis with other engineers, I think that's a lot more challenging and that's something where you need to be in a similar time zones to be able to operate effectively.
Holly Hester-Reilly:
Yeah. And are you at the scale yet where you're able to apply some of this retention analysis work that you've done previously to your own company?
Andrew Michael:
Yeah, we're definitely getting there now. And I think that's also a good thing as well. We set up a really good data stack from the start.
Holly Hester-Reilly:
What does it look like?
Andrew Michael:
I'm a big fan of segments. I know a lot of data engineers, but the thing I love about segments is essentially it allows you to have less data engineers to start. I think that's probably why they don't like it as much and allowed us to then get thing different tools and services up and running faster and also make sure switching costs almost zero. So if we want to change a different product to service, we can replay data into it. So we use segments for all our data collection on top of segment. We're using Mix Panel for data analytics. We set up Amplitude as well. But I find personally that it's a little bit more user friendly for the majority of the team, whereas maybe Amplitude is should get a little bit more geeky, but it's not as intuitive for everybody in the team. And what we really want is everyone on our team to be able to understand read data and use it.And we are using then as well Redshift for our data warehouse. We haven't laid on top in EBI tool yet. It hasn't been necessary, but most likely we'll either look into Tableau or Looker, probably lean towards Looker. This time round we had Tableau at Hotjar, but I think at the time I was a little bit more of a fan of Looker and what they were doing there. So that's pretty much a stack. And then we have other peripheral tools. So we use Hotjar obviously for certain analysis that we are doing for surveys as well and Intercom for other aspects. And we obviously all of these tools share data between each other. We can do some retention analysis and look at different activity that's happening across the different products and services as well.
Holly Hester-Reilly:
Yeah. So have you started to find any ideal customer profile details for Avrio?
Andrew Michael:
Yeah, so I think we started in the beginning with this ideal customer profile in mind just because we had done a lot of research in the beginning and we understood who had this pain point, who had this problem the most, what we're starting to notice is there's different segments that we didn't expect to be using our product or service. So that's been interesting. And what we are doing now with that is just really okay, the data is showing us one thing, we don't understand it. So we know what's happening, we don't know why. So now we're diving into user interviews and really just trying to pick some learnings out from there. But yeah, I mean our still ideal custom profile is fairly close to what we envisioned from the start, and it's what we researched at the start and is what we still see today. So those companies that are mid-market product teams, they tend to be really good active users retained. So in most cases we definitely see weekly usage and then a lot of them daily usage of the product.So that's where our focus still lies. But like I said, we are seeing interesting different segments that we didn't expect. So we see journalists using the product to research articles or students using our service to research for their master's thesis or putting together a dissertation. So these are segments.
Holly Hester-Reilly:
Oh, those are some interesting use cases.
Andrew Michael:
Yeah, interesting. But it's not something that we had envisioned or built a product for or have a good understanding of ourselves. So that's why we're busy leaning into that. Just trying to understand a little bit, is there maybe an opportunity?
Holly Hester-Reilly:
Yeah. One question that came to mind while I was hearing you say some of that is, do you have a north star metric that you've picked for how you're guiding your product team decisions?
Andrew Michael:
Yeah, so for us, we focused mostly on weekly active viewers as a metric that we've picked. We decided that if you're doing research and it's being shared effectively, the end goal is that people actually see the research. Essentially what we track is anytime somebody views either a highlight or a report that's generated within Avrio, we count them as active within that period and they obviously need to as well have been signed up more than a week ago. So we want to remove any bias from acquisition, playing a role and so forth. So weekly active viewers is our North star metric and then we cascade back from there and we understand whether there's certain activities and that need to take place in order to get there from activation perspective. And these are things we're working on now at the moment is really improving onboarding and activation for the different areas of the product.
Holly Hester-Reilly:
Yeah, cool. It sounds like things are all following a good process and growing well and you're in an exciting stage of this startup life.
Andrew Michael:
Yeah, I think it's an exciting stage. I gave this analogy as well to a friend earlier. It feels like at this point is that we've been struggling for the last year and a bits, we've just now managed to get to the top of the wall and we can see over the other side and it's balancing on the top now either they're going to go forward and things are going to go extremely well or you got to get prepared for that next downhill that's going to come inevitably in every startup. The ups and downs are always there, but at the moment it feels like we really on a good track and the things are moving in the right direction.
Holly Hester-Reilly:
And what has the role of pricing and packaging been for the stage that you're at?
Andrew Michael:
Yeah, so we did a bit of initial early research as well on pricing and packaging just to understand the market and the way things were at. I think this is something for us that we haven't quite figured out yet and we need to do a better of job of, I think one of the things looking at the business currently today is around six to eight months ago before should hit the fan everywhere I said to our investors, I was like, that doesn't look like things are going great in the world, even though evaluations are the all time high and everything is great. So I think as a business maybe we need to start focusing on monetization a little bit earlier and trying to generate revenue and instead of focusing on engagement and retention and the growing virality. And I think this is maybe something that was good in the sense that, okay, now in a slightly better position when we try to go fundraise that we actually have got revenue can start to show growth.But at the same time, I've definitely seen it's impacted the engagement and retention of the product from that perspective. And this is all to do with pricing and packaging the way that we restrict users at certain levels. So I think for us, we are definitely going to be iterating, experimenting a lot with the pricing and packaging, which is one is like how do we enable a freemium product that's powerful enough that enables anybody to continue using the service without any roadblocks. And then focusing more on what are the certain features that we can drive for upsell and what are those features that are more predominant from our ideal customer profile that we see as a good fit.And then perhaps the students themselves, we see them as more as a long-term growth play where they might use us for the dissertation and then they end up joining a company and because they love our software and they're familiar with it, they'll be the champions then to bring it in. This is something that we are going to be looking into over the next few months just to see, okay, how can we experiment with this to get a good medium where we get some good virality out of the freemium version product, but then we still have good checks in place for conversion for those that should be paying us and those that are within ideal customer profile.
Holly Hester-Reilly:
Yeah, that's interesting. I like the long-term play perspective on the students.
Andrew Michael:
It's nice to have it if you have the luxury, but I think that's also in the early days you need results to start generating those proof points and generating revenue. But definitely I think there's a lot of good examples of companies that have taken this approach with students. And then Qualtrics I think spoken about this too as their early customer adoption and base was predominantly students and that's actually what made them into the business they are today because ultimately ended up taking them into the companies where they went to get jobs later.
Holly Hester-Reilly:
Yeah, that's interesting. I think we're almost out of time, so I want to ask where can people find you if they want to follow you?
Andrew Michael:
Yeah, so for myself, on LinkedIn, always active there, Twitter as well. If you want to check out Avrio, it's avrio.com. We obviously, we have a freemium plan and you can get started and check things out for yourself there. But obviously always love to chat about these topics. So if anybody wants to follow up, always happy to just jump on a call or have a chat over a LinkedIn or Twitter.
Holly Hester-Reilly:
Wonderful, awesome. Thank you so much, Andrew. It's been a pleasure talking to you today.
Andrew Michael:
Thank you Holly. It's been great.
Holly Hester-Reilly:
The Product Science podcast is brought to you by H two R Product Science. We teach startup founders and product leaders how to use the product science method to discover the strongest product opportunities and lay the foundations for high growth products, teams and businesses. Learn more at h2rproductscience.com. Enjoying this episode, don't forget to subscribe so you don't miss next week's episode. I also encourage you to visit us @productsciencepodcast.com to sign up for more information and resources from me and our guests. If you like the show, a rating and review would be greatly appreciated. Thank you.
