The Jason Keramidas Hypothesis: With Enterprise Products, the Sales Team is Your Biggest Asset

Jason Keramidas is always thinking about how to build products that will make people’s lives better. He has over 25 years experience working with a range of companies from early stage startups to driving growth at public companies. He lives just north of NYC with his wife and two children.

In this episode of the Product Science Podcast, we cover Jason’s early days in product management, what is different about product in B2B, why Jason argues that the sales team is your greatest asset, and what his experience at a nonprofit was like.

Subscribe for the full episode on Apple, Google Play, Spotify, Stitcher, YouTube and more. Love what you hear? Leave us a review, it means a lot.

Resource Links

Follow Jason on Twitter

Follow Jason on LinkedIn

Follow Holly on Twitter

Follow Holly on LinkedIn

Questions we explore in this episode

How has Jason found B2B to be different when it comes to product management and experimentation?

• Jason has found that in particularly risk-averse industries such as mortgage servicing, it’s hard to run lean experiments. The product management team has to work with lots of review and compliance steps.
• Since you can’t do as much A/B testing or live experiments, you have to spend a lot of time with clients and with the call center data.
• Any changes have to be really coordinated with the customer, so the product manager works closely with customer success to prepare the customer for any changes.
• Find the customers who are more enthusiastic about embracing innovation and use them to test new ideas before you roll changes out to the entire customer base.

How and why does Jason build a strong relationship with the sales team?

• Jason argues that if you are building enterprise products, the sales team is your greatest asset.
• Recognize that the sales team has more qualitative data than you will ever have access to.
• For example, the sales team at Shutterstock was able to say the workflow doesn’t make sense, or that discovering new content was difficult.
• Working with the sales team, Jason identifies target customers for early testing or the development of new ideas.
• Understand the sales team is on the hook to hit revenue targets, so if you can explain how what you are doing will help them that goes a long way.

What were Jason’s experiences working in product management at a nonprofit like?

• Jason worked at Charity:water, which is a nonprofit that embraces technology in a way that’s atypical for nonprofits.
• Jason introduced pirate metrics (AARRR metrics) to the nonprofit so that they could start measuring more of their acquisition and referrals.
• They began running experiments to see how they could drive recurring giving.

Quotes from this episode

Working with enterprise customers, they didn't like AB testing or the results would get heavily skewed. So, you have to make a bet and sit here and say we have to have a high confidence level that this change is going to make a positive difference and roll it out and see what happens.

I would say, to anyone who's working with enterprise sales team, they are your greatest asset because they are on the front lines and it became very apparent very quickly talking to the sales team that, oh, these are the issues that people have.

The thing with sales is they're very outcomes driven, they're typically responsible for driving some revenue number. And if you can sit here and explain why what you are doing is either going to help them hit those revenue goals or, at the very least, not hamper them from hitting those revenue goals, that's the thing.

The Product Science Podcast Season 5 is brought to you by Productboard

Check out their eBook: 10 Dysfunctions of Product Management

If your product team is struggling, it could be one of the 10 dysfunctions of product management. Learn more about each dysfunction and how a product management system can help you address and avoid them.

Get Your Copy

Transcription

Holly Hester-Reilly:Hi and welcome to the Product Science Podcast where we're helping startup founders and product leaders build high growth products, teams and companies through real conversations with people who have tried it and aren't afraid to share lessons learned from their failures along the way. I'm your host, Holly Hester-Reilly, founder and CEO of H2R Product Science.This week on the Product Science Podcast, I'm excited to share a conversation with Jason Keramidas. Jason is always thinking about how to build products that will make people's lives better. He has over 25 years of experience working with a range of companies from early stage startups to drive and growth at public companies. He lives just north of NYC with his wife and two children. Welcome, Jason.

Jason Keramidas:Great to be here. Thanks for having me.

Holly Hester-Reilly:I'm super excited to have this conversation and also just to reconnect since it's been several years since we connected.

Jason Keramidas:Yeah, it is. It's been a while but it's so good to chat.

Holly Hester-Reilly:Yeah. So, as you know, I like to start with a bit of a background about how people got into products. So, what's your story?

Jason Keramidas:I have a very typical story and that I was a poli sci major who got a temp job at an internet startup cleaning Yellow Pages data in 1996. So, that's how I got into product management.

Holly Hester-Reilly:Super typical.

Jason Keramidas:Super typical, super typical. I graduated school, moved up to San Francisco where a bunch of my friends had moved after school and I was literally crashing on a friend's couch and I got a temp job at a company called Big Book which was the first nationwide online Yellow Pages. And what they needed to do is have someone sit and go through each Yellow Pages listing and delete any non-factual information. So, if you had Tony's Pizza, best pizza in New York, that is not a fact, that is marketing language.

Holly Hester-Reilly:Wow.

Jason Keramidas:So, I would sit there with my headphones on all day cleaning Yellow Pages data so that it could get then published online. So, that was my intro to the world but that quickly evolved. It was a company of 12 people, they saw, after a while, it was like, "Oh, this guy is relatively smart, we can have him do more than that." And so, I carved out this space for myself which actually ended up serving me in subsequent jobs where, little do people know, maybe they do, the regional Yellow Pages and, back in 1996, all this stuff was getting keyed in overseas and put in these databases. The issue is that each regional yellow page had different names for the same kind of business. So, it might be hair salon in one Yellow Pages, it might be hairdressers in another Yellow Pages. So, if you're trying to do a search for a national product, you need to normalize all that terminology. And so, I was tasked with creating a proprietary category hierarchy for this company off of these different category terms.So, I did all of that and I became the product manager for the categories at this online Yellow Pages and that took a variety of different shapes. Helping building out the listings products, how did we do this once we started selling small business products, websites basically. I was making sure that someone has assigned the right category, how does that come up and search and I was mapping everything to these standard industry classifications from the office of management and budget. So, I would sit there and say, "Okay, these are all the appropriate terms that map back to this."

Holly Hester-Reilly:Wow.

Jason Keramidas:But we were the first ones, we had driving directions and reviews and this was a big deal back then before everything got commoditized. And so, I carved out this expertise around business categories and so I left that startup, went to another startup that did white paper Yellow Pages products for newspapers that were looking to get into more local advertising around 1997, '98. They thought, "Oh, there's an opportunity for us to compete. We have a sales force that has relationships with local businesses, we can build a directory product into our new websites that we're building." And then I went from there to a startup which was reverse auctions. You would come in and say, "I'm looking for X, Y, Z," and we sold categories to businesses that sold those particular products.So, I spent all this time and built up this skillset, I guess, around Yellow Pages categories and what it did is it really exposed me. It was like, "Oh, this technology's really cool, there's all kinds of interesting applications," that's where I really decided I want to do this. I want to solve problems or use technology to make cool, different things and that was how it all kicked off.

Holly Hester-Reilly:That's awesome. I am still motivated by that desire to solve problems and make cool things with technology.

Jason Keramidas:It was great and it was a different world back then. So, you didn't necessarily need to have a technical background which helped because I didn't have that. And of course, everything back then was Waterfall so I grew up in the product world writing these gigantic requirements documents and everything. And so, as things evolved over time, I was very happy to embrace more agile methodologies so it wasn't giant, super long development periods with these large release windows where you have to do all this testing, all the requirements review and breaking things up into much more manageable chunks was a good shift.

Holly Hester-Reilly:Yeah, definitely. And if we could zoom out for a second, since you've worked in both, you've worked in Waterfall and certainly more modern methodologies, which one has had the worst situations of things being laid and scope creep and things like that?

Jason Keramidas:Scope creep, definitely more of an issue in Waterfall because so much of your timing was based on assumptions which were never right. I feel with Agile, at least, we have the ability to adjust expectations, manage scope, have more of a real-time assessment of what it is that we're working on and, if necessary, say, "Okay, well, we'll kick that out to a future release that's not going to be six months away so we can do it." But I would say there's not a best option, I would argue. I think there's times, especially having worked in some fintechs more recently, when there's compliance issues, you really need to think things through. When you're working with customers like banks, they don't want lots of releases, they want things much more predictable and so you have to do a little more management, may have internal Agile cycles but then you've got a more structured release cycle for the client.So, it's funny, I see, certainly more recently, working with customers where they want things a little more buttoned up before they go out because of the risk that it presents to them with something that hasn't been fully tested and vetted and signed off on and they don't have the resources to do that on a frequent basis.

Holly Hester-Reilly:Yeah. So, what are some of the places you've been at recently where you've seen that?

Jason Keramidas:So, a company called Brace, it's a mortgage servicing platform. I didn't know anything about mortgage servicing. I have a mortgage and I don't know anything about mortgage servicing but mortgage servicing represents 95% of the life cycle of a mortgage. So, all of the innovation in the mortgage space is typically focused on the origination side, how do we make the purchasing process easier. But the fact of the matter is, once you get past that, it's all paper in Excel and, like an old system written in the '60s, maybe it's been updated a little bit but the technology stack for mortgage servicing is incredibly old. And where does that rear its head? It's when people are in issues, which is called loss mitigation, when you have difficulty paying your mortgage, you need help, how do you do that?Well, the process is very paper driven. You call up, you say you need help, they send you a packet, you fill out all the paperwork, you mail it in, it's wrong. 10% of the time, the correct information actually comes in on the first try and, the mortgage servicers, there's so many regulations and guidelines that they have to follow around what gets offered to whom, how much time they have to evaluate an option. And if they don't follow those guidelines to the T, the fines that come out, in the hundreds of millions of dollars and there's just been a series of them. So, you can imagine that they are risk averse because they have a system that, in their minds, works well enough and so how do they introduce something new.So, Brace has built a product that enables for digital intake of the application, uses Plaid to connect to borrower assets and income statements to get the most accurate and up-to-date information, speeds up the process, more accurate applications coming in then there's workflow management and underwriting, et cetera, et cetera. But that's not the kind of thing where you're just like, "Oh, we're making a change." No, no, no, no, no, no, no. They're sitting there, when you have a release plan, they want to know what is the window when it's going out, we have to review everything, our compliance people need to know every change that's been made to make sure that what you're doing is in compliance, if it's something new, they have to understand what's going on. So, there's a lot of documentation and review that product management team works with our customer success team to go out and make sure that the client has full visibility into what's happening so that they are comfortable with what's getting pushed out.And so, we may live in this world where it's like, "Oh, we're pushing updates, live, continuous release cycles," or, even if it's sprint by sprint, they don't want that, that's too much for them to handle. They've maybe got an IT department that is focused on this and they're swamped with everything else. And so, you have to sit there and really align with the cadence, with the customer there to figure out what's comfortable for them and be super clear about the communications around what's coming out because you can't surprise them with anything. And if something goes wrong, site goes down or whatever, the amount of uptime you have to promise in your SLA, none of that stuff can go down because their call center is the one who's going to feel the burden if something's wrong with the application, not to mention the potential fines, et cetera, et cetera.So, it's a different world depending on the type of customer you have. It's not like a standard B2C that's like, "Oh, we'll just roll a bunch of stuff out and we'll do a bunch of experimentation." It's like, "No, no, no, no, no, no, we don't do that. Things are a little bit different." It's differently interesting, I think, because it forces you to change your mindset a little bit especially if you've gotten into these habits of, I don't want to say wild west, but just throwing stuff out there and see what sticks because that's just not an option for you.

Holly Hester-Reilly:So, how do you experiment or test or learn what will work?

Jason Keramidas:So, a lot of it is talking with the customer themselves. And so, this is an area where you really lean on building a good relationship with the customer and having a customer success team that has to be highly engaged. And so, like I said, they've got a call center so they're getting their own feedback all the time. There are different levels of comfort, really depends on the client on you sitting in and participating in their process because they are dealing with so much personal information when they're on the phone with someone that you can't just sit there and screen share with them. You're not allowed to see any of that so you have to really synthesize the feedback from the call center coming back through the client to sit here and say, "Okay, what are you hearing? What are the issues?""You can see in our own data," they're saying, "Oh, people seem to be dropping off at this point," or, "This is the number of people who are using a Plaid connection to pull data instead of taking photos and uploading them." So, it's a combination of data analysis and hearing from the customer success team. But then, if you want to make a change, you have to really coordinate with the customer and say, "Hey, this is what we've seen, this is what we're hearing from you. We think this will make the experience better," you have to talk them through the process. So, this is where a product manager really has to lean on the stakeholder management skills of some of the, what I always call, the sales process of product management which is why does this idea make sense, what's the data you have, whether it's qualitative or quantitative, to back up your assumptions on why it's going to work.And they're typically leery of AB testing because it means they've got two different experiences that they have to support. And we even ran into this back at Shutterstock working with enterprise customers, they didn't like AB testing or the results would get heavily skewed. So, you have to make a bet and sit here and say we have to have a high confidence level that this change is going to make a positive difference and roll it out and see what happens.

Holly Hester-Reilly:Yeah. I love what you said there about making a bet and having a high confidence level because I think that's a core piece of one of the things that's often different when you're doing B2B product. So many times, my B2B experiences have been such that we're not able to AB test at all and you really have to find other ways of figuring it out before you build it.

Jason Keramidas:Yeah. And so, ideally, you've got enough customers that you can start to identify those customers that are a little more, I don't want to say they're willing to take risks, but they're more enthusiastic about embracing innovation. And so, you find those customers and you sit here, it's like, "Hey, we've got something new or we've got a different idea on how to approach something," and you use them as your guinea pigs, basically, and say, "We think this is going to be good before you start rolling it out to the entirety of your customer base." We definitely did that at Shutterstock with enterprise customers. I remember working with the sales team and saying, "Tell me who your super enthusiastic customers are and we will roll out stuff to them first."Because for the enterprise customers there, we tried AB testing and we found that, if they saw a different experience, people would just go to the other person's desk and have them log in and use the test instead so it would just totally screw up any test data that we got in. And I was like, "All right, well, I guess they like this better so we're just going to roll with it."

Holly Hester-Reilly:Yeah. So, let's pull that string a little bit. So, for our listeners, what was your role at Shutterstock?

Jason Keramidas:So, for people who don't know, Shutterstock licenses stock content, photos, videos, music, et cetera, et cetera. And there's a huge e-commerce business but there's also an enterprise business which is ad agencies, media companies, publishing companies who have very, very different types of licensing needs and the volume of activity they do is much greater. And so, that is where the enterprise sales team at Shutterstock comes in and they manage those relationships directly. And so, I was brought in to elevate that experience, I guess we'll say, because, what they called Shutterstock premiere, when I got there, it was the old e-commerce site. It's like somebody had forgotten it in time and that was a big frustration point for the customers and for the sales team. It's like we're trying to pitch people on the benefit of moving from their e-commerce license to the enterprise license options and they look at the site and it's just, "Why would I use this thing? It's no good."And so, it was a matter of coming in trying to understand, all right, well, how do we make this experience better. And the big thing there, I would say, to anyone who's working with enterprise sales team, they are your greatest asset because they are on the front lines and it became very apparent very quickly talking to the sales team that, oh, these are the issues that people have. The workflow doesn't make sense for an enterprise use case and discovery is a pain for enterprise use cases. And so, it was really clear, it's, oh, these are easy things that we can fix, let's just figure out how to do it.And so, I would say, it's building that relationship was a huge part of what ended up, I think, being a successful 10-year working with that product. Because again, back to the stakeholder management piece, they are your front lines, they have more qualitative data at the very least than you will ever have access to so they need to be an integral part of the process. And so, I worked very, very closely with that team as we tried to evolve the experience.

Holly Hester-Reilly:Do you have any stories from experiences you had with that team or maybe even with some of your own reports working with that team, how they built that relationship?

Jason Keramidas:Sure. So, some of it was just really simple. They never had really regular contact with the product, it was ad hoc and sales felt that they were very much just telling product and engineering, oh, we need a button that does this and the team would just go ahead and build the button but that's not what they wanted. They're like, "We're salespeople. We don't know what to build, we can just tell you what's wrong with it." And so, I'll give you an example. What's an enterprise use case that didn't make sense for e-commerce?Well, enterprise, there were sub-accounts. So, you had a person who was responsible for paying the bill but then you add all of these researchers who would sit there and all they needed to do was discovery. So, part of the sales process was to go and set up the master accounts and then all the sub-accounts for this ad agency. And so, then, there was an activation process where we would send an email to the list of people that we'd been given from the customer to activate their new Shutterstock account and the process was eight steps long. And so, people would click on the link and they'd go through this process and they would just abandon. They say, "Screw this. I've already got a Getty account, I'm just going to use that. Why would I bother using this thing?"And that was really the first domino to fall because, if people aren't activating, they're not searching, if they're not searching, they're not licensing, et cetera, et cetera. So, I sat down there and that was just me and our designer and our tech lead just looking at the data. It's like, "Holy crap, this is horrible." And so, we took a step back and I sat there and I talked to the sales team. I was like, "Well, what data do we actually need from these people in order for them to verify?" And it was like, "Well, we just need them to verify that they have the right email address." I was like, "Well, okay, then why are we asking them all this other information?" And so, I sat down again with design and tech, I was like, "All right. So, if we removed all of these extra steps and just had them do one and two, what does that look like?"And so, we tried it, we rolled it out with one customer and, surprise, surprise activation rate for those customers just shot up because people was like, "Oh, this is super easy." And then we saw search engagement and, like I said, it was all leaning indicators. Activation leads to search, leads to licensing. So, we tried it, it worked and we just said, "Great, we're going to roll this out to everybody." And so, that gave sales the opportunity to reengage customers who maybe didn't have the level of engagement that we wanted from them to sit and say, "Hey, we've got a new activation process. Go out there, try it out," and that was one of the first big improvements that we made on the site.

Holly Hester-Reilly:Yeah. And how did you work with sales during that process? What was their role?

Jason Keramidas:So, a lot of it was, like I said, what do we actually need from these people to verify the accounts. So, just understanding, base level, what are the requirements for us to be able to say, yes, this is Holly responding back to it. It was really simple, send an email, click on the email, validate. And then it was sitting there saying, "Okay, this is the idea we have. Does it make sense?" Enthusiastic, yes. Let's find a partner to roll it out to. And so, what I had done there was there was a huge account management team, I didn't need to sit down with all of them so I asked for that team to select a couple of representatives to sit and talk to me and spoke to operations folks, et cetera, et cetera.We said, "All right, let's lay out the different options on the table. Who might be a good candidate for this to work with?" And we kicked around different options with the pros and cons of each one until we finally decided on that target customer with a couple of backups. And so, we sat there and said, "All right, now you're going to do the outreach. Talk to them, see if they're excited about it, they want to give it a shot." They did the outreach, we rolled out the change to that customer, explained to them what it's going to mean. It was a really straightforward change so not a whole lot to talk through. And then rolled it out and watched it for a couple of weeks to see what the difference was and it was a pretty quick change to see.So, this isn't the most sophisticated example but sales really managed that relationship of talking to them saying, "Hey, we've rolled it out. What are you seeing? What are you doing?" And meanwhile, I sat there and looked at the data coming in on our side and it showed us we sent out X number of emails on this date, this is the number of accounts that activated off of those emails compared to what we had seen previously for this customer and the numbers really spoke for themselves. And once we got that buy-in, it was pretty easy to then go out to other customers and say, "Hey, we're rolling out this feature. We've seen X, Y, Z as a result of testing it so just a heads-up."

Holly Hester-Reilly:So, one of the things that I'm noticing as you share that story is you almost make it sound easy to work with sales. Why is that? Has it ever not been easy for you?

Jason Keramidas:There have been times, definitely, that it's challenging working with sales. I think it depends on the nature of the thing you're trying to do. And so, I think how you approach a sales team really matters and, honestly, it also depends a lot on the personality of the sales team. And in this case, I was fortunate in that the bar was so low of what they were expecting. That going in and just treating them as equals and real participants in the process of building a better experience, they were like, "Yes, fantastic." The thing is, you can't take it for granted, you need to continue to build that relationship over time, you need to deliver. And that to me is, if you work closely and then you deliver results, sales can be a huge ally in what it is that you want to do.I remember the head of sales at Shutterstock was really excited about some of the things that we're doing so he became a big advocate with me when I was going out pushing that. It's like, "We need more people on the enterprise product," and he was pounding his fist. He's like, "Give us more people because we finally have this figured out." But sometimes you have to push back on sales and that needs to be managed delicately.When I was at Everyday Health, I was working with the sales team and we were selling basically advertising products to pharma customers and we were tasked with building out this healthcare professional product that I was working on. And so, it was a medical news site and there had to be this really clear wall between editorial content and advertising content and sales very much wanted to drive engagement on the advertising content and I heard the concerns from the editorial staff that it would look like the editorial team was shilling for pharma that they didn't want to cross that line.So, that's something where I had to work with design to make the content appealing but distinct from the editorial content and that was a series of somewhat heated conversations between me and editorial and sales around why that needed to be the case. But ultimately, again, you need to explain to them how you're still going to deliver the results and the proof is in the pudding. It's like, "Look, we are going to make sure that there is engagement on this content but it can't look like we're tricking people into reading stuff that looks like news," for example.So, again, the thing with sales is they're very outcomes driven, they're typically responsible for driving some revenue number. And if you can sit here and explain why what you are doing is either going to help them hit those revenue goals or, at the very least, not hamper them from hitting those revenue goals, that's the thing. I would say, just keep in the back of your mind around understanding what their motivations are compared to other stakeholders you might have and that's just in general. When you're talking to stakeholders, understand what's driving their decision-making processes and what's driving their thoughts.

Holly Hester-Reilly:Yeah, I love doing that with the people that I coach and helping them understand that perspective of what is motivating the person that you might be struggling to connect with or work well with.

Jason Keramidas:Right.

Holly Hester-Reilly:Okay. So, I love your stories from the Shutterstock days, I saw many of them up close, definitely could go into more there.

Jason Keramidas:Yeah, that's three podcasts in and of themselves.

Holly Hester-Reilly:Exactly. There's a lot to say there. I also am eager to hear about your time at Charity: Water because I myself have never done nonprofit product management and I'm fascinated by that. So, I'm curious to hear, what was that like?

Jason Keramidas:Yeah, well, talk about a 180-degree turn. So, I went from being then subsequently a VP at Shutterstock working on all these different projects to then going to a nonprofit and why the heck would you go work for a nonprofit. So, there was an election in 2016, if people remember, and it forced me to sit here and say, "Well, how can I" ... Nothing against Shutterstock, fantastic business model, very successful business but I was sitting there saying I need to work on something that has a little more meaning to it. And I'll be honest, I had no idea what shape that would take. My son had literally just been born and I'm sitting here figuring out, okay, what can I do to make the world a better place using product management.And I was stewing on it for a bit and, as fate would have it, a recruiter reached out to me and she said, "Have you ever thought of doing nonprofit work?" And I said, "Well, no, but I am always interested in having a conversation." So, I met the folks at Charity: Water, people who don't know Charity: Water, it is very self-explanatory, it is a nonprofit focused on ending the world water crisis. And the CEO there, Scott Harrison, who's got his own fascinating story, I won't get into all the details around how he founded this but he always treated Charity: Water as if it was a tech company. And so, what does that mean from a nonprofit perspective? It's that Charity: Water has embraced technology in a way that is atypical for nonprofits.And so, as I was saying there evaluating opportunities, it struck me as this is probably the softest potential landing spot for me to transition into a completely different space with different types of motivations. And so, how does Charity: Water embrace technology? The first is Charity: Water's flagship partnerships have been in Ethiopia and so they've installed thousands and thousands of wells in Ethiopia but the interesting part is they've installed sensors on each of those wells. And so, there is a live data feed coming back where you can see the water being pumped through the different wells. You can see, not just the volume of water, but like, huh, this one hasn't pumped water in a while, is something wrong with it? So, it contributed to understand we need to send a maintenance team out to that well to check out if something's wrong with the equipment or has the well gone dry or whatever. So, it helped as an analysis tool. So, that was a big thing with Charity: Water.The other thing is they have this 100% model. So, 100% of the donations you make to Charity: Water go out into the field. Charity: Water eats the credit card fees, et cetera, et cetera because they have a separate fundraising effort around their operational costs. And so, when I was brought in, it was like, "Okay, how do we leverage all this stuff that we've done to grow a recurring giving product?" And so, that was the big shift that Charity: Water was looking to make is to move into a monthly giving instead of one-time donations. Their real flagship product to that point had been people giving up their birthdays for water. So, instead of gifts, I'm going to have a money target, I'm going to raise $1,000 for water, you tell your friends to give to your campaign and then Charity: Water, part of the data and technology, they track where every dollar is sent.And so, if you do a campaign and raise $1,000, it takes about $10,000 to fund a new well, for example. They will sit here and say, "Oh, your money went to drill this specific well in this country, you can see who else donated to fund it, here's a picture of the barcode, essentially, for the well," and you can see this is where my money went. So, that was one of the big innovations. They wanted to sit here and say how do we have this proof concept built into our recurring giving product and how do we grow that model?

Holly Hester-Reilly:Can I ask you a question?

Jason Keramidas:Yeah, sure.

Holly Hester-Reilly:So, how did the ability to see exactly where the money was going impact givers' likelihood of giving?

Jason Keramidas:So, the thing people don't know about nonprofits generally is how opaque they are in terms of how their money is spent and so that's really one of the big differentiators for Charity: Water. With this 100% model, you know exactly where your dollars are going, we're not hiding anything and you're not contributing to the operations of the business at all. So, you know where your donation is going, we prove it, we'll show it to you, et cetera, et cetera. So, that was, in the nonprofit world, revolutionary when it came up and it all came out of when Scott started the thing now 15, no, more years ago.

Holly Hester-Reilly:Yeah. And has it spread throughout the nonprofit world?

Jason Keramidas:No, because it's very hard to do. And in fact, a lot of nonprofits don't like Charity: Water for that very reason.

Holly Hester-Reilly:Is it making them look bad?

Jason Keramidas:Well, you have to embrace this notion that you're going to show and, yeah, call a spade a spade, yeah, it makes them look bad. It's like, "Look, we can be transparent, you could do it too if you embrace this," but they don't want to do it. And so, there's all these charity ratings, how people treat the accounting and everything, that's a whole rabbit hole that you can get into. But when people sit here and they evaluate how do I have confidence that the money I'm giving is going to be actually used, Charity: Water is able to stand behind it. And there have been, again, scandals like the Wounded Veterans one where money is siphoned off and funding lavish lifestyles, et cetera, et cetera, there's none of that at Charity: Water.

Holly Hester-Reilly:It sounds really cool. So, how did that journey play out for you?

Jason Keramidas:So, I got there and they had just started this process of trying to go to recurring giving. And again, I'm going from working on enterprise products at Shutterstock and looking at this gigantic migration project that we were doing at Shutterstock to now how do I grow a subscription business at Charity: Water. And what's, again, interesting with nonprofits is that, for the most part, they're years behind in terms of modern acquisition strategies, for example. So, okay, they had gotten a mention in a newsletter that had driven a thousand or so signups which was great but, beyond that, there wasn't much of an acquisition or retention strategy. So, I had to sit here and say, "All right, well, let's look at what our options are. We've got referral networks that we can leverage, we've got advertising that we can do." By the way, Facebook advertising, certainly back then, is worth its weight in gold. And I introduced the basic pirate metrics concept to them.So, it's acquisition, activation, revenue, retention and referrals. And sitting here I was like, "Let's start measuring ourselves against these types of metrics which, again, for a nonprofit, is atypical." So, just having some framework to evaluate what it is that we're doing and then we looked at all the different types of initiatives that we could try out that would help drive those different goals. And so, the first one was like, "Well, how do we diversify our acquisition channels?" So, the first thing was certainly running some tests and validating that social media advertising, but very specifically Facebook advertising, generated a huge ROI. So, it's working closely with a demand gen person at Charity: Water. So, I was like, "How do we advocate for her having that budget to go out and try that certainly improving SEO performance, all that kind of standard stuff."But then the other piece was leveraging referrals. And so, there are all kinds of influencers who Scott has access to because of how popular he is as a speaker and people who are really inspired by the mission and the history of Charity: Water and leveraging those networks. So, the first experiment we did along those lines is, you may not know this, Seth Godin, who has written a number of marketing books, he is a huge Charity: Water supporter and, every year on his birthday, he's done a Charity: Water campaign.

Holly Hester-Reilly:Oh, cool.

Jason Keramidas:So, historically, that was always the standard Charity: Water birthday product. But when I got there, I was like, "Hey, let's ask Seth to do an ask for people to join what we call the spring," it's this recurring giving, "Just to see what the conversion rate and the traffic looks like." So, we did something new, we built a bespoke landing page specifically for Seth's audience with a unique URL so we had all that tracking in place to see what the engagement looked like and it was really an experiment to sit and see does this thing work. And we saw that, he put it in his newsletter, which goes out daily, some people clicking on the link, we saw the conversion rate super high. We had a video, there's this Charity: Water video that we use everywhere, we tried different pricing options just to see what it would look like and it looked like, wow, referrals can be a big source of signups for us as opposed to the standard mechanisms and it seemed to have value that it was related to Seth himself, that Seth Godin invites you to join the spring, here's a video.So, then we started to iterate on that product. We went from just doing a, admittedly, hacked together bespoke page in putting up there that, as we did more of these experiments and the data continued to validate that this is a thing that has real legs, well, then we built an internal tool to where we had a person who managed these types of influencer relationships that they could spin up a bespoke page with a custom URL, we could change the background image to have a picture of this person on there, we could customize the messaging, we had the ability to customize the video content where they could do an intro and/or an outro to the video content that we presented to people. And we just built and built and built on top of these things and continue to optimize the product.It's like they're looking to do one-time donations instead of subscriptions. Oh, they want the price level to be here because they know that their audience is more likely to spend this amount of money, blah, blah, blah. And it became a whole thing and it accounted for about 40% of our acquisition was driven through this channel.

Holly Hester-Reilly:Wow. That's an awesome story of the iterations and ...

Jason Keramidas:Oh, that was the thing. And again, you always have in the back of your head when you know every dollar we raise here is going to get someone access to clean water and we hammer that message home. And the typical amount that Charity: Water asks for is 30 bucks because 30 bucks is what it averages out to, that's what gets one person access to clean water. And when you have that as your own motivation as a product manager to make this thing work, it's a different inspiration to make something successful. I am helping people in a way that I could never have done as an individual and, by making this thing perform better, we're helping more people and so we just doubled down on that experience.

Holly Hester-Reilly:That's so awesome. So, why did you ever leave?

Jason Keramidas:So, I was the chief product officer at Charity: Water, big title but I oversaw engineering product, data and design. I had 11 product managers rolling up to me when I left Shutterstock, I had eight engineers, a designer, a product manager and a data person at Charity: Water. And it's not a product led organization by any stretch of the imagination, never going to be, it doesn't need to be, that's not how they function. Most of Charity: Water is fundraising and strategic partnerships and everything and then the operations side of the house, oh, actually executing the projects.

Holly Hester-Reilly:Right. Actually drilling the wells.

Jason Keramidas:Exactly. And so, Charity: Water, they have an operations team that works with these NGOs in the country so that we work with local partners who know the world water crisis.

Holly Hester-Reilly:And for any listeners who don't know, NGO is a non-government organization, right?

Jason Keramidas:Correct. And so, the size of the team there, that's as big as it was ever going to get. And so, when I sit there and say, "Okay, for my own career trajectory, et cetera, et cetera, what is it that I need to do?" And the reality was I need to go to a space that was going to have ... I could build cross-functional teams and get back to really more product led companies.

Holly Hester-Reilly:Yeah, awesome. So, I love these stories and I wish we could just keep talking forever but, unfortunately, we're about out of time.

Jason Keramidas:Too bad but, again, always fun to chat.

Holly Hester-Reilly:Yeah. So, where can people find you if they want to learn more about Jason?

Jason Keramidas:So, certainly LinkedIn and Twitter, it's the same thing, jkeramidas. One of the benefits of having a Greek last name is that stuff what's available. So, yeah, people can find me there.

Holly Hester-Reilly:Awesome, all right. Well, we'll put that in the show notes as well and thank you so much. This has been so much fun, Jason.

Jason Keramidas:Thank you, Holly. It was a great time, great to catch up and I'll keep listening.

Holly Hester-Reilly:All right, sounds great.The Product Science Podcast is brought to you by H2R Product Science. We teach startup founders and product leaders how to use the product science method to discover the strongest product opportunities and lay the foundations for high growth products, teams and businesses. Learn more at h2rproductscience.com.Enjoying this episode? Don't forget to subscribe so you don't miss next week's episode. I also encourage you to visit us at productsciencepodcast.com to sign up for more information and resources from me and our guests. If you like the show, a rating and review would be greatly appreciated. Thank you.

‍