There is no single or perfect way to get the full view of what customers really feel and need from our product. And that’s OK.
Survey Sparrow, the makers of your favourite online survey software, was recently fortunate enough to chat with Daniel Zacarias, the founder of Folding Burritos, one of the best project management sites online, as well as the co-founder of career.pm, a website dedicated to helping people navigate the tricky world of product management careers. Daniel has a wealth of knowledge to share with you today about what ingredients are required to launch a successful software product.
Let’s dive into the interview.
SurveySparrow: Hello and thank you for joining us today to talk about your involvement in the product management space. You have an amazing podcast series where you talk to 14 product managers about many topics, but you have a focus on talking about setting up effective customer feedback strategies. Out of the 10 hours of podcasts you’ve produced, what are the three most common recurring pieces of advice you receive from your guests on this topic?
Daniel: Thanks for having me! I think the most important bits of advice are:
1. Consider the context.
Any bit of feedback you get will only be meaningful if you’re aware where it’s coming from, across many different dimensions. That is, which user segment is it from?
And that’s where it gets interesting. There are so many interesting dimensions along which we could segment our user or customer base. We may use demographic characteristics or behaviors, although they’re often not very helpful for PM purposes.
We may use a Needs-based segmentation (AKA “Jobs-to-be-Done”), which is extremely useful when we both know which jobs our product serves and how users map to them.
There’s also usage-based segmentation (frequency and extent of use), longevity (how long has a user been with us?), perceived value (satisfaction and importance score for certain feature sets) and of course, actual invested value (total revenue or time spent by a customer with the product).
All of these things are interesting for different purposes, and completely change the meaning and importance of any bit of feedback we get.
It’s a two-level processing system:
- Do we know “Who” is giving us this feedback and why?
- Is this something that we want to focus on right now?
If the answer to the latter is No, then you can safely move on to whatever else might move your needle — there are never enough resources, so you might as well focus on what matters to your goals.
When the answer is Yes, you can proceed with a clearer definition of how to evaluate success.
That’s why is so important to be mindful of context when analyzing customer feedback.
2. It’s a team effort
This one should be a no-brainer for all PMs, but often isn’t. We need to be aware that
(a) we’re not the only ones receiving solicited and unsolicited feedback,
(b) not everyone is aware of what customers are saying and need,
(c) we have to work with the broader cross-functional team to help them achieve their needs and they capture the right kind of feedback we need as PMs.
Getting to each of my points:
a) There are many different teams interacting with customers and prospects every day. Support, Sales, Marketing are the most obvious ones. Those teams need to be part of your user feedback process, or you’ll lose a ton of really valuable insight.
b) PMs are uniquely positioned in the product development process to get a sense of what’s going on everywhere else, as well as with our customers and users.
That’s not the case for many people in your organization, which have different responsibilities and might not be exposed to customer inputs at all, or are biased by their own roles (For instance, support usually deals with people when things are going wrong, whilst Sales are talking to prospects who are trying to negotiate and understand if the product will be a right fit for them).
c) This means that we have to bring everything together, make sense of it and share it widely across the organization. That’s a critical starting point for further conversations and alignment on the product strategy.
3. Something is better than nothing
There is no single or perfect way to get the full view of what customers really feel and need from our product. And that’s OK. We just have to try to get at it from multiple angles to get the best view possible. However, we can’t let the ideal be the enemy of the good.
Having any sort of user feedback on our hands is better than having none, so we shouldn’t worry too much about not being able to reach the right customer segment, or using a tool like NPS which has a lot of caveats. It’s something. It’s helping us get to a more informed place.
SurveySparrow: Software and burritos…. What’s the connection?
Daniel: Well, when I was trying to come up with a name for the blog, I wanted something original.
Then, it hit me.
I love burritos. I sometimes make them at home. But they’re tricky to get right. If you cram too much stuff in them, flavors overpower each other and they’ll fall apart and make a mess. If you don’t fold them properly, they also fall apart. Burritos need to have the right amount and mix of ingredients so they’re tasty and not overwhelming. You also need proper technique so they hold their structure.
After one such episode, I realized the same thing was true with software. You need to choose the right set (and amount) of features and possess the know-how to put the whole thing together properly and deliver it to your customers.
SurveySparrow: On one of your blog posts on the topic of using the Kano Model to help with the prioritization of product features, you talk about the importance of ensuring your survey questions are as succinct as possible. What are some pieces of advice you would give to product survey creators when it comes to creating effective questions?
Daniel: There are tons of things to consider, but I’d say the most immediately actionable are:
- Keep them short and to the point – clear language is key to get better responses.
- Don’t ask what you already know – don’t annoy users asking them for data you have in your database, just because your survey tool isn’t integrated. Figure it out.
- Don’t ask something you don’t absolutely need to know – it’s simple: the longer the survey, the lower response rates you get.
SurveySparrow: What are three of the biggest mistakes you see product teams make when it comes to developing surveys around product development?
Daniel:
Not having a clear objective of what they want from the survey (or having too many)
This reflects surveys that are long, complicated or confusing (or all of the above). The more focused we make them, the better.
It sometimes seems as though there’s an actual physical cost into putting a survey together and people try to cram as much stuff as they can into them. That just doesn’t make sense on digital.
Surveying the wrong cohort at the wrong time
This reflects a very high variance of results or inconsistent responses. We need to be careful to reach the right group of people at the right time, so the results we get are better able to serve our survey’s objective.
Using surveys for things that should probably be asked on interviews or user tests
Asking people for open-ended feedback in forms is certainly useful and has a very important place, but it can be overused and lead to lower response rates or lower quality ones. If people don’t like to type a long answer, are too polite (or too rude), we won’t get to dig further and understand “what’s behind” their answer (or lack of one).
There’s a richness we get from observing a person replying to a question or being able to follow-up with them – and we need to keep that in mind for certain kinds of questions.
SurveySparrow: In one of your blog posts, you point out that the main goal of a product manager is to deliver user value. However, you mention that feature growth is not equal to user value. What do you mean by this? Why are software developers getting this wrong?
Daniel: What I mean is that adding features doesn’t equate to creating value for users. It can actually have the opposite effect.
Delivering value for users is about helping them solve a problem or need in a better way than what they previously did (or couldn’t do).
I see team after team working really hard to add features that only create marginal value (or none at all because they’re not used).
When you add too many of those, you get to a product that is complex to use and expensive to change, which means that value for users can go down – because they can’t use it or because the product doesn’t evolve as fast as they need it to (or as fast as competitors might)
SurveySparrow: You talk about the importance of rejecting “most” feature requests. Why is this important? What are the main consequences of being too open to new feature requests?
Daniel: The main issue is that if we take feature requests, we are reinforcing this message that “we’re feature builders”. We’re putting ourselves in what Melissa Perri calls the Build Trap and John Cutler calls the Feature Factory. We’re here to solve those problems for our customers that can also move the needle for our business.
The other aspect to this is that feature requests are usually solutions framed within what the product already does or what people imagine is possible. This tremendously constraints the solution space and biases us towards a particular path, where there might be already existing workarounds or more effective solutions otherwise.
Finally, the other major problem with “feature requests” is that they might not be aligned with our product strategy at all… so why should we consider them? It can set us on a reactive path, depending on the scope & effort required to build them. Also, they generate expectations if we just “take note” of them–whoever’s asking will at some point want to see it.
The right approach when it comes to handling feature requests is to dig further, understand the problem, and at that point set the expectation to the requesters that their need is (or isn’t) aligned with the product’s strategy, and if they can expect to see something that helps them sometime in the near future or not.
If you can’t provide that to them, they will at least understand why and you have an opportunity to provide a workaround. Of course, if there’s enough demand for a certain need, then you need to evaluate your strategic priorities and address it.
SurveySparrow: You have an interesting take on this, where you talk about the importance of thinking about problems rather than solutions first. You go as far to say that solutions “suck” and problems “rock”. What do you mean by this?
Daniel: This is related to what I described before about feature requests. Features are solutions, and people tend to come up with their ideas (features) to solve some need they have, either because they feel it’s helpful to you, or because features are more tangible than problem descriptions, or because they really would like to see their “intellectual offspring” work its way into the product.
No matter the reason behind this, it generates a solution-first culture around you as a PM, which you must counter and help educate everyone around you to dig deeper into the problem they’re trying to solve and come to you with that, rather than a feature.
Just to elaborate on why solutions suck…
1. The product is viewed as a set of features
When you take in feature ideas and requests, you’re incentivizing people to think of a product as a collection of features. As this mindset deepens, it becomes increasingly easier to add things without fully considering how they fit into the broader picture. The result is that the collective understanding of the product vision gets ever fuzzier and the strategy gets increasingly harder to define.
That’s why it’s critical to emphasize that a product is a mechanism to solve customers’ problems. There are a number of ways to achieve the same result, and often they might not even be technological–perhaps we just need a better educational collateral, or position the product differently to attract the right kind of customers.
2. Solutions and ideas are often half-baked
As we all know from being PMs, there are tons of implications and corner cases we need to consider for most features. Those who aren’t used to considering all of these issues, will come to you with ideas that are, for our purposes, half-baked. This means, they’re often not helpful at all.
3. Solutions are harder to break down
Often, we can’t just build the entirety of what we’d like. We break things down into phases, and progressive releases. That’s one of the major benefits of agile development.
However, when we have a solution idea someone requested, without further context, it’s very hard to take that iterative approach. When we don’t clearly know the problem we’re solving, how can we progressively solve it?
4. Solutions get outdated
Another issue with solutions is that if too much time passes from the moment they’re described and defined, until the moment we decide to pick them up, they might even not make sense any more.
The product might have added support for something similar that serves as a workaround; the architecture, APIs, and database might have changed. There are a number of things that might make a solution specification outdated. When that happens, then the time spent specifying and documenting that solution gets wasted as we put it on the backlog.
5. Solutions are harder to prioritize
It’s much harder to prioritize a list of features than a list of problems. There are added layers of indirection that make us evaluate priorities in a different way. It gets difficult to determine the intent and expected impact from a feature.
On the other hand, a problem (“we have a low number of transactions per customer”) can more easily lead to an objective (“increasing number of transactions per customer per month by 30%”). Since objectives map to a business strategy in a much more evident way than features do, then prioritizing them becomes much more straightforward.
6. Solutions don’t empower the team
A modern product team is made of highly intelligent, creative engineers and designers. It’s a vast underutilization of these skills to be accepting fully-formed features from the outside world. Any team loves a purpose and problem to solve. That’s what should be coming in.
My absolute favorite quote to frame this idea is by Antoine de Saint-Exupéry: “When you want to build a ship, do not begin by gathering wood, cutting boards, and distributing work, but awaken within the heart of man, the desire for the vast and endless sea”
SurveySparrow: In one of your blog posts you mention that “Making sure that everyone’s on the same page is the key to avoid random, opinionated, and self-serving requests.” You go onto say that, “aptly identifying and involving the product’s stakeholders throughout the development cycle is the most effective tool we have.” I love this comment. How can teams protect themselves against individuals within the team who want to strong-arm their ideas?
Daniel: We have to be evangelists for PM best practices around our organization. We need to help stakeholders follow the same line of thinking we do.
I call this creating a “problem case”. It involves iterative questioning and discussion, starting from a feature idea, and getting to a clear understanding of:
The problem statement
The root problem that a proposed idea or feature is trying to solve. The ‘5 Whys’ technique is extremely useful here, especially when we use it to explore multiple paths.
Situations leading to it
Ask the stakeholder to take the perspective of the user or customer. Which situations lead the user to face this problem? What creates this problem? Why is it a problem?
Variations on the Problem
Explore different takes on the problem. What if some unforeseen (by the stakeholder) situation happens? What if some of the stakeholder’s assumptions don’t happen?
Alternative solutions (that are already possible)
How could the user solve this problem with what we’ve already got? What if the solution were smaller than proposed? Would that be “good enough”?
Additional problems that might be caused by solving this one
If we work on this, what other things would we have to solve? Which problems are caused by the proposed solutions? Are those problems aligned with our current plans?
Understand the value premises and hypotheses
Agree on concrete hypotheses that can be tested. How many people does it need to affect for this to be considered a pervasive problem? How often does it need to happen? Finally, ask: is solving this problem contributing to our value proposition or not?
SurveySparrow: Who are some companies you think are doing an exceptional job at product management? Why are they excelling?
Daniel: Exceptional product teams excel because they have a deep understanding of their customers, the problems they solve, and have the autonomy and empowerment to deliver solutions for them.
There are tons of examples, but I will shy away from mentioning major companies we all know and admire. Instead, I will just refer to a couple of products that I truly enjoy using every day, which are produced by smaller companies: GoodNotes and 1Password. They each provide an incredible UX and really solve their respective problems extremely well.
This has been a very insightful interview. Thank you Daniel. To our blog readers, if you’d like to learn more about Daniel and the work he does, you can follow him on Twitter, or visit his website here.
