Data Is Your Friend, Not Your Savior
Three product managers walk into a bar. The bartender takes one look at them and says, “I’ll let you drink free all night if you can get one of those girls to go on a date with you.” The first product manager identifies the girl he likes and begins interviewing everyone who has come into contact with her to devise the perfect pick-up line. The second product manager decides to send two drinks with different pick-up lines written on a napkin to every women at the bar, but he is not convinced by the results because the sample size is too small. When he turns to ask the last product manager what to do, he is surprised to see him sitting at the bar laughing and drinking with a women already.
“How did you do that?” the second product manager asks.
The third product manager answers, “ I walked over and asked her if she wanted to drink free all night.”
The allure of data
Making good decisions about building a product is rarely simple. There are a myriad of factors that combine to form a logic maze. Prioritizing one work item over another opens you up to second-guessing, and often third or fourth-guessing. That kind of scrutiny is driving many product managers to don protective armor lined with analytics. They cannot be wrong if the data backs them up. At least, that is the hope.
Data-driven product development is all the rage in Silicon Valley. Making a decision without running an A/B test is considered tantamount to driving blindfolded the wrong way down a one-way street.
The only way to be right is to setup an experiment and prove it with statistical significance. One is left to wonder how great products like Adobe Photoshop or the iPod were ever built without data buttressing every choice along the way.
The truth is that data is a thrilling (yes, thrilling) addition to the product management toolbox. It just happens to be one tool, not the only tool.
There is sweet relief in knowing with certainty that one design outperforms another. Nothing shuts down endless debate or executive meddling better than statistical proof. It has the added benefit of increasing your credibility, which is every product manager’s most precious commodity.
Leveraging data in guiding a product is about more than bucket tests and experiments. Any product manager who does not know where users are falling out of the funnel would be wise to get out of their hot tub time machine and return to this century.
It is a gift to see what percentage of users get past sign-up, and what percentage access different aspects of the product. Applying scientific and economic principles to the complexity of building scalable product value is possibly the most important advancement in technology since Red Bull.
The problem is that many are mistaking data as their salvation from tough decisions — hoping that an experiment result will always cover their backs or that analytics reports will act as a dotted line on a treasure map leading to the big X. Sorry, product managers, you are still on the hook for making good choices, even with a big satchel of data by your side.
Data-wise, not data-driven
Consider the car. Humans drove them for decades fully relying on maps, signs and landmarks to navigate. The introduction of GPS changed all that. Many of us have become reliant on the technology to get anywhere. The idea of needing to remember directions or pay attention to street signs feels like a near-tragic inconvenience. The next fundamental shift will likely be driverless cars. You will no longer even be required to execute the directions — just get in, pick a destination, and get out.
Some would like to believe product development is headed toward this level of simplicity and certainty. Just pick the product you want to build and market you want to reach, and let data take the wheel. Not so fast.
There are multiple factors that must be considered — I call these signals — when building a product.
Stakeholders
Everyone building a product answers to someone. Entrepreneurs must satisfy their investors or the bank. Product managers within a company need to be aware of what their executives want. Fail to capture and maintain their belief, and you will find yourself trying to take flight on a very short runway.
Let’s stick with the car analogy. Stakeholders have the power to change your destination mid-trip. You thought there was an agreement on going after this market, but the strategic sands have shifted and now you need to go after another. Data won’t tell you that.
Similarly, patience can run out. You may have picked the right product and the right market and data might be helping you chart a terrific course to get there, but your stakeholders could look at the arrival time and decide they would rather go somewhere else, and you are driving a company car.
User interviews
How do you pick your destination in the first place? Part of that process has to be talking to the people who you ultimately want to use your product. Market research and industry trends play a big role, but it is tough doing anything meaningful without chatting with at least a few target customers.
Trying to make product decisions without talking to users is a bit like trying to resolve a conflict over email. Too much valuable context is lost.
We, on the Adobe Spark team, have found that data is very helpful in teeing up the right questions (e.g., Why are users falling out of the funnel at this step?). Getting to actionable answers, though, often comes from talking to people. Data is certainly part of your GPS system, but marrying that with user interviews is like adding in the traffic patterns to choose a better route.
User testing
I could — and probably will — write a whole post just about this signal alone. It has had at least as much impact on the way I approach product refinement as data has. Our excuse as product managers used to be that we did not have the time or money to recruit, run and interpret the results of user tests. No longer.
There are plenty of services that allow for on-demand user tests to be run at the drop of a hat. There is no reason to spin endlessly at the whiteboard trying to convince one another about the right feature design when you get unbiased user feedback in less than hour. Taking that debate out of the meeting room not only helps product managers make better decisions, but defuses what can be morale-sapping conflicts within the team.
This signal is best used to improve usability. Consider this the wind tunnel for that car you are driving.
Your team
Too many product managers try to maintain some air of superiority when leading their teams. Not only is it silly to assume you are smarter than the engineers, designers, marketers and testers working on the product, it makes life far more stressful.
Every person on the team wants to believe the time they are spending will result in something meaningful. They are invested, and often come at the problem in different ways. Reaching out to them to get their wisdom does not weaken your authority. It builds trust and tribal knowledge.
Velocity is a great way to assess how bought in your team is. You may have the destination picked out and your dashboard is glowing green, but if your pace is crawling, something is wrong.
This is your engine. Good luck getting anywhere without them.
Your gut
The least tangible factor in making decisions on your product comes from pure instinct. It also may be the most important. I have observed many product managers who are great at gathering data, running user research, building out a market assessment, but consistently struggle to synthesize all those signals into a pragmatic next step.
This is something that can honed over time, but like the difference between a good musician and a great one, some of this is potential is predetermined. Your ability to be insightful enough to know how much to trust your gut and how much to lean on others will greatly determine your ceiling as a product manager. There will be moments when your instincts are screaming to make a decision that conflicts with the other signals. What will you do?
Using data wisely
Just as there is a rising tide of people who oversell or hide behind the role of data in guiding product development, there are people who belittle and avoid it. As with so many things in life, either extreme is unproductive.
Experimentation
Experiments, used judiciously, add much-needed certainty to design decisions. Tests for discoverability and click-through are two examples of concrete outcomes that A/B tests work reliably for. The rush that comes from testing a few variations and finding one that increases user success by 200% compared to another can be seductive. It can lead to seeing everything as a possible measurable experiment.
I have sat in meetings where hours have been spent trying to devise the data point that will be used to judge whether adding a set of buttons would help move our numbers. Even the Ph.D researcher in the room acknowledged it would not be conclusive. The truth was we probably just needed to add the buttons and move on. We could monitor how they performed, even if we could not scientifically prove that they increased the metric we were hoping they would impact.
If devising the experiment is taking longer than the design and implementation of the change, you may want to rely on some of your other signals — including common sense — to move forward. Experiments are supposed to run for at least four weeks, and preferably longer. Making too many things an experiment can have an adverse effect on decision pace. Velocity is a precious commodity. Be wary of any aspect of your development process, even experiments, that slow you down.
A good rule of thumb for deciding whether an experiment is appropriate is whether everyone involved can agree on the structure (i.e., the deciding metric, the competing designs, etc.) in a single day. Taking a week to setup an experiment does not scale, and will end up doing more harm than good.
Measurement
A good dashboard is like a mirror. Take a look and you will see your product for all its beauty and imperfections. How many new users are coming in the front door? How many are returning? How often? How many are leaving? Those are the basics. They are clean and clear.
The next click down is where things tend to get cloudier. Why are they leaving? You can do an analysis of all the features of in your product and look for correlation between usage of certain features and return rate, but you will still be left with interpretation to decide what to do with that information.
For example, we found that people who add text to a Spark Page are considerably more likely to become a repeat user. Funneling everyone to add text as part of their first-use experience would be a knee-jerk reaction. It would also invalidate this indicator (i.e., if everyone was forced to add text, we could no longer consider that an indicator of future retention). A wiser move would be to identify users who add text and those who do not and give them different messaging or experiences based on their behavior.
What should that messaging or experience be? Now you are back to creative thinking and using some of those other signals.
Having the information does not mean you know what to do with it, or about it. That is what separates great product owners from the rest of the pack.
You also must weigh the potential gain of following this thread versus other priorities that may feel like more impactful levers to pull. The courage to do absolutely nothing with a data discovery is underappreciated. It is okay to file it away and come back to it when the path forward is more clear.
Having a good data scientist on staff is becoming as important as design, development and product management. The right person will be naturally curious and be well-versed in the outcomes you are trying to achieve. Instead of overwhelming you with all the data, they will pick the most promising leads or most surprising insights and trends, and bring those to your attention.
Expecting product managers to both prioritize the work and get to the depth necessary to gain meaningful data insights is hopeful, at best. We need to respect the skill set and time required to be great at gathering, analyzing, and reporting on data. Before hiring your next product manager, engineer, or designer, consider whether you should bring aboard an economist or someone of that ilk.
The yellow brick road
Congratulations. Someone just bought into your pitch and gave you a pair of ruby slippers to start your journey toward product nirvana. There will be no yellow brick road to follow. You and your team will be building the path brick-by-brick. Data can help you find the North Star, but you will still need courage, a brain, and heart to succeed.
Embrace what data can add to your product development process, but be careful about abdicating too much power to the man behind the curtain. Your ability to draw wisdom from a variety of sources is what must set the course.
