The Metrics That Lie: What Your Dashboard Isn’t Telling You About Your Product

A few years ago I worked with a team that was obsessed with their seven-day retention number. Every Monday, the first thing anyone looked at was the retention chart. It was climbing — slowly, but consistently. The team felt good. Investors felt good. There was a narrative forming about product-market fit.

Then one of the engineers ran a different query. She broke the retention cohort down by signup source, something nobody had thought to do. What she found: retention for users who came through a particular partnership channel was extraordinary. Retention for everyone else was somewhere between mediocre and bad. The aggregate number — the one everyone was watching — was being carried by a small subpopulation of unusually engaged users while the broader product quietly failed to hold anyone.

This is not a data infrastructure story. It’s a story about what metrics do to thinking.

The Aggregation Problem

Most product dashboards are built around aggregate metrics because they’re simple to compute, simple to display, and simple to explain in board meetings. DAU divided by MAU. Week-one retention across all cohorts. Average session length. Conversion rate from signup to activation.

None of these numbers are wrong. The problem is what gets lost in the aggregation.

Every aggregate metric is a population-level average that collapses the distribution underneath it. A 40% seven-day retention rate could mean 40% of users are genuinely engaged and 60% churned immediately. Or it could mean 80% of users from one segment are retained and 0% from another. Or it could mean some middle path with its own particular shape. The number 40% doesn’t tell you. And the strategies for improving it are completely different in each scenario.

Teams that steer by aggregate metrics alone are essentially flying with a single altimeter reading when what they actually need is topographic data. The aggregate tells you altitude. It doesn’t tell you what’s below you.

Activation: The Metric Most Teams Get Wrong

Activation rate is one of the most widely tracked metrics in early-stage products and one of the most frequently misunderstood.

The typical definition looks something like: the percentage of new users who complete some set of onboarding actions within the first N days. You define a milestone — created their first project, invited a teammate, completed the tutorial — and track how many new users hit it.

The problem is that “activation” as most teams define it is a proxy for a proxy. What you actually care about is whether the user experienced the core value of your product. Completing a tutorial doesn’t necessarily mean that happened. Creating a project doesn’t necessarily mean that happened. These are activities that correlate with value realization in your mental model of how the product works. They may or may not correlate with it in reality.

The teams that get activation right tend to work backward from retention. They look at users who retained well — who were still using the product six weeks in, who expanded their usage, who became advocates — and ask: what did those users do in their first week? Not what did we tell them to do. What did they actually do?

Sometimes the answers are surprising. The activation milestone you’ve been tracking turns out not to predict retention at all. The thing that predicts it is something you haven’t been tracking — a specific feature interaction, a depth of engagement with a particular workflow, an integration connected early.

Your activation metric is a hypothesis about what constitutes value realization. Like all hypotheses, it should be tested rather than assumed.

The North Star Problem

The “North Star Metric” framework has become almost universal in product circles. Pick the one metric that best represents the value your product delivers. Make sure everyone on the team knows it and is working toward it. Let it guide prioritization.

It’s a good framework — or rather, it was a useful corrective when it emerged, pushing teams away from vanity metrics and toward measures that actually correlated with delivered value. The problem is how it gets implemented in practice.

Most teams pick their North Star metric once, usually early, and then treat it as settled. The metric becomes an organizational fixture. Dashboard real estate, quarterly goals, and performance reviews are built around it. Changing it feels like admitting you were wrong before, which carries its own organizational cost.

But what a metric means changes as a product matures. A North Star that accurately represented value delivery at 1,000 users may be masking structural problems at 100,000 users. The user population is different. The use cases have diversified. The original assumptions about what drives engagement have been complicated by real-world data.

I’ve seen teams hold onto North Star metrics long past the point where those metrics still meant what they originally meant — because changing them was awkward, not because changing them was wrong.

Engagement Metrics and the Engagement Trap

Here is an uncomfortable truth about engagement metrics: they’re designed to be optimized, and optimizing them doesn’t necessarily mean your product is getting better.

Daily active users go up when you send more push notifications. Session length goes up when navigation is confusing and users can’t find what they need. Feature usage metrics go up when you prompt users to use features whether or not those features help them accomplish their goals.

Consumer apps figured this out a long time ago and built business models around it. The engagement metrics at most social platforms look excellent. Whether those products are delivering genuine value to their users is a different question.

B2B and developer tools are less susceptible to this pattern but not immune to it. It shows up more subtly: teams optimizing for “daily active users” in a context where daily usage isn’t actually the right pattern for the product. It shows up as treating “session length” as a proxy for engagement quality when shorter, more focused sessions might actually be a sign the product is working better. It shows up as celebrating feature adoption numbers without distinguishing between users who find a feature genuinely useful and users who tried it once because it was in the onboarding flow.

The question to ask about any engagement metric is: does more of this thing actually mean users are getting more value? Not “does it correlate in the aggregate.” Does more of this thing, for a specific user, mean they’re getting more value from the product?

What to Measure Instead (Or Alongside)

This isn’t an argument against measuring things. It’s an argument for measuring the right things, and for understanding what your measurements can and cannot tell you.

A few principles that hold up across different product contexts:

Measure outcomes, not activities. “Connected their first integration” is an activity. “Saved measurable time on a workflow they were doing manually before” is an outcome. Outcomes are harder to measure but they’re what you actually care about. Proxy metrics that correlate with outcomes are useful; proxy metrics that don’t correlate with outcomes just feel like they should.

Segment before you conclude. Before drawing any conclusion from an aggregate metric, ask whether the conclusion holds across segments. User acquisition channel, cohort month, plan tier, company size, geography — there’s almost always a segmentation that reveals the aggregate hiding something important. If a trend shows up consistently across segments, that’s real signal. If it’s being driven by one segment, the strategy implications are completely different.

Measure the rate of bad outcomes, not just good ones. Retention rates measure who stayed. They don’t capture why people left, or what they experienced before they did. Teams with good data discipline track both the positive funnel and the failure modes: support ticket volume broken down by category, the specific friction points in onboarding where users abandon, the workflows users start and don’t complete. Where you’re failing is as important as where you’re succeeding, and it’s usually more actionable.

Track leading indicators alongside lagging ones. Revenue and retention are lagging indicators — they tell you what happened over the past period. They’re important but they don’t give you much steering information. Leading indicators are things you can observe today that predict lagging outcomes weeks or months from now. For most products, the leading indicators are some combination of early engagement patterns, feature adoption sequences, and behavioral signatures that correlate with eventual churn or expansion. Finding yours requires analysis; once you have them, they’re enormously more actionable than waiting for retention or revenue numbers to tell you what already happened.

The Qualitative Data You Keep Ignoring

There’s a type of data that most product teams have access to and systematically underweight: what users say, directly, when you ask them.

Not NPS surveys. Not satisfaction scores. Actual conversations. Support tickets. User interviews. The messages your sales team gets from churning customers. The thing users say when they cancel their subscription and you have the option to ask them why.

Quantitative metrics are good at telling you that something is happening. They’re often poor at telling you why it’s happening or what to do about it. The qualitative signal fills that gap. A retention curve that’s declining can look like ten different problems. Four or five conversations with recently churned users usually narrows it down to one or two.

Teams that have figured this out don’t treat qualitative data as a supplement to quantitative analysis. They treat them as complementary systems that answer different questions. The quantitative data tells you where to look. The qualitative data tells you what you’re looking at.

Metrics and Organizational Behavior

The most important thing about metrics isn’t the numbers. It’s what they do to how teams make decisions and where they focus attention.

Metrics shape behavior through three mechanisms: they determine what gets visible attention in meetings, they influence how people in the organization perceive their own performance, and they create implicit signals about what kinds of work are valued.

A team that tracks weekly active users in every hands-on meeting will unconsciously optimize for things that move weekly active users, whether or not those things are the highest-value work available. A team that tracks revenue per customer and time-to-value for new cohorts is having different conversations and making different tradeoffs.

This means that choosing what to measure is not a technical decision. It’s a leadership decision about what kind of attention you want the organization to pay. The metrics you choose to put in front of people regularly are the things you’re asking them to care about. Choose accordingly.

The Measure Worth Having

The single question that cuts through most metric confusion is deceptively simple: are users getting better at accomplishing what they came here to do?

Not “are they using the product more.” Not “are they clicking on things we built.” Are they getting better outcomes from the work they do with this product over time? Are the problems they came here to solve getting easier to solve? Are they solving problems they couldn’t have solved before?

That question doesn’t map neatly to a single metric. It requires a combination of behavioral data, qualitative feedback, and honest assessment of whether the product’s core value proposition is actually delivering. It’s harder to put in a weekly dashboard than DAU or seven-day retention.

But it’s the question that matters. And teams that keep it close — that use their metrics as windows onto that question rather than substitutes for it — tend to build products that hold up over time.

The metrics that lie aren’t the wrong numbers. They’re the right numbers, read too shallowly. The dashboard is never the territory. The territory is what’s happening with your users, and you have to keep looking for it.

---

Priya Sharma writes about product strategy, team dynamics, and the gap between what we measure and what we mean. She has worked with early-stage and growth-stage teams across SaaS, developer tooling, and AI products.