The Growth Experiment Scorecard: Pick, Run, Kill

Most teams run experiments to feel productive. The teams that grow run experiments to reduce the cost of being wrong.

📋 Read time: 14 minutes. Use time: every sprint you run an experiment.

Why This Exists

Growth experimentation has a productivity theater problem. Teams run A/B tests on button colors while the core value proposition is unclear. They celebrate statistical significance on metrics that don't connect to revenue. They let inconclusive experiments sit in a backlog while the next one starts, which means the learning never compounds.

The teams that actually grow from experimentation share one habit: they score experiments before they run them. Not to gatekeep ideas, but to allocate attention correctly. A bad experiment that takes two weeks is not a small mistake. It's two weeks of engineering time, two weeks of distracted product thinking, and often two weeks of users in a degraded experience.

This framework gives you a scoring system and a running protocol. Use it to pick the right experiments, run them tightly, and kill them cleanly when they stop being worth it. The goal is not more experiments. The goal is experiments that teach you something you can act on.

How to Use This

1. Score before you build. Run every experiment idea through the scoring system in Step 2 before any code gets written or designs get made. This takes 20 minutes and saves weeks.
2. Set the kill condition upfront. Before an experiment starts, write down the condition that ends it: the metric threshold, the timeline, or the signal that says "we've learned what we needed."
3. Run the debrief before the next one. Don't start a new experiment until the last one has a written summary. One paragraph is enough. The discipline is what matters.
4. Use the templates. The experiment brief template and debrief template at the end are copy-paste ready. Make them your team's default.

Step 1: Define the Belief You're Testing (Not the Feature)

Every experiment is a bet on a belief. The mistake most teams make is designing experiments around features instead of beliefs.

"Let's test a new onboarding flow" is not a testable belief. "Users who complete step 3 in onboarding are twice as likely to activate, but most drop off there because the value isn't clear yet" is a testable belief. One of these produces learning. The other produces a before/after metric that you'll argue about for a week.

Before anything else, write the belief in this format:

We believe that [target user] will [take this action] because [underlying reason].
We'll know we're right if [specific metric] moves by [threshold] within [timeframe].

A completed example:

We believe that new signups who see a pre-populated project template will complete setup
because the blank-slate experience creates decision paralysis.
We'll know we're right if setup completion (within first session) increases by 15%
over a 2-week window.

This format forces three things: who the experiment is for, what behavior you expect to change, and what "right" looks like. If you can't fill it out, the experiment isn't ready to run.

Step 2: Score the Experiment Before You Build It

Not all experiments deserve the same investment. This scoring system takes 15 minutes per experiment and keeps your roadmap from filling up with low-signal tests.

Score each dimension from 1 to 3. Total score guides investment level.

Score interpretation:

• 12-15: High-priority experiment. Full brief, clear owner, run it this sprint.
• 8-11: Medium priority. Worth running, but batch with other work or simplify scope first.
• 5-7: Low priority. Redefine the hypothesis or wait until you have more user signal.
• Under 5: Do not run. Replace with a user interview or a smoke test first.

The most common trap: teams run 4-point experiments because someone senior suggested them. The scorecard gives you a neutral way to push back. "Here's how it scored, here's what would need to change to run it."

Step 3: Design the Simplest Version That Teaches You Something

The instinct when an experiment scores well is to build the full version. Resist it. The goal of an experiment is to learn, not to ship a finished feature.

Ask this question before you build: What is the cheapest version of this experiment that produces a valid signal?

Here's a hierarchy of experiment types, ordered by implementation cost:

A common mistake: teams skip straight to full A/B tests because they want clean data. But a concierge test you run with 10 users this week tells you more about why behavior changes than a statistically significant result that gives you a percentage but no reason.

Start lower in this hierarchy than feels comfortable. If the signal is strong, you'll know when to invest more.

Step 4: Set the Exit Conditions Before You Start

This is the most skipped step and the most expensive one to skip. An experiment without exit conditions runs forever, and a running experiment that no one is actively learning from is just technical debt with a hypothesis attached.

Before the experiment goes live, write down three exit conditions:

Condition A: Success threshold. The metric movement that confirms the belief. If you hit this, you document the learning, decide whether to build the full version, and close the experiment.

Condition B: Failure threshold. The metric movement (or lack of it) that disconfirms the belief. If you hit this, you document what the data suggests, kill the experiment, and update your model of the user.

Condition C: Time limit. The date by which you make a call regardless of where the data stands. If you reach this date without a clear signal, the experiment is inconclusive. Inconclusive is a valid result. Run a different version or a different test.

Experiment: [Name]
Runs: [Start date] to [End date]
Success: [Metric] moves [direction] by [threshold]
Failure: [Metric] moves less than [lower threshold] or moves the wrong direction
Time limit: Decision made by [date] regardless of outcome

Teams that don't write this down will extend experiments indefinitely when results are ambiguous, which defeats the whole purpose. The discipline of writing exit conditions in advance is the discipline of treating your experiments like bets, not like open-ended research.

Step 5: Run the Debrief and Write the Learning, Not Just the Result

The experiment ended. Now comes the part most teams skip: the debrief. Not a retrospective on process. A written summary of what the data actually taught you about your users.

The result (metric went up / metric went down) is the least interesting output of an experiment. The learning is the interesting part. And the learning only compounds if it's written down and connected to your next decision.

A debrief is not a deck. It's one page. It has four parts:

1. What we believed: [Paste the original belief statement]
2. What happened: [Data summary, 2-3 sentences]
3. What we now believe: [Updated belief about the user or the problem]
4. What we're doing next: [One sentence on the action this learning drives]

Keep a shared document of these. After ten experiments, you'll have a model of your user that no tool can give you, because it's built from your actual users responding to your actual product. This is compounding knowledge. It doesn't get stale the way a persona document does.

One more thing: share the failure debriefs more than the success ones. Teams that normalize failed experiments accumulate more useful learning faster. The failure told you something true. The success often just confirmed what you already suspected.

Templates 📎

EXPERIMENT BRIEF

Name: [Short, memorable name for this experiment]
Owner: [Person responsible for outcome]
Date created: [Date]
Target run dates: [Start] to [End]

---

BELIEF STATEMENT
We believe that [target user] will [take this action] because [underlying reason].
We'll know we're right if [specific metric] moves by [threshold] within [timeframe].

---

SCORING
Belief clarity: /3
Learning value: /3
Speed to signal: /3
Reversibility: /3
Implementation cost: /3
TOTAL: /15

Priority: [ ] High (12-15)  [ ] Medium (8-11)  [ ] Low (5-7)  [ ] Do not run (<5)

---

EXPERIMENT DESIGN
Type: [ ] Fake door  [ ] Concierge  [ ] Feature flag  [ ] Prototype  [ ] Partial build  [ ] Full A/B
Implementation: [1-3 sentences on what's being built or changed]
Audience: [Who sees this experiment, what % of traffic or which segment]

---

EXIT CONDITIONS
Success threshold: [Metric] moves [direction] by [amount]
Failure threshold: [Metric] moves less than [amount] or goes wrong direction
Time limit: Decision made by [date] regardless of data

---

DEPENDENCIES
Engineering: [What's needed]
Data/tracking: [What events need to be instrumented before launch]
Approvals needed: [If any]

EXPERIMENT DEBRIEF

Name: [Experiment name]
Owner: [Person responsible]
Ran: [Start date] to [End date]
Result: [ ] Success  [ ] Failure  [ ] Inconclusive

---

WHAT WE BELIEVED
[Paste the original belief statement verbatim]

---

WHAT HAPPENED
Metric tracked: [Metric name]
Baseline: [Starting value]
Result: [Ending value or observed change]
Sample size: [Users exposed]
Notes on data quality: [Any instrumentation issues, contamination, anomalies]

---

WHAT WE NOW BELIEVE
[1-3 sentences on what this result tells you about the user, the problem, or the solution.
This is the most important section. Don't just restate the result.]

---

WHAT WE'RE DOING NEXT
[One clear sentence: build it, kill it, run a different version, or wait for more signal]

---

OPEN QUESTIONS
[Any questions this experiment raised that a future experiment could answer]

I have a list of growth experiment ideas. For each one, help me:

1. Rewrite the hypothesis in this format: "We believe [user] will [action] because [reason].
   We'll know we're right if [metric] moves by [threshold] within [timeframe]."

2. Score it on these five dimensions (1-3 each):
   - Belief clarity
   - Learning value (does it answer a roadmap or strategy question?)
   - Speed to signal
   - Reversibility if it fails
   - Implementation cost (inverted: low cost = high score)

3. Recommend the cheapest experiment type that would still produce a valid signal.

4. Flag any hypothesis that isn't ready to score yet and explain what's missing.

Here are the experiment ideas:
[Paste your list]

Don't add ideas I didn't include. Just help me sharpen and score what's here.

Common Pitfalls

Running experiments to validate decisions you've already made.
If the outcome wouldn't change what you ship, it's not an experiment. It's a press release. The belief statement catches this: if "we're wrong" has no consequence, don't run it.

Measuring what's easy to track instead of what actually matters.
Click-through rate is easy to instrument. Whether the user reached their first meaningful outcome is harder. Easy metrics produce confident conclusions about the wrong things.

Starting new experiments before closing old ones.
Every open experiment is cognitive load on your team. It also pollutes your data. Run fewer experiments at higher fidelity instead of a parallel track of half-finished tests.

Calling an inconclusive experiment a success.
"The numbers were slightly up but not statistically significant" is not a success. It's an inconclusive result. Treat it that way: document what you'd need to see, run a cleaner version, or accept that this hypothesis isn't testable at your current scale.

Letting an experiment run past its time limit because the data "almost" looks good.
The time limit exists for exactly this moment. If you extend because you're hoping the trend continues, you've stopped doing science and started doing wishful thinking. Call it, document it, move on.

Skipping instrumentation checks before launch.
More than half of bad experiment data comes from events that weren't firing correctly, not from the experiment itself. Before any experiment goes live, verify that your tracking is working on the exact flow you're testing.

Treating user behavior in an experiment as universal truth.
Experiment results are context-specific. The users who saw your experiment, at that time, in that sequence, responded a certain way. Generalize carefully. The debrief should say "users in this segment" not "users want."

Why We Built This

At ProductOS, we work from a specific belief: the highest-leverage moments in product development happen before a single line of code gets written. Experimentation sits right at that boundary. A team with a clear, scored, well-defined experiment running for two weeks learns more than a team that ships for three months without a hypothesis.

The framework in this guide reflects how we think about the relationship between belief, design, and evidence. Most tools give you a place to track experiments after you've decided to run them. What's missing is the upstream work: choosing the right experiments, scoping them to the cheapest valid version, and closing them cleanly so the learning compounds. That's the gap this covers.

If any of this lands and you want to see it in action, we're at productos.dev. No pressure. The toolkit stands on its own.

If you'd rather have humans plus AI run this for you on a real product today, that's what 1Labs AI does.

Built by Heemang Parmar, Founder & CEO of ProductOS. 10+ years in product, 150+ builds. Also runs 1Labs AI, an AI product development agency.