Why the best ideas often fail in testing (and how to change that).

Many product managers and their research colleagues have a love-hate relationship with concept testing. When done well, it can flag weak ideas early, validate promising directions, and inform go-to-market plans. But too often, it fails to do the very thing stakeholders expect of it: tell us whether a concept has real potential in-market.

The problem isn't with concept testing itself. It's with who we're testing with—and how.

At CoinJar Insights, we believe the solution lies in involving a different kind of respondent earlier and more systematically: the Catalytic Customer.

The Limits of “Average” Feedback

Traditional concept testing frameworks—monadic, sequential monadic, protomonadic—are built to quantify consumer response to new ideas. They help us compare concepts, understand appeal, relevance, uniqueness, and purchase intent. But the underlying assumption is that better performance on these metrics correlates with market success.

Seasoned practitioners know it’s not so simple.

Concepts that test well often underperform. And ideas that test poorly sometimes go on to win in-market. Why? Because traditional tests are good at capturing mainstream reactions, but poor at identifying whether a concept speaks to emergent needs, solves unsolved problems, or taps into latent motivations.

These signals are often drowned out in general population samples—or missed entirely by respondents who lack the context or motivation to engage deeply with the ideas being shown.

Who Are Catalytic Customers?

Catalytic Customers are a distinct segment of highly engaged, knowledgeable, and forward-looking users within a category. They are not early adopters in the classic Rogers curve sense. Nor are they influencers in the social media sense.

They are more like “category stewards”—individuals with:

• Deep, lived experience in the category

  
  

• A strong point of view on what works and what doesn’t

  
  

• High expectations and an eye for detail

  
  

• An appetite for improvement, rather than mere novelty

  
  

They tend to be critical but constructive. Not only can they tell you what’s off, but often why, and what might make it better.

Think of the urban cyclist who’s ridden every brand of commuter bike and can critique your folding mechanism. The edtech teacher who beta-tests every classroom tool before rolling it out to students. The skincare enthusiast who reads ingredient labels like a chemist.

These people don’t just react. They interrogate.

What Catalytic Customers Add to Concept Testing

When you embed Catalytic Customers into your concept testing workflow—whether qualitative or quantitative—you start to surface deeper insight. Not just what people like or dislike, but why the idea matters, what it’s missing, and what it could unlock if improved.

Here’s what they bring to the table:

• Signal Amplification:They detect value where others don’t—especially in early-stage ideas that are still rough or unfamiliar. Their feedback helps identify concepts that may not perform well in general testing but have strong potential if refined.

  
  

• Friction-Finding:They spot functional or experiential barriers that generic respondents gloss over. Their critiques help teams refine execution and anticipate adoption hurdles.

  
  

• Future Orientation:They evaluate ideas not just in terms of current habits, but in terms of where the category is heading. This makes them especially useful for testing innovative or disruptive concepts.

  
  

• Insight-Rich Feedback:Their comments often contain the seeds of next-generation features, messaging angles, or entirely new use cases—offering more than a thumbs-up or down.

When and How to Use Them

We’re not advocating for abandoning broader samples or normative databases. Catalytic Customers don’t replace traditional respondents—they complement them.

But we do recommend involving them earlier in the concept lifecycle—especially in the exploration and refinement phases before broader validation.

Tactically, this might look like:

• Pre-quantitative in-depth interviews with Catalytic Customers to sharpen concepts before fielding

  
  

• Augmenting quant tests with a booster sample of Catalytic Customers to analyze response deltas

  
  

• Creating "Catalyst Panels" that provide ongoing feedback across development sprints

  
  

• Using open-ended diagnostics or co-creation formats to tap their insight-rich thinking

  
  

Importantly, identifying Catalytic Customers takes some legwork. They’re not always visible via traditional targeting criteria. It requires behavioral signals, niche forums, expert referrals, and sometimes a qualitative screener with teeth.

But the effort is worth it. When you start thinking with them—not just about them—your ideas get sharper, bolder, and more aligned with where the category is going.

Closing Thought

For insight leaders pushing innovation agendas, Catalytic Customers are a hidden asset. They don’t just test ideas—they make them better. They force clarity, pressure-test assumptions, and often illuminate a path forward when a concept is stuck in the “meh” middle.

At a time when many teams are rethinking how they test and learn, bringing Catalytic Customers into the process isn’t just a novel tweak. It’s a strategic shift that can make the difference between incremental ideas and truly catalytic ones.

Would you like help identifying Catalytic Customers in your category—or testing a concept that needs sharper feedback? Let’s talk.