Idea Mining: why good ideas need a mine, not a workshop

Many of the innovation processes I have seen over the years treat ideas as if they fall from the sky. Book a workshop, fill a wall with sticky notes, vote on the favourites, and expect the list to produce something useful to the business. Six months later very little has been built, and nobody is quite sure why. The problem is the mental model underneath the workshop. Ideas are not a renewable crop you simply harvest. The good ones already exist inside a business: in the heads of staff and customers, or implicit in technology that has arrived since the last business plan was written. They do not surface on their own. They have to be deliberately extracted, and that is a job for a mine, not a single field trip.

I developed what I now call Idea Mining in 2018, drawing on years of running innovation within technology businesses of various sizes. It is a way of finding good ideas under uncertainty: ideas that solve real problems for paying customers, not ideas that simply sound exciting in a strategy meeting. The metaphor is deliberate. Mining is patient, structured and repeatable, and it respects the fact that the best material is rarely lying on the surface.

Why mining is the right metaphor

Three features of real mining apply well to the way good ideas surface inside organisations.

The valuable material is mixed in with a great deal of less valuable material. Useful ideas rarely arrive labelled as such. They sit alongside half-formed observations, dead-end suggestions, employee complaints, slightly modified repeats of last year's failed initiative, and the occasional pet project from someone senior. You don't recognise an idea as valuable until you have extracted it from that context and held it up against the alternatives that came out of the same process.

A mine is different from a sample. A one-off ideation workshop is, in mining terms, a sample. It produces a handful of interesting fragments, then closes, and the business goes back to what it was doing. A mine is structural. It is a recurring process with a known cadence, a permanent invitation that staff and customers come to recognise, and a place where they know an observation will be taken seriously rather than politely noted. How often it should run depends on the speed of the market: faster-moving markets need more frequent cycles.

Old seams sometimes become viable again under new conditions. An idea that was not commercially feasible five years ago may be feasible now. A new technology has arrived, a manufacturing process has matured, a cost curve has fallen, or customer attitudes have shifted in ways nobody predicted. In my experience, revisiting old seams is one of the highest-leverage moves an innovation function can make. It is also one of the most consistently neglected, mostly by businesses that treat ideation as an occasional event rather than a continuous discipline.

The three stages

The framework has three stages. Each has its own inputs, a distinct kind of output and a different team. The output of one stage is the input to the next, and the cycle can run as quickly or as slowly as the surrounding environment requires.

What feeds the system

The mind-map below shows the wider ecosystem of inputs that feed each stage: competitor activity, customer feedback, the technology available at any given moment, market changes, and threats appearing on the horizon. The framework depends on those inputs being continuously refreshed rather than left to go stale. Idea Mining is only ever as good as the information flowing into it from the surrounding business.

How it fits with the other frameworks I use

Idea Mining is not a standalone framework. It is most useful inside a wider set of disciplines, and three in particular shape how I apply it.

Alongside Cynefin

The Cynefin framework tells you what kind of problem you are mining for, and different problem zones call for different mining responses. Complex problems, the unknown unknowns, are best approached through what I call probe mining: small, safe-to-fail experiments designed to surface emergent patterns rather than deliver a definitive answer. Complicated problems are known unknowns, and respond better to sense mining: expert analysis applied to existing data and known options. Obvious problems want options mining, where the team chooses among well-understood alternatives rather than inventing new ones. Chaotic problems also want options mining, but at speed, with the analysis arriving after the response has been chosen.

Real strategic questions are almost always a mixture of these. Idea Mining gives you the muscle to respond. Cynefin tells you which muscle to use.

Alongside Conscious Collaboration

Idea Mining is the productive activity that runs inside Conscious Collaboration. Conscious Collaboration describes the structured process, shared mindset and supporting technology that let a diverse team work together effectively. Idea Mining describes what that team should be doing while it works together. Without Conscious Collaboration, the mining session reliably collapses into whatever the dominant voice wants to talk about. Without Idea Mining, Conscious Collaboration tends to produce well-managed conversations that never lead anywhere actionable.

Alongside Black Swan theory

Nassim Taleb's observation that the most consequential events are rare, unpredictable and obvious only in hindsight can look like a reason not to plan at all. In my reading it is nothing of the kind. It argues against a narrow form of planning: forecasting that quietly assumes the future will look like the past. Idea Mining is a useful counter-move because it is proactive. It deliberately extracts a wide range of possible responses to a wide range of possible futures while the organisation still has room to think about them carefully. When a Black Swan arrives, whether a financial crisis, a pandemic or a sudden technology shock, the business that has been mining has options to choose from. The business that has not is improvising under pressure with whatever it has to hand.

The pitfall to watch for: ego

In my experience, the commonest failure of Idea Mining is the ego of one or two individuals quietly dominating the mine. This is not limited to senior people. It can happen at any level where one voice carries more weight than the others. The ego's need for recognition leads it to put its own ideas ahead of others, regardless of merit. The result is the worst of both worlds: a process that looks like structured ideation, producing an output that reflects what one person wanted to do anyway.

The defence is structural, not personal. Run the mine inside a Conscious Collaboration shell, with explicit listening, structured turn-taking, a deliberate separation between generating ideas and evaluating them, and a norm of surfacing assumptions before they harden into convictions. Treat dissent as a feature of the process, not a problem to be managed away.

Where I use it

I most often use Idea Mining as a CTO or interim leader, brought into a business that has got itself stuck. Stuck businesses almost always have plenty of ideas below the surface. The staff know what the problems are, the customers complain about the right things, and the relevant technology is waiting to be used. But the mine no longer exists, or has quietly collapsed under the weight of operational demands.

The first job is usually to rebuild it: a recurring rhythm, a properly cross-functional team, and a clear boundary between the opportunity, idea and smelting stages. Within a few cycles the stuckness tends to lift, because the organisation can finally see what it has to work with, and the leadership team can make deliberate choices instead of improvising in response to whatever lands on their desk.

It is not a complicated framework. The discipline lies in running it consistently over time, especially when something more urgent is always demanding the team's attention.

Four things worth taking seriously

For boards: ask whether the business has a mine or just holds workshops. A recurring cadence that staff and customers recognise is worth more than any single ideation event.

For founders: decide where to look before deciding what to build. Skipping Opportunity Mining is the fastest route to solving problems nobody is paying for.

For product and innovation leaders: smelt, don't just filter. Revisit old seams and try changing the customer, the timing or the price point before discarding an idea.

For anyone facilitating an ideation session: the defence against a dominant ego is structure, not goodwill. Separate generating from evaluating, and treat dissent as part of the process.

I would be interested to hear which ideas your organisation set aside a few years ago that might be worth mining again under today's conditions.

Related: Conscious Collaboration · Cynefin and Conscious Collaboration · all writing