How Innovation Works

Why do improbable things—like smartphones, airplanes, and antibiotics—exist at all? Matt Ridley’s How Innovation Works argues that innovation is the process by which energy and human cooperation bring improbable order into the world. Just as life locally defies entropy through energy use, innovation harnesses energy, knowledge, and institutions to create new arrangements that improve life. Ridley’s central claim is that innovation is not magic or genius; it is a collective, iterative process of turning energy and ideas into value through trial, error, and diffusion.

From Entropy to Improbability

Ridley starts by grounding innovation in physics: every orderly artifact is a local victory over entropy achieved with energy. A light bulb, an iPhone, or a DNA molecule is improbable compared with random disorder, made possible because energy was expended to hold it in that configuration. Ridley invites you to see technology as part of nature’s continuum—just as life uses sunlight to organize molecules, civilization uses fossil fuels, electricity, and now data to order the world around you. This physical grounding reframes creativity as an energy-driven recombination of ideas, not spontaneous spark.

The Process: Trial, Error, and Recombination

Across the ages, every genuine breakthrough arises from hundreds of attempts, not one leap. Edison’s 6,000 filament tests, Watt’s endless tinkering, and the Wright brothers’ glider iterations all show how innovation accumulates incrementally through “variation and selection.” Ridley emphasizes recombination—borrowing old components and joining them in novel ways—as the main source of novelty. The smartphone exemplifies this: microprocessors, batteries, touch interfaces, and connectivity were each old technologies that, recombined, transformed the world. Innovation, like genes, evolves by recombining successful parts, not by inventing from nothing.

The Social Nature of Genius

Just as no organism evolves in isolation, no innovation is born in solitude. Ideas thrive where people exchange knowledge, imitate, and cooperate. From Edison’s Menlo Park team to Borlaug’s global crop network, Ridley shows that innovation is a team sport involving inventors, adopters, regulators, and investors. Adoption—the moment an invention becomes practice—is the true mark of success. You may picture Newton or Einstein as lone geniuses, but even they built upon dense webs of conversation, tools, and prior inquiry. Ridley’s “collective brain” concept echoes thinkers like Kevin Kelly and Joe Henrich: progress is cumulative social learning on an ever-expanding scale.

Institutions and the Freedom to Tinker

For improbable things to appear, you need institutions that tolerate failure and allow energy, capital, and ideas to flow. Ridley argues that open societies—those with secure property rights, minimal barriers to experimentation, and cultural tolerance for error—consistently innovate more. Examples abound: the decentralized networks of 18th-century Britain, the American frontier’s entrepreneurial laws, and Silicon Valley’s forgiving venture ecosystem. Conversely, excessive regulation and risk aversion throttle the iterative discovery process. The repeated contrast between nuclear stagnation and shale gas diffusion illustrates how permissionless small-scale experimentation drives down cost and raises learning speed.

Energy, Freedom, and the Future

Ridley’s book ultimately presents innovation as civilization’s “infinite improbability drive.” Energy fuels it, error refines it, and freedom sustains it. Whether in prehistoric tools, steam engines, vaccines, or digital code, the same logic prevails: dense exchange, abundant energy, and tolerant institutions make complex innovations inevitable. If you want more progress and fewer crises, Ridley suggests, don’t worship genius or impose top-down control—build conditions where minds can mingle, materials can be tested, and failure is survivable. Innovation is the physics of progress translated into human form.

Ridley begins his history deep in prehistory, showing that innovation is not recent but ancient. Fire, farming, and animal domestication were not single moments of invention but long, overlapping processes that flourished when climates stabilized and human groups grew larger and denser. In the Pleistocene’s volatile cold, the cost of experimentation was high; when the Holocene brought stable warmth and predictable rainfall, agriculture and pottery emerged independently worldwide. Innovation, he argues, concentrates where resources are abundant and communication is easy.

Social Density and the Collective Brain

You innovate best in crowds, not solitude. Archaeologist Curtis Marean’s excavations at Pinnacle Point show that coastal abundance allowed ancient humans to settle, specialize, and share tricks. This density created a “collective brain”—Ridley’s preferred metaphor for civilization’s innovation engine. Joe Henrich’s studies confirm that small or isolated populations, such as prehistoric Tasmania, lose complex technologies when social links thin. The lesson: innovation scales with the network of minds in communication.

Genes That Follow Culture

Cultural innovations can drive biological evolution. Dairying selected humans for lactase persistence, while domestication softened wolves into dogs—and humans into less aggressive, more cooperative beings (Richard Wrangham’s “domestication syndrome”). Innovation thus loops between technology, society, and biology. Ridley shows that civilization itself has biologically rewired you to be more collaborative and inventive across generations.

Abundance as a Catalyst

The book connects environmental stability, population density, and exchange as the preconditions for every technological leap. From prehistoric coastal foragers to Renaissance merchants to modern tech clusters, the pattern repeats: environments rich enough to support specialization and trade also generate innovation. You cannot program invention from scarcity alone—you need energy, communication, and a community large enough to remember and recombine ideas.

Every great surge of innovation follows a breakthrough in harnessing energy. Ridley calls the transition from heat to work—the steam engine to turbine to nuclear—the hinge of modernity. Once humans learned to apply combustion’s random heat for predictable mechanical motion, productivity exploded, and with it came the modern world.

Steam and the Birth of Industry

Denis Papin, Thomas Savery, and Thomas Newcomen pioneered early steam engines between 1690 and 1712, building on each other’s ideas. James Watt’s separate condenser and partnership with Matthew Boulton then made steam economical, powering the mine pumps, mills, and locomotives that defined the first Industrial Revolution. Ridley’s history reveals innovation as collective, not heroic; even Watt’s design evolved within a web of shared notes and licensed improvements through communities like Lean’s Engine Reporter.

Turbines, Electricity, and Modular Progress

Charles Parsons’s 1884 turbine turned steam into electricity efficiently enough to light cities and drive ships. The principle of modularity—small, repeatable units—let engineers learn by doing and drive costs down. Later, the shale gas revolution of the 2000s mirrored this: George Mitchell and Nick Steinsberger’s thousands of drilling experiments unlocked vast new energy supplies. Cheap modular experimentation allows innovation; centralized, once-only projects do not.

When Innovation Stalls

Nuclear power illustrates “disinnovation”: too large, regulated, and politicized to permit trial and error. Many promising reactor types—liquid-salt, thorium, fast breeders—remained unrealized because each build was bespoke and burdened by red tape. Ridley contrasts this with the agile pace of open markets where many actors iterate. His conclusion is stark: the more decentralized and experimental an energy technology is, the faster it evolves and the cheaper it becomes.

Public health and agriculture show innovation’s moral power: experimentation, courage, and diffusion save billions of lives. Ridley tells parallel stories—of vaccines, sanitation, antibiotics, fertilizers, and genetic engineering—to reveal how luck meets persistence and how society determines adoption.

Discovery and Serendipity

Lady Mary Wortley Montagu’s 1717 promotion of variolation came from firsthand Ottoman observation, while Edward Jenner’s cowpox inoculation succeeded before germ theory existed. Later innovators—from John Leal’s secret chlorination of Jersey City water in 1908 to Alexander Fleming’s accidental penicillin mold—show that serendipity requires courage to act. True innovation happens when curiosity meets willingness to test under uncertainty.

Scaling Health Technologies

You see the same learning loop in vaccines and antibiotics: Kendrick and Eldering’s whooping-cough trials leveraged community trust, while Fleming’s discovery needed Florey, Chain, and Heatley’s engineering to mass-produce penicillin during WWII. Modern malaria nets and vaping policies illustrate that even brilliant ideas demand enabling regulation to spread safely. Ridley argues that public health depends less on genius than on systems of testing, verification, and adoption.

Feeding the World

Food innovation pivots on chemistry and genetics. Haber and Bosch’s nitrogen fixation (“bread from air”) turned scarcity into abundance, while Borlaug’s adaptation of Japanese dwarfing genes prevented famine in the 1960s. Genetic engineering and CRISPR continue the theme: recombination guided by data. Yet regulation often slows lifesaving biotech—EU and activist resistance to GMOs and Golden Rice delayed global benefits. Ridley insists that moral panic about new seeds or methods must be balanced against the moral cost of hunger.

The Shared Formula

Across medicine and food, the pattern repeats: serendipitous insight + experimentation + social uptake = transformation. Whether chlorine, penicillin, or Bt cotton, innovations succeed when institutions permit visible trials and trustworthy diffusion. Your takeaway is that safety and progress require balance—precaution without paralysis, experimentation without fraud.

Ridley’s tour of transport and communication shows innovation as an evolutionary chain—each system recombines old parts, scales through networks, and transforms society by reducing friction and distance. The railway, telegraph, and internet embody the same principle: connect more people faster, and invention accelerates everywhere.

From Steam to Flight

George and Robert Stephenson’s early locomotives were incremental marvels—fire-tubes, better rails, blast-pipes—culminating in the global railway boom. Screw propellers, internal combustion engines, and aviation followed the same learning-by-doing path. Henry Ford’s Model T and Orville and Wilbur Wright’s 1903 flight demonstrate how persistence and iteration turn experiments into mass adoption. Deregulated airlines and new safety systems later showed that competition plus shared learning can make travel safer and cheaper simultaneously.

Communications Collapse Distance

Morse’s telegraph (1844), Bell’s telephone, and Marconi’s wireless built an ever-denser web of exchange. These systems, amplified by computing, multiplied collective intelligence. Ridley emphasizes that computing’s story is similarly recombinant: Turing’s logic, Shannon’s circuitry, and hardware pioneers like Mauchly and Eckert together laid the path to digital abundance. Gordon Moore’s insight—that transistor counts double predictably—gave the industry a roadmap for perpetual improvement, driving today’s digital society.

AI as the Latest Phase

Artificial intelligence epitomizes cumulative innovation. Algorithms from the 1980s, GPUs from gaming, and petabytes of new data combined to produce deep learning’s leap—from AlphaGo’s creative move 37 to modern medical diagnostics. Yet with power come new challenges: bias, opacity, and trust. Ridley frames AI not as alien intelligence but as collective pattern recognition running on ever-faster hardware. The lesson: every communication revolution raises both capability and responsibility. Transparency and human oversight remain core to sustainable progress.

The final chapters connect innovation’s past to its uncertain future. Ridley warns that culture and policy can either accelerate or stifle progress. Innovation thrives on what he calls “permissionless evolution”—a social ecology of freedom, transparency, and decentralized trial and error. When regulation hardens into precaution, or when incumbents capture the state, stagnation follows.

Markets, Government, and Risk

Economic theory once assumed diminishing returns, yet innovation reveals increasing ones: ideas beget more ideas. Paul Romer’s endogenous growth theory formalized this, while Ridley debates Mariana Mazzucato’s claim that the state drives technological leaps. His view is pragmatic: governments can fund basic research, but top-down direction and patent thickets often block diffusion. Watt’s restrictive patents slowed engine improvements; modern patent wars waste billions. The best policy is openness: protect property, but promote competition and experimentation.

Resistance and Moral Panic

Every innovation—from coffee to margarine to GMOs—meets opposition. Ridley documents how moral panic, vested interest, or fear of novelty delay progress. The precautionary principle, by demanding proof of zero risk, locks in older dangers. Golden Rice’s decade-long delay cost lives to vitamin-A deficiency. The wiser path is comparative risk assessment: measure harms of action and inaction together.

Fraud, Failure, and the Innovation Famine

Innovation requires failure—but not deceit. From Theranos’s fraud to Google X’s honest flops, Ridley insists that societies must draw the line between experiment and deception. Overregulate and you freeze genuine creativity; fail to punish fraud and you erode trust. Today’s slowdown in startups, rising corporate concentration, and heavy compliance hint at an “innovation famine.” The cure, Ridley contends, is moral and institutional: reward curiosity, tolerate transparent failure, and keep the doors of exchange open.

The Freedom Imperative

Ridley closes with optimism grounded in realism: innovation is a renewable human process, but it needs air to breathe. Where people can dissent, test, and trade, the collective brain expands; where fear and bureaucracy dominate, it shrinks. Freedom—to experiment, to err, to recombine ideas—is civilization’s energy source. If you want more progress, defend that freedom—and then, improbably but inevitably, progress will resume.