sciencebriefs
10:18in productionCh. 1 · Backpropagation wasn’t born here—but it grew up here/ 10:18 · ceiling 15 min
Computing & AI

Neural network (machine learning)

Geoffrey Hinton and others

Hinton didn’t build today’s AI—he built the grammar that let it speak.

Geoffrey Hinton and colleagues established core methods—backpropagation for representation learning, and Boltzmann machines for unsupervised deep generative modelling—between 1985 and 1995. Their work was grounded in statistical mechanics, experimentally verified on small-scale networks, and explicitly recognised as foundational for machine learning with artificial neural networks. It did not deliver production AI, nor claim to.

Computing & AIArtificial intelligenceMachine learningAlgorithmsArtificial intelligence across the network →
Chapters & takeaways4
  1. 0:54
    Backpropagation wasn’t born here—but it grew up here

    Backpropagation was popularised—not invented—by Hinton, Rumelhart and Williams in 1986, and their experiments proved multi-layer networks learn useful internal representations.

  2. 2:35
    Unsupervised learning, grounded in physics

    From 1985–1995, Hinton and colleagues designed probabilistic neural architectures inspired by statistical mechanics to learn generative models without labels.

  3. 4:27
    The Boltzmann machine: the first deep generative architecture

    The Boltzmann machine—co-invented by Hinton in 1985—was cited as a foundational invention enabling machine learning with artificial neural networks.

  4. 6:22
    What they delivered: deep generative models, not classifiers

    These architectures were designed for unsupervised learning of deep generative models—and that is what they did.

Worth your time?

Yes. Study the whole thing.

4/ 5
What works
  • enabling unsupervised deep generative modelling
  • popularising backpropagation for multi-layer networks
  • proving internal representation learning in experiments
  • linking neural computation to statistical mechanics
What does not
  • deliver scalable inference
  • show real-world deployment
  • establish consensus on training stability
  • provide error bounds or confidence intervals
Study it if
  • historians of computing
  • AI researchers tracing generative model lineage
  • physics-informed ML practitioners
Skip it if
  • engineers building LLM pipelines
  • policy analysts assessing AI risk
  • clinicians evaluating diagnostic AI
The written brief1 min read

What the work claims

That neural networks can learn internal data representations via backpropagation, and that probabilistic architectures inspired by statistical mechanics—like the Boltzmann machine—can perform unsupervised learning of deep generative models.

How it was done

Hinton, Sejnowski, Dayan, Rumelhart and Williams developed neural network architectures—including the Boltzmann machine (1985), restricted Boltzmann machine, Helmholtz machine, and wake-sleep algorithm—using principles from statistical mechanics. They applied backpropagation to multi-layer networks in experiments showing internal representation learning.

What holds up

The Boltzmann machine and wake-sleep algorithm enabled unsupervised learning of deep generative models. Backpropagation experiments confirmed multi-layer networks can learn useful internal representations. These are foundational methodological contributions—not speculative claims.

What does not

The work does not establish that these models generalise reliably across domains. It does not show robustness to noise, scalability to large datasets, or compatibility with modern hardware. It does not claim or demonstrate real-time inference, classification accuracy benchmarks, or integration into production software.

Why it matters beyond the lab

These methods underpin later advances in generative AI, but only as conceptual ancestors. The 2024 Nobel Prize in Physics recognised their role in enabling machine learning with artificial neural networks—not their direct application to physics problems.

Is it worth your time

Yes—if you need to understand how unsupervised deep generative models originated, or why backpropagation became central to neural network training. Not if you seek scalable, real-world AI systems: none of these architectures were deployed at scale in the 1980s–90s.

Same field · Computing & AI4 of 46
13:00
Perceptron1957Frank Rosenblatt, working at Cornell Aeronautical Laboratory, built the perceptron, a machine that adjusted its own connection weights from labelled examples rather than being explicitly programmed. First simulated on an IBM computer in 1957 and then built as physical hardware, the Mark I Perceptron, by 1960, it drew wildly overstated press coverage about machines that would soon think and reproduce themselves. In 1969, Marvin Minsky and Seymour Papert published a proof that a single layer of the kind Rosenblatt had built could not learn certain patterns, most famously exclusive-or, and although their book acknowledged that multiple layers could solve the same problems, the narrower message was widely read as a verdict against neural networks generally, contributing to roughly a decade of reduced funding and interest before the approach was revived with multilayer networks trained by backpropagation.
10:48
Ada Lovelace1815–1852Ada Lovelace’s 1843 notes established the first published algorithm for a general-purpose computing machine, articulated symbolic manipulation beyond arithmetic, and set an early boundary on machine agency — all without hardware execution.
13:00
Transformer (deep learning)2017The 2017 paper Attention Is All You Need, from a team of researchers at Google, proposed the transformer architecture, which replaced the step-by-step recurrent processing used by earlier language models with a self-attention mechanism letting every part of an input be processed against every other part at once, making the whole computation far more parallelisable on modern hardware. That parallelism is a major reason transformers could be scaled up to the very large models trained since, including BERT in 2018, the GPT series beginning that same year, and the wave of systems from GPT-3 in 2020 through ChatGPT's public release in late 2022 and beyond. These large language models are typically built in two stages, an initial pretraining phase in which the model learns to predict the next word across enormous quantities of text, followed by a fine-tuning phase, often using human feedback, that shapes the model into a more instructable assistant. Even as capabilities have grown substantially, well-documented limitations persist, including confidently stated factual errors known as hallucinations, weaknesses in multi-step planning and reasoning, and reliance on human-selected training data whose biases the models can reproduce, and researchers remain divided over how much of what these models do reflects genuine understanding versus sophisticated pattern recombination.
10:48
Shakey the robotShakey the robot was the first general-purpose mobile robot capable of reasoning about its own actions. Developed at Stanford Research Institute’s Artificial Intelligence Center, it used LISP and the STRIPS planner to break high-level commands into physical actions—travelling, flipping switches, opening doors, climbing, pushing objects. It produced A*, the Hough transform, and the visibility graph method. It did not learn, adapt, or operate in real time. Its significance lies not in performance but in proving that logical planning could drive physical action—a conceptual foundation for AI and robotics.
Up next in Science

Hydrothermal vent

· 13:00

A 1977 dive to the Galapagos Rift found an ecosystem running entirely without sunlight, and a graduate student's follow-up work explained why, which this brief traces from that dive through the chemistry that still feeds tube worms without mouths.

Play nextBack to Computing & AI
13:00