sciencebriefs
13:00in productionCh. 1 · Attention instead of recurrence/ 13:00 · ceiling 15 min
Computing & AI

Transformer (deep learning)

2017

A 2017 Google paper argued attention alone, without the recurrent processing every prior language model relied on, could handle sequences better and in parallel, and that architectural bet underlies essentially every large language model built since.

The 2017 paper Attention Is All You Need, from a team of researchers at Google, proposed the transformer architecture, which replaced the step-by-step recurrent processing used by earlier language models with a self-attention mechanism letting every part of an input be processed against every other part at once, making the whole computation far more parallelisable on modern hardware. That parallelism is a major reason transformers could be scaled up to the very large models trained since, including BERT in 2018, the GPT series beginning that same year, and the wave of systems from GPT-3 in 2020 through ChatGPT's public release in late 2022 and beyond. These large language models are typically built in two stages, an initial pretraining phase in which the model learns to predict the next word across enormous quantities of text, followed by a fine-tuning phase, often using human feedback, that shapes the model into a more instructable assistant. Even as capabilities have grown substantially, well-documented limitations persist, including confidently stated factual errors known as hallucinations, weaknesses in multi-step planning and reasoning, and reliance on human-selected training data whose biases the models can reproduce, and researchers remain divided over how much of what these models do reflects genuine understanding versus sophisticated pattern recombination.

Computing & AIArtificial intelligenceMachine learningComputingArtificial intelligence across the network →
Chapters & takeaways6
  1. 0:08
    Attention instead of recurrence

    The 2017 transformer paper proposed replacing step-by-step recurrent processing with self-attention, letting every part of a sequence be compared to every other part at once.

  2. 2:10
    Why parallelism mattered so much

    Removing recurrence made the computation far more parallelisable on modern hardware, a major reason transformers could be scaled up to very large models.

  3. 4:20
    From BERT to ChatGPT

    The architecture underpinned BERT in 2018, the GPT series from the same year onward, and the wave of systems from GPT-3 through ChatGPT's late 2022 public release.

  4. 6:30
    How these models are actually trained

    Large language models are typically pretrained to predict the next word across enormous amounts of text, then fine-tuned, often with human feedback, into more instructable assistants.

  5. 8:40
    The limitations that have not gone away

    Confidently stated factual errors, weaknesses in multi-step reasoning, and biases inherited from training data remain well-documented limitations even in the most capable systems.

  6. 10:50
    An open argument about what is actually happening

    Researchers remain divided over whether these models' more impressive outputs reflect genuine understanding or sophisticated recombination of existing text.

Worth your time?

Yes. Study the whole thing.

4/ 5
What works
  • identifies the specific 2017 architectural innovation rather than treating recent AI progress as an undifferentiated blur
  • separates pretraining from fine-tuning clearly, since the two stages do different jobs
  • documents real limitations, hallucination and reasoning gaps among them, rather than presenting the technology as unqualified progress
What does not
  • does not resolve the genuine, unsettled debate over whether these models understand language or only recombine it
  • cannot guarantee factual reliability, since hallucination remains a documented and unresolved limitation
Study it if
  • readers who want to know what specific architectural change made today's language models possible
  • anyone curious how these systems are actually trained in two distinct stages
  • people who want the limitations stated plainly rather than glossed over
Skip it if
  • readers wanting a settled verdict on whether large language models genuinely understand language
The written brief3 min read

Attention instead of recurrence

The specific claim behind this technology is narrower than it might seem: a particular way of processing sequences of text turned out to work better, and to scale up more easily, than the methods that came before it. Earlier neural network approaches to language processed text step by step, one word at a time, in a chain where each step depended on the one before it, an approach known as recurrence. The 2017 paper Attention Is All You Need, from researchers at Google, proposed instead a self-attention mechanism, letting the model compare every part of an input sequence with every other part directly and simultaneously, rather than passing information along a sequential chain.

Why parallelism mattered so much

The practical significance of removing recurrence was less about elegance and more about hardware. A sequential, step-by-step computation cannot easily be split across many processors working at once, since each step needs the result of the previous one, but a computation built around comparing all parts of a sequence to each other simultaneously can be parallelised extensively on the graphics processing units that power modern machine learning. This parallelism is a major reason the transformer architecture could be trained at a scale, in terms of both data and model size, that earlier recurrent architectures struggled to reach in practice.

From BERT to ChatGPT

Following the original 2017 paper, transformer-based models proliferated quickly. BERT, released in 2018, used the architecture for understanding rather than generating text, and the GPT series, beginning around the same time, used it to generate text by predicting what should come next. Successive GPT models grew substantially larger through systems including GPT-3, released in 2020 and initially accessible only through a limited interface, culminating in ChatGPT’s public release in late 2022, which brought this style of model to a mass general audience for the first time and prompted rapid, widespread public adoption of the technology.

How these models are actually trained

Training one of these large language models typically happens in two distinct stages. The first, pretraining, involves exposing the model to enormous quantities of text and training it to predict the next word given what came before, a task that requires no manually labelled data and can draw on vast amounts of existing text. The second stage, fine-tuning, adjusts the pretrained model’s behaviour, often using techniques that incorporate human feedback on which outputs are more helpful or appropriate, turning a raw next-word predictor into something closer to a usable, instructable assistant.

The limitations that have not gone away

Even as these systems have grown more capable, certain limitations have persisted across model generations rather than disappearing with scale. Hallucination, the tendency to state incorrect information confidently and fluently as though it were fact, remains a documented problem across large language models. Researchers have also identified persistent weaknesses in multi-step planning and certain kinds of reasoning, and because these models learn from human-generated text, they can reproduce biases and errors present in that training data, a limitation tied directly to the data-driven nature of how they are built rather than a simple engineering oversight likely to vanish with more computing power alone.

An open argument about what is actually happening

Underlying all of this is a genuine, unresolved scientific and philosophical disagreement about what these models are actually doing when they produce fluent, often useful text. Some researchers argue that certain capabilities, particularly around structured reasoning tasks, indicate a real, if different from human, form of understanding, while others maintain that the models are fundamentally recombining patterns present in their training data without anything that should be called comprehension. This is worth taking seriously precisely because it remains unsettled: the practical usefulness of these systems is not in question, but what that usefulness actually tells us about machine understanding is a live argument rather than a closed one.

Same field · Computing & AI4 of 46
10:48
Ada Lovelace
10:30
Alan Turing
10:16
AlexNet
9:31
Arieh Warshel
Up next in Science

Transposable element

· 13:00

Barbara McClintock's 1940s maize experiments revealed that genes can change position within a genome, a finding dismissed for decades until molecular biology confirmed it and belatedly earned her an unshared Nobel Prize.

Play nextBack to Computing & AI
13:00