PaLM: Scaling Language Modeling with Pathways

By 2022, the AI community had a critical question: does scaling continue to deliver? GPT-3 showed impressive results at 175B parameters, but was this a dead end or just the beginning? PaLM answered definitively: scaling to 540B parameters produces qualitatively new capabilities that smaller models simply cannot exhibit.

Think of it like heating water. From 0 to 99 degrees Celsius, water just gets warmer — a smooth, predictable change. But at 100 degrees, something qualitatively different happens: it boils. PaLM showed that language models have similar phase transitions. Tasks where a 62-billion-parameter model scores literally zero suddenly become solvable at 540 billion parameters. Not slightly better — suddenly possible. This was the strongest evidence yet that intelligence-like capabilities can emerge from scale alone.

PaLM's contribution was threefold. First, it demonstrated the Pathways system could efficiently train across 6,144 TPU v4 chips spanning two separate pods — a feat of distributed systems engineering as impressive as the model itself. Second, it introduced architectural innovations (parallel layers, multi-query attention, SwiGLU) that maximized quality per FLOP, squeezing every drop of capability from each computation. Third, it revealed discontinuous scaling improvements — capabilities that appear suddenly at 540B that are absent at 62B, challenging the assumption that progress is always gradual.

Training a 540B parameter model requires enormous compute. PaLM was trained on 6,144 TPU v4 chips organized across two separate TPU Pods, connected through the datacenter network. The Pathways system orchestrated this distributed training, achieving record hardware utilization.

Imagine trying to coordinate an orchestra of 6,144 musicians spread across two concert halls connected only by a phone line. Every musician needs to play in perfect sync, and if even one falls behind, the entire performance suffers. That is the challenge Pathways solved. The chips within each pod communicate via ultra-fast inter-chip interconnect (ICI) at ~1.6 Tbps per chip, but the two pods communicate through the much slower datacenter network (DCN). PaLM's parallelism strategy was designed to minimize cross-pod traffic — sending only what absolutely must be shared, while keeping the heavy computation local.

The result: 57.8% hardware FLOPs utilization, meaning over half of the theoretical peak compute was used productively. For context, most large-scale training runs at the time achieved 30-50%. Every percentage point matters when you are burning through millions of dollars of compute — going from 35% to 57.8% means getting 65% more useful work per dollar. The training took approximately 46 days and consumed roughly 2.5 x 10^24 FLOPs.

PaLM uses a decoder-only Transformer with key modifications designed to squeeze maximum capability from every computation. Each innovation is a bet that pays off more at larger scale: parallel attention and FFN layers (15% training speedup), multi-query attention (faster inference), SwiGLU activation (better quality per FLOP), and RoPE embeddings (relative position encoding).

The most counterintuitive choice is parallel layers. In a standard Transformer, each layer first computes attention, then feeds the result through the FFN — sequentially, like a factory assembly line. PaLM computes both at the same time from the same input. This sounds like it should lose information (the FFN never sees what attention found), and indeed at 8B scale it slightly hurts quality. But at 62B and 540B, the degradation vanishes — suggesting the model has enough capacity to compensate, while the 15% speed gain compounds into weeks of saved training time.

Multi-query attention is equally clever. In standard multi-head attention, each of the 48 heads maintains its own Key and Value projections, which all need to be cached during text generation. PaLM shares a single K and V across all heads, keeping only the Query head-specific. This shrinks the KV cache by 48x during inference, making generation dramatically faster and cheaper with virtually no quality loss. This technique, first proposed in 2019, proved its worth at PaLM's scale and has since become standard in most large language models.

PaLM was trained on a carefully curated mixture of 780 billion tokens spanning web pages, books, Wikipedia, news, code, and social media conversations in over 100 languages. The diversity of this corpus is key to PaLM's versatility — a model can only be as capable as the knowledge it has been exposed to.

The data mix tells a story about design priorities. The largest single component is social media conversations at 50% — this is where the model learns dialogue patterns, informal reasoning, and the messy, contextual way humans actually communicate. Filtered web pages at 27% provide broad world knowledge. Books (13%) supply long-form coherent reasoning. And code (5%) is deliberately included not just for programming ability, but because code training has been shown to improve logical reasoning across all tasks — structured thinking in one domain transfers to structured thinking generally.

An important context: Chinchilla (published around the same time) showed that GPT-3 was undertrained relative to its size, suggesting 175B parameters should see ~3.5T tokens to be compute-optimal. PaLM at 780B tokens for 540B parameters is similarly undertrained by Chinchilla scaling laws. This means PaLM could have been even better with more data — the results we see are a lower bound on what 540B parameters can achieve.

PaLM's most striking finding: certain capabilities appear discontinuously with scale. Tasks where 8B and 62B models score near zero suddenly become solvable at 540B. This provides the strongest evidence yet that scale unlocks qualitatively new abilities.

Consider word unscrambling as an example. Given "uehaotgrs" and asked to rearrange the letters into an English word, the 8B model scores 2% and the 62B model scores 8% — essentially random guessing. The 540B model scores 68%. It did not get slightly better at unscrambling; it went from unable to able. Something in the internal representations crossed a threshold where the model suddenly "understood" letter manipulation. Similar jumps appeared for ASCII art recognition, emoji movie interpretation, and navigate-maze tasks.

On BIG-Bench, a suite of 150+ diverse tasks designed by 450+ researchers to probe capabilities beyond standard benchmarks, PaLM 540B achieved a 5-shot aggregate score of 69.3% — surpassing average human performance of 65.0%. This was the first time any LLM crossed the human baseline on this diverse benchmark, and it happened not through fine-tuning or specialized training, but purely through scale and few-shot prompting.

At 540B parameters, PaLM's few-shot in-context learning matches or exceeds fine-tuned models on many benchmarks. On 28 of 29 evaluated tasks, PaLM surpassed all prior large language models. This is remarkable: you can steer one model to do almost anything just by changing the prompt, without touching a single weight.

Think about what few-shot learning really means. Traditional ML requires you to collect thousands of labeled examples, train a specialized model, tune hyperparameters, and deploy — a process that takes weeks and produces a model that does exactly one thing. With PaLM, you write a prompt with 5 examples, and the model figures out the pattern. It is pattern recognition at its most general: the model has seen so many patterns during training that it can recognize a new one from just a handful of demonstrations. This makes AI accessible — you do not need ML expertise, you just need to explain what you want through examples.

When combined with chain-of-thought prompting, PaLM achieves breakthrough results on reasoning tasks. It solved 58.1% of GSM8K math problems with just 8-shot prompting, outperforming fine-tuned GPT-3 (55%). The combination of scale + reasoning prompts proved to be more powerful than fine-tuning smaller models.

Here is why this matters. Fine-tuning GPT-3 on GSM8K required 7,500 hand-labeled examples of step-by-step math solutions, plus the compute cost of fine-tuning. PaLM achieved a higher score with just 8 examples in the prompt. The difference in effort is three orders of magnitude. What changed? Scale gave the model the internal capacity to reason step-by-step, and chain-of-thought prompting gave it permission to show its work rather than jump to the answer.

Without chain-of-thought, PaLM 540B scored only 17.9% on GSM8K — forcing the model to answer in one step wastes its reasoning capacity. With CoT, the same model scores 58.1%. The reasoning ability was always there; it just needed to be elicited through the right prompting strategy. This insight — that capabilities can be latent until activated by the right interface — became a central theme of post-PaLM research.

PaLM's training data includes source code from GitHub, giving it strong code generation abilities. PaLM-Coder 540B (fine-tuned on additional code) achieved 36% pass@1 on HumanEval, surpassing specialized models like Codex 12B (28.8%). This was notable because PaLM is a general-purpose model that was not specifically designed for code.

The relationship between code and reasoning is bidirectional. Code in the training data improves logical reasoning on non-code tasks (structured thinking transfers), while the model's broad language understanding helps it generate code that matches natural language descriptions. PaLM demonstrated that you don't need a specialized code model — a large enough general model with code in its training mix can outperform purpose-built tools. This finding influenced the field's move toward general-purpose foundation models rather than task-specific ones.

Trained on data in over 100 languages, PaLM demonstrates strong multilingual capabilities. It can translate, answer questions, and reason in many non-English languages, showing that scale benefits transfer across linguistic boundaries.

Perhaps the most surprising multilingual result is cross-lingual transfer. PaLM can sometimes solve tasks in languages with very limited training data by leveraging knowledge learned from high-resource languages like English. Teach it a reasoning pattern in English, and it can apply that same pattern in Bengali or Swahili — even though it saw far less text in those languages. This suggests the model builds language-agnostic internal representations that capture meaning independently of surface form, which becomes more robust at larger scale.

The tokenizer uses a 256K SentencePiece vocabulary with full Unicode coverage, ensuring every language can be represented without falling back to byte-level encoding. This large vocabulary reduces the "token tax" for non-English languages — where each word requires multiple tokens — improving both efficiency and quality for multilingual tasks.

PaLM established that scale continues to unlock new capabilities far beyond what smaller models achieve. It set the template for Google's subsequent AI efforts, leading directly to PaLM 2 (which powers Gemini/Bard) and influencing the entire field.

The paper's influence extends beyond Google. PaLM's architectural choices — parallel layers, multi-query attention, SwiGLU, RoPE, no biases — became a checklist that nearly every subsequent large model adopted. Meta's LLaMA, for instance, uses SwiGLU and RoPE directly from PaLM's playbook. The emergence findings influenced how the entire field thinks about scaling — not just as getting more of the same, but as potentially crossing capability thresholds that enable entirely new applications.