Perceiver & Perceiver IO

Transformers revolutionized NLP with self-attention, but they have a fundamental scaling problem: self-attention costs O(N^2) in sequence length. For text with 512 tokens, that is only 262,144 attention entries — perfectly manageable. But real-world perception operates at a completely different scale. A 224x224 RGB image is 50,176 pixels. Two seconds of audio at 16kHz is 32,000 samples. A 10-second video clip at 224x224 is over 1.5 million tokens. Running self-attention on these inputs would require terabytes of memory and astronomical compute.

Vision Transformers (ViT) found a clever workaround: split images into 16x16 patches, reducing 50K pixels to just 196 tokens. This works brilliantly for images, but it is a modality-specific hack. It throws away per-pixel detail, it does not work for audio, it does not work for video, and it does not work for point clouds. Every new modality needs its own tokenization trick. The Perceiver asks a more fundamental question: can we build one architecture that handles any input size, for any modality, with no modality-specific engineering?

The answer is yes — but it requires rethinking what attention is for. Instead of having the input attend to itself (which scales quadratically with input size), the Perceiver introduces a small learned latent array that reads from the input via cross-attention. This decouples the processing cost from the input size entirely. The input can be 50K pixels, 100K audio samples, or 1M video frames — the processing cost stays the same because it depends only on the small latent array size.

The Perceiver's core innovation is elegantly simple: a learned latent array of M tokens (typically 256-1024, often 512) that cross-attends to the input of N tokens. In this cross-attention, the queries come from the small latent arrayand the keys and values come from the large input. This costs O(N * M) instead of O(N^2). Since M is fixed and much smaller than N, the cost is linear in input size.

Think of it like a small committee of experts (the latent array) interviewing a large crowd (the input). Each expert asks their own questions (queries) and listens to everyone in the crowd (keys/values). The committee has 512 members, so you only need 512 sets of questions — regardless of whether the crowd has 50,000 people (image pixels) or 100,000 (audio samples). The cost depends on the committee size, not the crowd size.

Crucially, the latent array is learned from scratch during training — it is not derived from the input. Each latent token starts as a learned embedding and gets populated with information from the input through cross-attention. Over training, these latent tokens learn to “ask” for the information they need: one token might specialize in attending to edges, another to colors, another to spatial relationships. The network discovers what to extract from the input, rather than being told.

The full Perceiver architecture has three conceptual stages, and their simplicity is what makes the model so powerful. Stage 1: Cross-attention from latent to input — this is where the small latent array reads information from the large input. Stage 2: Self-attention among latent tokens — standard transformer blocks that process the compressed information, repeated L times. Stage 3: a task-specific output head (e.g., a linear classifier for classification tasks).

The key insight is that all the expensive processing happens in the latent space, which is small and fixed-size. The only place where the full input appears is in the cross-attention step, which is linear in input size. After cross-attention compresses the input into M latent tokens, all subsequent self-attention layers operate on just M tokens. Since M is typically 512, you can stack 6-8 self-attention blocks without any memory or compute issues, even for inputs with 50K+ tokens. The total cost is O(N * M + L * M^2) — linear in input size N, quadratic only in the small latent size M.

The Perceiver treats all modalities the same way: flatten the input into a 1D sequence of tokens, add position encodings that describe the spatial/temporal structure, and feed it through cross-attention. Images, audio, video, point clouds, and even combinations are all just differently-shaped sequences of feature vectors. The architecture does not need to know or care what modality it is processing.

This is a radical departure from how most architectures work. CNNs are designed specifically for 2D grid data (images). RNNs are designed for sequential data (text, audio). Graph neural networks are designed for graph-structured data. Each modality gets its own specialized architecture with its own inductive biases. The Perceiver throws all of this away and says: every modality is just a bag of tokens with position information.

For images, each pixel becomes a token with (R, G, B) features and (x, y) position encoding. For audio, each time step becomes a token with amplitude features and (t) position encoding. For video, each pixel in each frame becomes a token with (R, G, B) features and (x, y, t) position encoding. For multimodal data (e.g., image + audio), just concatenate the token sequences with their respective position encodings. The architecture handles it all identically.

A single cross-attention pass may not capture all the information in the input. Consider a complex image: one pass might capture the coarse structure (large objects, overall layout), but miss fine details (textures, small objects, spatial relationships). The Perceiver addresses this by repeating the cross-attention step multiple times, allowing the latent array to iteratively refine its representation — attending to different aspects of the input at each iteration.

The typical pattern is: cross-attend to input, do some self-attention, cross-attend to input again, do more self-attention, and so on. Each cross-attention pass has the opportunity to extract new information from the input based on what the latent array has already learned. Crucially, the cross-attention weights can be shared across iterations, keeping the total parameter count low. This is similar to how recurrent networks reuse weights across time steps — more computation without more parameters.

The original Perceiver had a limitation: it could only produce fixed-size outputs (like a classification vector). But many tasks need structured, variable-size outputs — per-pixel segmentation maps, token-by-token language model outputs, per-point classifications. Perceiver IO solves this with a beautiful symmetric extension: output queries that cross-attend to the latent array.

The idea mirrors the input cross-attention but in reverse. For input, the latent array queries the input (latent queries, input keys/values). For output, a set of output query tokens query the latent array (output queries, latent keys/values). The number of output queries determines the output shape. For classification: 1 query that asks “what class is this?” For dense segmentation: H x W queries, one per pixel. For language modeling: seq_len queries, one per token position. This makes the architecture truly general-purpose for any input-output mapping.

The Perceiver achieves competitive results across multiple modalities with the same architecture — no modality-specific components, no specialized preprocessing (beyond basic tokenization), and the same training recipe. This is remarkable because specialized architectures have had years of tuning for their specific modalities, while the Perceiver uses a single general-purpose design.

On ImageNet, the Perceiver achieves 79.0% top-1 accuracy processing raw pixels — competitive with specialized vision models, though not quite matching the best ViT variants that benefit from patch-based inductive biases. When combined with a small convolutional preprocessor (Conv + Perceiver), accuracy jumps to 84.5%. On AudioSet, processing raw audio waveforms, it achieves 38.4 mAP — outperforming many audio-specific architectures. On ModelNet40 point clouds, it reaches 85.7% accuracy. The Perceiver proves that generality and competitiveness are not mutually exclusive.

Since the Perceiver treats all inputs as 1D sequences of tokens, it needs a way to tell tokens apart based on their original position in the input structure. This is where Fourier position encodings come in — sinusoidal functions at different frequencies applied to each coordinate dimension. For images: (x, y). For audio: (t). For video: (x, y, t). These are concatenated with the token features before cross-attention.

Why Fourier features instead of learned positional embeddings (like in BERT or ViT)? Because learned embeddings are fixed to the training resolution. A ViT trained on 224x224 images with 196 patch tokens has exactly 196 learned position embeddings. If you try to use it on a 384x384 image (529 tokens), the learned embeddings cannot generalize. Fourier features, on the other hand, are defined by a continuous function — you can evaluate them at any resolution, making the Perceiver naturally resolution-agnostic.

The Perceiver makes a fundamental trade-off: it gains modality agnosticism and linear scaling, but it gives up the inductive biases that help specialized architectures learn efficiently. A CNN “knows” that nearby pixels are related (locality) and that features can appear anywhere in an image (translation equivariance). The Perceiver knows none of this — it must learn these patterns from data alone, which typically requires more training data and compute.

The latent bottleneck is both a strength and a limitation. It enables linear scaling, but it also means the model must compress all relevant information into M tokens. For tasks requiring fine-grained spatial detail (like dense segmentation or super-resolution), 512 latent tokens may not be enough to preserve all the necessary information. Increasing M helps but increases the self-attention cost quadratically.

The Perceiver demonstrated something that many researchers doubted: a single, general-purpose architecture can handle diverse modalities without task-specific engineering. This was not just a technical contribution — it was a philosophical statement about the nature of perception. The paper argued that all perception is fundamentally the same problem: extracting relevant information from high-dimensional input. The modality is incidental; the architecture should be universal.

The cross-attention bottleneck became a standard architectural patternin multimodal AI. DeepMind's Flamingo uses Perceiver-style cross-attention to condition a language model on visual inputs. Gato, DeepMind's generalist agent, uses a Perceiver-like architecture to handle text, images, and robot actions with a single model. Google's RT-2 uses cross-attention to ground language instructions in robot visual observations. The pattern of “compress large input via cross-attention, then process cheaply” has become ubiquitous.

Perhaps most importantly, the Perceiver paved the way for today's generalist AI systems. The idea that one architecture can perceive any modality — that vision, audition, and language are just different flavors of the same underlying problem — is a cornerstone of the path toward artificial general intelligence. While we are still far from that goal, the Perceiver showed that modality-specific architectures are not a fundamental requirement, just a historical convenience.