NeRF: Neural Radiance Fields

Imagine you have photographs of a room taken from different positions. Can you generate a photorealistic image from a completely new viewpoint — one you never photographed? This is novel view synthesis, one of the oldest and most important problems sitting at the intersection of computer vision and computer graphics. It sounds simple, but think about what it requires: the system must understand the full 3D geometry of the scene, how light interacts with every surface, and how appearance changes with viewpoint.

Previous approaches built explicit 3D models (meshes, point clouds) or used image-based methods (light fields, multiplane images). Each had painful trade-offs. Meshes require complex surface reconstruction and struggle with thin structures, transparency, and complex topology. Point clouds produce holes and artifacts. Light fields need dense captures. Before NeRF, no single method could produce photorealistic novel views from a handful of casually captured photographs.

NeRF took a radically different approach: instead of building an explicit 3D model, it trains a tiny neural network to BE the scene. You can ask this network “what color and density exists at point (x, y, z) when viewed from direction (theta, phi)?” and it answers. To render an image, you cast rays through each pixel and integrate the network's responses along each ray. The entire scene fits in ~5MB of network weights — roughly the size of a single JPEG photograph.

A neural radiance field is a function F(x, d) → (sigma, c)that maps a 3D spatial position x = (x, y, z) and a 2D viewing direction d = (theta, phi) to a volume density sigma and an RGB color c. But why 5D? Why not just 3D position? Because appearance depends on viewing angle. Think about a shiny metal surface — it looks completely different from different angles due to specular reflections. A matte wall, on the other hand, looks the same from every direction. NeRF needs to model both behaviors.

Here is the critical design decision that makes NeRF physically consistent: density sigma depends only on position, not viewing direction. Why? Because a surface either exists at a point or it doesn't — that is a geometric fact independent of who is looking at it. But color depends on both position and direction, because the same surface can appear different colors from different angles (think of how a soap bubble shimmers). This separation ensures that the reconstructed geometry is multiview-consistent — you won't get a surface appearing in one view but vanishing in another.

Here is a surprising fact that almost derailed the entire NeRF project: if you feed raw (x, y, z) coordinates into an MLP, the results are disastrously blurry. The network can learn the rough shape of objects but completely fails to capture sharp edges, fine textures, and intricate details. Why? Because MLPs have a well-known spectral bias — they preferentially learn low-frequency functions first, and struggle to represent high-frequency variations no matter how long you train.

Think of it this way: raw coordinates (x, y, z) are inherently “smooth” inputs. Moving from position (0.5, 0.3, 0.1) to (0.5, 0.3, 0.11) is a tiny change in input, so the MLP wants to produce a tiny change in output. But in reality, that tiny spatial shift might cross the edge of an object — a massive change in density and color! The MLP's smoothness prior fights against the sharp discontinuities that define object boundaries.

The solution is beautifully simple: map each input coordinate through a set of sinusoidal functions at exponentially increasing frequencies. Instead of giving the network x, give it [sin(x), cos(x), sin(2x), cos(2x), sin(4x), cos(4x), ... sin(2^(L-1) x), cos(2^(L-1) x)]. A tiny change in x now creates large changes in the high-frequency components, giving the MLP the “leverage” it needs to represent fine detail. This is the same trick that transformers use for position embeddings — and for good reason.

NeRF's architecture is surprisingly simple: an 8-layer MLP with 256 hidden units and ReLU activations. In an era of billion-parameter transformers, this is almost comically small — roughly 1.2 million parameters. But two design choices make all the difference and elevate this simple network into a photorealistic scene representation.

The first trick: a skip connection that re-injects the positional encoding at layer 5. Why? Because after passing through 4 layers of nonlinear transformations, the original input signal can get “washed out.” The skip connection ensures the network always has direct access to the high-frequency positional information, preventing it from forgetting fine spatial details. This is inspired by the DenseNet idea of maintaining information flow through deep networks.

The second trick: a two-stage output design. After the 8-layer trunk processes the positional encoding, it outputs density sigma directly (as a single scalar with a ReLU to ensure non-negativity). The same trunk features are then concatenated with the viewing direction encoding and passed through one additional 128-unit layer to produce RGB color. This enforces the physical constraint that density is view-independent while color is view-dependent. Without this separation, the network could “cheat” by making objects appear and disappear depending on the camera angle, producing multiview-inconsistent geometry.

We now have a network that can tell us the density and color at any point in space. But how do we turn that into an actual image? The answer is volume rendering, a technique borrowed from medical imaging and physics simulation. The basic idea: for each pixel in the output image, shoot a ray from the camera through that pixel into the scene, sample points along the ray, query the MLP at each point, and accumulate color weighted by how “opaque” each point is.

The critical concept is transmittance T(t) — the probability that the ray travels from the camera to point t without hitting anything solid. If a ray encounters a dense region early on (like the front surface of an object), T drops sharply and light from behind is blocked. This is exactly how occlusion works in the real world: you cannot see through a solid wall because its high density blocks the transmittance of light from behind it.

In practice, NeRF approximates the continuous integral with a discrete sum over N sample points. For each sampled point t_i, the network provides density sigma_i and color c_i. The contribution of each point is weighted by its alpha value alpha_i = 1 - exp(-sigma_i * delta_i) (where delta_i is the distance between adjacent samples) and the accumulated transmittance T_i = product of (1 - alpha_j) for all j before i. The final pixel color is the sum of these weighted contributions.

Here is a computational problem: if you uniformly sample 256 points along every ray, most of those samples will land in empty space (air, background) or inside solid objects where they contribute nothing useful to the final pixel color. You are wasting the vast majority of your MLP evaluations on uninformative regions. For an 800x800 image with 256 samples per ray, that is 164 million MLP forward passes — and most of them are wasted.

NeRF's solution is elegant: hierarchical sampling with two networks. First, a “coarse” network evaluates 64 uniformly-spaced points along the ray. The coarse network's density predictions reveal where surfaces likely exist — regions with high density are interesting, empty regions are not. Then, an “importance sampling” step draws 128 additional points concentrated around the high-density regions. A “fine” network evaluates all 192 points (64 coarse + 128 fine) to produce the final high-quality rendering.

This is like a photographer first scanning the scene with their eyes (coarse), then focusing their expensive high-resolution camera exactly on the interesting parts (fine). The coarse network acts as a cheap scout, and the fine network does the real work where it matters. The combined result is dramatically better than either network alone, because computation is concentrated exactly where it produces the most visual benefit.

NeRF's training is conceptually beautiful in its simplicity: render pixels through the neural radiance field and compare them to real pixels from photographs. The loss is just MSE between rendered and ground truth pixel colors. No 3D supervision, no depth maps, no segmentation masks, no silhouettes — just photographs. The 3D geometry emerges entirely as a byproduct of making 2D renderings match 2D photos.

Think about how remarkable this is: the network has never been told what 3D shape the scene has. It has never seen a depth map or a point cloud. It only knows that “when I render rays through my density field, the resulting colors should match these photos.” Yet from this 2D supervision alone, it reconstructs complete 3D geometry. This works because the multiview consistency constraint is enormously powerful — the only way to simultaneously match all photos from all viewpoints is to build a correct 3D model internally.

Each training iteration samples a random batch of 4,096 rays from the training images (not full images — individual rays). This is efficient because each ray is independent, and random sampling across different images and pixel positions provides a rich training signal with every batch. The process runs for 200K-300K iterations, typically taking 1-2 days on a single NVIDIA V100 GPU.

After training, NeRF can render the scene from any viewpoint. Simply specify a camera position and orientation, cast rays through each pixel, and use volume rendering to produce the image. The results are photorealistic, with accurate view-dependent effects (specular highlights that shift realistically as you move), fine geometric details (thin structures, complex topology), and correct occlusion (objects properly blocking what is behind them).

What makes NeRF's results particularly impressive is the level of detail in challenging regions: thin structures like chair legs and plant leaves, reflective surfaces like glass and metal, and semi-transparent materials like curtains and smoke. Previous methods either required explicit surface reconstruction (which fails on complex topology) or were limited to simple, low-frequency appearance. NeRF's continuous representation and positional encoding together capture both the macro structure and micro texture of real scenes.

The rendering process is inherently slow, however. Each pixel requires ~192 MLP evaluations. An 800x800 image has 640,000 pixels, meaning ~123 million forward passes per frame. At inference time, this takes about 30 seconds per frame on a V100 GPU — far from real-time. This computational cost is the original NeRF's biggest practical limitation and became the main target for subsequent research.

NeRF dramatically outperformed all previous methods on standard benchmarks. On both synthetic (Blender) and real (LLFF) datasets, it achieved significantly higher PSNR, SSIM, and LPIPS scores. The improvements were not marginal — NeRF achieved 5+ dB PSNR improvements over prior methods on synthetic scenes, which translates to dramatically more realistic images visible to the naked eye.

The ablation study in the paper is particularly revealing. Removing positional encoding drops PSNR by about 4 dB — the images become noticeably blurry. Removing view dependence (making color only position-dependent) drops PSNR by ~1 dB and eliminates all specular highlights. Using a single network instead of hierarchical sampling costs ~1.5 dB. Each component contributes meaningfully, and the full system is far more than the sum of its parts.

NeRF's success inspired an explosion of follow-up work, with hundreds of papers published within two years. Each addresses specific limitations of the original method. The paper sparked what the community calls the NeRF revolution in 3D computer vision — a shift from explicit geometric representations to neural implicit ones.

Understanding NeRF's limitations is essential for knowing when to use it and when alternative approaches are better. The original NeRF is slow (1-2 days training, 30+ seconds per rendered frame), per-scene (one network per scene, no cross-scene generalization), static(cannot handle moving objects or dynamic scenes), and requires known camera poses (typically from expensive COLMAP preprocessing). It also suffers from aliasing at different scales and struggles with unbounded outdoor scenes.

NeRF didn't just solve view synthesis — it created an entirely new paradigm for representing 3D scenes. The paper's impact extends far beyond its specific application, influencing how researchers think about 3D representation, rendering, and the relationship between 2D observations and 3D structure. With over 8,000 citations and counting, NeRF ranks among the most influential computer vision papers of the 2020s.

The core idea — representing complex 3D structures as neural implicit functions — has spread into dozens of fields. In robotics, NeRF-style scene representations help robots understand and navigate 3D environments. In autonomous driving, neural radiance fields reconstruct street scenes from vehicle cameras for simulation and planning. In medicine, similar techniques reconstruct 3D anatomy from sparse medical scans. In entertainment, NeRF powers virtual production, visual effects, and immersive content creation.

Perhaps NeRF's most significant successor is 3D Gaussian Splatting (Kerbl et al., 2023), which swings the pendulum back toward explicit representations. Instead of an implicit MLP, 3DGS represents scenes as millions of tiny 3D Gaussians — anisotropic ellipsoids with position, color, opacity, and covariance. These can be rendered via differentiable rasterization at 100+ FPS, achieving real-time performance that the original NeRF could only dream of. Yet 3DGS owes its existence to NeRF: it was NeRF that proved neural 3D representations could achieve photorealistic quality, inspiring the search for faster alternatives.