Gaussian splatting rendered radiance fields in real time

↗

Millions of blobs instead of one neural network

On 8 August 2023, Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler and George Drettakis posted a paper, later published in ACM Transactions on Graphics, introducing 3D Gaussian Splatting. The abstract states the method achieves "state-of-the-art visual quality while maintaining competitive training times" and, critically, enables "high-quality real-time (>= 30 fps) novel-view synthesis at 1080p resolution." The paper is explicit about the problem this solves: prior neural methods, meaning approaches in the lineage of Neural Radiance Fields, could produce comparable image quality but, in the authors' words, no existing technique could reach real-time display rates at that quality and resolution. The claim concerns frame rate at a stated resolution, not universal real-time performance on arbitrary hardware.

The mechanism: splatting instead of marching

The method starts from sparse 3D points generated during ordinary camera calibration, the same structure-from-motion step NeRF also depends on, and represents the scene as a large collection of 3D Gaussians, each with its own position, orientation and colour. Training interleaves optimisation with density control, adding or removing Gaussians so the representation covers surfaces without wasting computation in empty space. The decisive difference from a NeRF is how an image gets produced: instead of marching rays through a queried function, the renderer projects, or "splats," the Gaussians directly onto the image plane, a fast rasterisation technique rather than a per-pixel neural evaluation. That is what makes the stated frame rate possible on ordinary GPU hardware, according to the authors' project page, which confirms the method targets complete, unbounded scenes rather than isolated objects.

What a display still needs to add

A splat scene is a point-like primitive cloud with view-dependent colour, not a video and not, by default, a mesh. The paper describes rendering to a conventional 2D image from a chosen camera pose; it says nothing about stereo pairs, light-field view grids, or headset optics. Getting a splat scene onto a spatial display means rendering it once per required viewpoint, twice for a stereo headset, dozens of times for a lenticular panel, exactly the kind of downstream requirement the paper does not test. Looking Glass Factory's own site lists Gaussian Splats among the content types advanced users can bring into its light-field pipeline, alongside NeRFs, confirming the format has reached that kind of display workflow but saying nothing about the fidelity achieved once it is split across dozens of simultaneous views rather than one.

Questions to bring to a demo

• Is the frame rate quoted for one 2D viewpoint, or for the full set of views a stereo or light-field display actually needs?
• How many Gaussians make up the scene, and how does that scale with the display's resolution and view count?
• Was the scene captured with ordinary photographs and calibration, or does it depend on additional sensors not mentioned in the paper?

3D Gaussian Splatting answered a specific, documented complaint about neural view synthesis, that it was not fast enough to feel interactive, without addressing what any particular spatial display does with the result. That remaining step belongs to the display pipeline, not the paper.