Extracting Surfaces from Gaussian Splats

This article was based on my work on next best view planning for 3D Gaussian splats. It explains how Poisson surface reconstruction (PSR) can be used to reconstruct the surfaces of a scene from 3D Gaussain splats (3DGS). First we will give background on PSR and 3DGS then we will discuss how to use to use Gaussian splats as input to PSR. Finally, we will briefly explain using stochastic surface reconstruction for next best view planning.

Define the model as with surface , and an indicator function which is inside the model and outside. PSR does not fit directly. It reconstructs and then extracts as a level set.

The gradient of is zero everywhere except at the surface, where it is the inward normal. This is a distribution rather than a function, but convolving with a smoothing filter gives

so a set of normals sampled over is a set of gradients of a smoothed indicator function.

The input to the algorithm is , a noisy sample of the surface's normals, where each sample has a position and an inward normal . Replacing the integral with a sum over the surface patch which each sample represents gives a vector field

is assembled from independent samples, so in general it is not the gradient of any function. Taking the least squares closest gives Poisson's equation. The surface is the level set at the isovalue the samples themselves take,

3DGS represents a scene as a collection of Gaussians, each parameterised by a position, rotation, scale, opacity and a set of spherical harmonic coefficients. A view is synthesised by projecting the Gaussians into screen space and alpha blending them in depth order. The parameters are optimised by gradient descent against a photometric loss between the synthesised and source views. An adaptive density control algorithm periodically adds Gaussians in poorly represented regions of the scene and removes them from dense or transparent ones.

The direct approach treats each Gaussian as one sample. The position is the Gaussian's position, and is the axis of least variance, which is the direction the Gaussian is flattest along.

This fails each of the four requirements.

Each of these problems is a consequence of reading geometry from individual Gaussians. Sampling the rendered images instead avoids them, because alpha blending combines the overlapping Gaussians before the sample is taken.

Gaussian Surfels constrains each Gaussian to be flat by driving the third component of its scale to zero. A flat Gaussian has a well defined normal, which satisfies the first requirement. We use Gaussian Surfels with its additional regularisation terms removed, which simplifies comparison against the unmodified algorithm at the cost of some surface alignment.

The scene is rendered at each input pose . Normals cannot be blended directly, as neither component wise nor spherical interpolation of the Gaussians' normals produces a usable result. Depth blends well, so the normal is reconstructed from the depth map instead. The depth is back projected into a view space position map , and the normal is the cross product of central differences,

A pixel is only shaded by a surface facing the camera which rendered it, so is oriented towards the camera and negated to give the inward normal. This determines the sign which was undetermined when sampling the Gaussians directly.

The position and normal maps are downsampled with a box filter, which reduces the number of samples and averages out noise. Downsampling also fixes . Each remaining sample is one block of pixels back projected onto the surface, so up to foreshortening the patches are equal in area, is constant, and it can be dropped from .

This holds only for blocks which are covered by the surface throughout. A block which straddles a silhouette back projects partly onto the surface and partly onto the background, and its average position lies off the surface. We accumulate a mask alongside the other maps, taking the product of the alpha channel over each block rather than the mean, so that a block containing any uncovered pixel is rejected.

The samples which pass the mask are transformed into world space by and accumulated across the input views. Remaining outliers are rejected by comparing each sample's distance to its neighbours against the distribution over the whole point cloud, and the point cloud is downsampled spatially to bound the size of the solve.

For online capture of a scene it is useful to know where in the scene has not been appropriately covered. We can use the total uncertainty of a camera angle as a measure of it's importance.

Stochastic Poisson surface reconstruction (Sellán and Jacobson, reformulated by Holden et al.) reformulates PSR. The output is a Gaussian process over , so each point in space has a mean and a variance, where the mean is analogous to the reconstructed by PSR. Integrating the probability that a point is misclassified over the domain gives a total uncertainty.