Novel-views: New images from different POV of a scene, basically we have images that create a scene, we can get a new image from this scene even if it was not from the original images
Continuous scene representation: U can evaluate any point in space at any time
Splat: spread data or throw, imagine it arabic as “e7df aw btrosh maya” or maybe even better “bet tartash” and you’ll get it
Nerf uses rays to get a pixel and every point in the pixel has color and opacity
-blending: Opacity or see through
Adaptive density control: You control the amount of presence of an object in a world
NOTE: take a loot at flash attention
Density field: A volume of space where every point of this space has a density related to it, this isn’t the physical density, but a density value used to calculate things
Point cloud: a volume of space that that has a large number of infinitely small points that represents or model a structure or a 3D model
An-isotropic: not the same in every direction, specially in 3d Gaussian the variance may be different in every direction
The Gaussian splatting paper talks about three methods:
- Covariance optimizing
- Interleaved density control, and ill look into interleaved
- Depth sorting
The input to the Gaussian splatting rendering method is a set of static images, with the corresponding cameras calibrated by SfM, the produce a sparse point cloud, from these points - from my current understanding - we initialize the 3D Gaussians (Their means as the points positions)
Another understanding I started to have is that first we get a point cloud, after getting a point cloud we check where there is a structure and start to initialize Gaussians there in that place, not necessarily a Gaussian to every point
The Gaussian have the following parameters: mean(position), covariance, and opacity(alpha), the color is stored in the spherical harmonics.
We then splat these Gaussians into the 2d space
The Gaussian formula is given by
Affine transformation - from my understanding - is multiple transformation to the object but still preserving the parallel lines, take a look here
J: the Jacobian of the affine approximation of the projective transformation. What does that really mean ?
This part is really important in talking about the projection into 2d space, when we discard the third row and third column we get the 2d projection,
“Regarding the “sketchy” discard of the 3rd covariance matrix row and column at 1:09:07, they discard this information after the view transform has been applied, as noted by the apostrophe decorating the capital sigma (easy to miss). After the camera transform, the third dimension presumably only describes how stretched the gaussian is in screen depth, which is probably how they get away with this.”
Because after we use the viewing transformation mentioned, the gaussian is projected to the view space, where we can discard the depth without losing any information
While we want to optimize the covariance matrix, we can’t optimize it directly with gradient descent, because covariance matrices must be positive semi-definite i.e. never negative. The best option is to represent the covariance matrix as a scale and rotaion of a unit ellipsoid
The quaternions are then converted to rotation matrices to allow for matrix multiplication with the scaling matrix, since quaternions are 4x4
Questions that I have ?
- Why do we multiply by transposes in the Covariance prime, also dive deeper in the covariance with scaling
- What are quaternions ?
- Space compaction ? all these stuff
- what is PSNR