--- license: mit tags: - 3d-gaussian-splatting - novel-view-synthesis - compression - feed-forward ---

CodecSplat: Ultra-Compact Latent Coding for Feed-Forward 3D Gaussian Splatting

Paper | Code | Models

CodecSplat is a latent coding framework for feed-forward 3D Gaussian splatting. Instead of compressing the final irregular 3D Gaussian primitives, it entropy-codes an intermediate 2D feature representation used for depth and Gaussian prediction. This keeps the scene representation compact while preserving fast feed-forward reconstruction. ## Installation The code is developed with Python 3.10 and PyTorch 2.9.1. Please refer to the official [PyTorch installation guide](https://pytorch.org/get-started/locally/) if your CUDA version or platform differs from the example below. ```bash conda create -y -n py310torch291 python=3.10 conda activate py310torch291 pip install torch==2.9.1 torchvision==0.24.1 torchaudio==2.9.1 \ --index-url https://download.pytorch.org/whl/cu130 pip install -r requirements.txt ``` ## Model Zoo Pretrained weights are available on [Hugging Face](https://huggingface.co/pengpeng-yu/CodecSplat): | Checkpoint | Dataset | Input views | Resolution | Notes | | --- | --- | ---: | --- | --- | | [CodecSplat-Re10K-2view](https://huggingface.co/pengpeng-yu/CodecSplat/resolve/main/codecsplat-base-re10k-256x256-view2-e3a545db.pth) | RealEstate10K | 2 | 256x256 | Variable-rate, lmb=16-1024 | | [CodecSplat-DL3DV-8view](https://huggingface.co/pengpeng-yu/CodecSplat/resolve/main/codecsplat-base-dl3dv-256x448-view8-075062d0.pth) | DL3DV | 8 | 256x448 | Variable-rate, lmb=16-1024 | After downloading a checkpoint, place it under `pretrained/` or update the `checkpointing.pretrained_model=...` entry in the corresponding test script. ## Datasets CodecSplat is evaluated on 256x256 RealEstate10K and 256x448 DL3DV. The datasets used by the provided scripts are available from: - [RealEstate10K 360p torch format](https://huggingface.co/datasets/lhmd/re10k_torch) - [DL3DV-ALL-480P](https://huggingface.co/datasets/DL3DV/DL3DV-ALL-480P) - [DL3DV-Benchmark](https://huggingface.co/datasets/DL3DV/DL3DV-Benchmark) The RealEstate10K release above is already in the expected torch-chunk format. For DL3DV, convert the downloaded source data with: ```bash mkdir -p datasets # DL3DV benchmark/test split. python src/scripts/convert_dl3dv.py \ --split test \ --input_dir /path/to/DL3DV-Benchmark \ --output_dir datasets/DL3DV-Benchmark-480P-Processed \ --img_subdir images_8 # DL3DV training split. Test scenes are excluded by index. python src/scripts/convert_dl3dv.py \ --split train \ --input_dir /path/to/DL3DV-ALL-480P \ --output_dir datasets/DL3DV-ALL-480P-Processed \ --img_subdir images_8 \ --test_index_path datasets/DL3DV-Benchmark-480P-Processed/test/index.json # Use the DL3DV benchmark split as the test split for DL3DV evaluation. ln -s ../DL3DV-Benchmark-480P-Processed/test datasets/DL3DV-ALL-480P-Processed/test ``` The resulting layout should look like: ```text datasets ├── DL3DV-ALL-480P │ ├── 1K │ │ ├── 0a1b7c20... │ │ │ ├── images_8 # 270x480 │ │ │ └── transforms.json │ │ └── ... ├── DL3DV-Benchmark │ ├── 0a1b7c20... │ │ ├── images_8 # 270x480 │ │ └── transforms.json │ └── ... ├── RE10K-Torch-360p │ ├── train │ │ ├── 000000.torch │ │ ├── ... │ │ └── index.json │ └── test │ ├── 000000.torch │ ├── ... │ └── index.json ├── DL3DV-ALL-480P-Processed │ ├── train │ │ ├── 000000.torch │ │ ├── ... │ │ └── index.json │ └── test -> ../DL3DV-Benchmark-480P-Processed/test └── DL3DV-Benchmark-480P-Processed └── test ├── 000000.torch ├── ... └── index.json ``` ## Evaluation The main codec evaluation entry point is: ```bash # Run both datasets at the default rate points, lmb=16 and lmb=1024. python scripts/test_codec_bitstream_rd.py all # Run one dataset at selected rate points. python scripts/test_codec_bitstream_rd.py dl3dv 16 128 1024 python scripts/test_codec_bitstream_rd.py re10k 16 128 1024 ``` Equivalent single-dataset shell scripts are also provided: ```bash # DL3DV 8-view 256x448 evaluation. lmb=1024 bash scripts/test_dl3dv_view8_256x448.sh # RealEstate10K 2-view 256x256 evaluation. lmb=1024 bash scripts/test_re10k_view2_256x256.sh ``` Useful test flags: - `test.save_image=true`: save rendered target views. - `test.save_gt_image=true`: save ground-truth target views. - `test.save_input_images=true`: save context/input views. - `test.save_depth=true`: save predicted depths. - `test.save_gaussian=true`: save reconstructed Gaussian primitives as PLY. - `test.save_video=true`: save rendered target-view sequences as MP4 when the evaluation index contains consecutive frames. - `model.encoder.codec_lmb_range=[L,L]`: evaluate a fixed rate point L. - `model.encoder.codec_eval_use_bitstream=true`: run actual compress/decompress bitstream evaluation. ## Video Rendering CodecSplat supports video rendering through `test.save_video=true`. ```bash CUDA_VISIBLE_DEVICES=0 python -m src.main +experiment=dl3dv \ mode=test \ dataset/view_sampler=evaluation \ dataset.view_sampler.num_context_views=12 \ dataset.view_sampler.index_path=assets/dl3dv_start_0_distance_100_ctx_12v_video.json \ dataset.roots=[datasets/DL3DV-Benchmark-480P-Processed] \ dataset.image_shape=[256,448] \ dataset.test_len=1 \ test.save_video=true \ test.compute_scores=false \ test.render_chunk_size=10 \ test.stabilize_camera=true \ model.encoder.codec_lmb_range=[1024.0,1024.0] \ model.encoder.codec_eval_use_bitstream=true \ checkpointing.pretrained_model=/path/to/dl3dv_codec_checkpoint.ckpt \ checkpointing.no_strict_load=true \ output_dir=outputs/codecsplat_video_smoke ``` ## Training Training is performed in two stages. Stage 1 trains the feed-forward Gaussian reconstruction model without the learned feature codec. Stage 2 enables the feature codec and trains the compression path. ```bash # DL3DV 8-view, 256x448. bash scripts/train_dl3dv_view8_256x448_stage1.sh bash scripts/train_dl3dv_view8_256x448_stage2.sh # RealEstate10K 2-view, 256x256. bash scripts/train_re10k_view2_256x256_stage1.sh bash scripts/train_re10k_view2_256x256_stage2.sh ``` The comments in the training scripts use notation such as `1 x 8 GPUs` or `4 x 8 GPUs`. This means per-GPU batch size times the number of GPUs. In our experiments, the 8-GPU setting refers to 8 NVIDIA RTX 5880 Ada 48GB GPUs. On machines with fewer GPUs, gradient accumulation can be used to approximate the same effective batch size. For example, replacing 8 GPUs with 4 GPUs can often be approximated by doubling `trainer.accumulate_grad_batches`. Most provided training commands use `trainer.precision=bf16-mixed`. Geometry, depth probability, entropy-probability, and rasterization-sensitive operations are kept in float32 internally where needed. The provided stage-1 training scripts warm-start from DepthSplat Gaussian splatting checkpoints. Download the corresponding checkpoints before training: ```bash mkdir -p pretrained # RealEstate10K 2-view 256x256 warm start. wget https://huggingface.co/haofeixu/depthsplat/resolve/main/depthsplat-gs-base-re10k-256x256-view2-ca7b6795.pth -P pretrained # DL3DV 2-6 view 256x448 warm start. wget https://huggingface.co/haofeixu/depthsplat/resolve/main/depthsplat-gs-base-dl3dv-256x448-randview2-6-02c7b19d.pth -P pretrained ``` Stage-2 training should be initialized from the corresponding stage-1 checkpoint. Update the `checkpointing.pretrained_model` paths in `scripts/train_*_stage2.sh` to your stage-1 output. ## Camera Conventions The camera intrinsic matrices are normalized, with the first row divided by the image width and the second row divided by the image height. The camera extrinsic matrices follow the OpenCV camera-to-world convention: `+X` right, `+Y` down, and `+Z` pointing into the scene. ## Citation If you find this project useful, please cite CodecSplat: ```bibtex @article{yu2026codecsplat, title = {CodecSplat: Ultra-Compact Latent Coding for Feed-Forward 3D Gaussian Splatting}, author = {Yu, Pengpeng and Jiang, Runqing and Zhang, Qi and Li, Dingquan and Wang, Jing and Guo, Yulan}, journal = {arXiv preprint arXiv:2605.25563}, year = {2026} } ``` ## Acknowledgements This project builds on several excellent open-source projects, including [DepthSplat](https://github.com/cvg/depthsplat), [ReSplat](https://github.com/cvg/resplat), [gsplat](https://github.com/nerfstudio-project/gsplat), [Depth Anything V2](https://github.com/DepthAnything/Depth-Anything-V2), and [lossy-vae](https://github.com/duanzhiihao/lossy-vae). We also thank the dataset authors of [RealEstate10K](https://google.github.io/realestate10k/) and [DL3DV](https://github.com/DL3DV-10K/Dataset). We are grateful to the authors and maintainers for making their work available.