lhphanto/depth-anything-v2-small-1p58bit

Depth Anything V2 Small — Distilled, with a 1.58-bit (ternary) variant

Two DepthAnything V2 Small (ViT-S) depth students distilled from a frozen full-precision DepthAnything V2 Large (ViT-L) teacher on unlabeled images (Google OpenImages), with no ground-truth depth — pure teacher→student distillation against the teacher's relative-depth output.

This repo provides two checkpoints:

Checkpoint	Encoder	Quantization	DA-2K overall acc
baseline_student_final.safetensors	ViT-S	none (full precision)	0.6489
student_final.safetensors	ViT-S	1.58-bit ternary weights ({-1,0,1}) + int8 activations on the DINOv2 encoder linears	0.6407

Ternary quantization costs only −0.0082 overall accuracy (−1.3% relative) while making the encoder weights amenable to ~8× compression (see Storage below).

These are early distillation runs on unlabeled data, not fully trained production models, so both sit below the published DAV2-S DA-2K numbers. The meaningful quantity here is the relative fp ↔ 1.58-bit gap, which is small.

DA-2K accuracy by scene

Scene	Full precision	1.58-bit	Δ
indoor	0.6333	0.6119	−0.0214
outdoor	0.6599	0.6453	−0.0146
non_real	0.7129	0.7030	−0.0099
transparent_reflective	0.6028	0.5981	−0.0047
adverse_style	0.6128	0.6067	−0.0061
aerial	0.6082	0.6289	+0.0207
underwater	0.6667	0.6752	+0.0085
object	0.7230	0.7095	−0.0135
OVERALL	0.6489	0.6407	−0.0082

Metric: pairwise relative-depth accuracy on the DA-2K benchmark (2068 point pairs over 1033 images).

Important: these are NOT transformers models

The checkpoints are raw DepthAnythingV2 state_dicts. They do not load via AutoModel.from_pretrained. You need the model code from the Depth-Anything-V2-Bit repo (a fork of Depth Anything V2).

Load the full-precision student

import torch
from safetensors.torch import load_file
from depth_anything_v2.dpt import DepthAnythingV2

cfg = {'encoder': 'vits', 'features': 64, 'out_channels': [48, 96, 192, 384]}
model = DepthAnythingV2(**cfg)
model.load_state_dict(load_file('baseline_student_final.safetensors'))
model.eval()

depth = model.infer_image(bgr_image, input_size=518)   # HxW, higher = closer

Load the 1.58-bit student

The encoder linear layers must be swapped to BitLinear before loading, so the ternary weights land in the right modules:

import torch
from safetensors.torch import load_file
from depth_anything_v2.dpt import DepthAnythingV2
from bitnet import convert_linear_to_bitlinear

cfg = {'encoder': 'vits', 'features': 64, 'out_channels': [48, 96, 192, 384]}
model = DepthAnythingV2(**cfg)
convert_linear_to_bitlinear(model.pretrained)            # swap DINOv2 nn.Linear -> BitLinear
model.load_state_dict(load_file('student_final.safetensors'))
model.eval()

depth = model.infer_image(bgr_image, input_size=518)

Storage note (the 1.58-bit file is fp-sized as shipped)

student_1p58bit.safetensors stores ternary weights as fp32 masters, so it is the same size as the full-precision file. This is the form needed for the fake-quant / fold-inference paths above. To realize the ~8× weight reduction, bit-pack the encoder (4 ternary weights per byte) with the helper in the repo:

from bitnet import pack_model_for_storage
pack_model_for_storage(model.pretrained)                 # -> packed_weight (uint8) + w_scale
torch.save(model.state_dict(), 'student_packed.pth')     # ~8x smaller encoder weights
# load back with: prepare_packed_load(...) + load + unpack_model_from_storage(...)

On inference speed

1.58-bit gives accuracy parity + weight-size reduction, but not faster inference at ViT-S scale on GPU/CPU. Low-bit GEMM speedups are an LLM-scale, low-batch, memory-bandwidth-bound phenomenon; a ViT-S at 1370 tokens/image is small-matrix and compute-bound, where cuBLAS bf16 (+ xFormers + AMP) is already near-optimal. Measured int8 kernels ran ~5× slower than bf16 at these shapes. See the project report for the full latency analysis.

Training summary

• Teacher: DepthAnything V2 Large (ViT-L), frozen, relative-depth output as pseudo-labels.
• Student: DepthAnything V2 Small (ViT-S); 1.58-bit variant ternarizes all DINOv2 encoder nn.Linear weights (BitNet b1.58: absmean ternarization, per-token int8 activations, STE).
• Data: unlabeled Google OpenImages (no ground truth).
• Loss: scale-shift-invariant L1 + multi-scale gradient matching (MiDaS / DAV2 objective for relative depth).
• Optimizer: AdamW, separate LRs (5e-6 encoder / 5e-5 DPT head), cosine schedule.

Citation

Built on Depth Anything V2:

@article{depth_anything_v2,
  title={Depth Anything V2},
  author={Yang, Lihe and Kang, Bingyi and Huang, Zilong and Zhao, Zhen and Xu, Xiaogang and Feng, Jiashi and Zhao, Hengshuang},
  journal={arXiv:2406.09414},
  year={2024}
}

License: Apache-2.0 (inherited from Depth Anything V2). No OpenImages images are redistributed here — only learned weights.