docs/DEPENDENCIES_AND_WEIGHTS.md

Dependencies, attribution, and weight downloads

This guide is written for a first-time user. AccessPath is inspired by and adapted from prior work on amodal completion, segmentation, geometry, and visual 3D reconstruction. It explains both what is actually used at runtime and what is only cited or retained as an optional comparison. Do not download a model merely because its name occurs in a related-work note.

1. What AccessPath implements

AccessPath implements an accessibility-focused workflow inspired by these research directions, while adding project-level adaptation around external models:

1. prompt selection and target/obstacle mask proposal orchestration;
2. constrained visible/hidden/amodal mask construction and validation;
3. mask-restricted 2D inpainting candidate generation and selection;
4. depth/geometry diagnostics, quality gates, rendering, and review bundles;
5. an adapter that sends an RGB image plus a three-value mask to an external visual-3D backend.

It does not claim authorship of SAM 3, Stable Diffusion, Depth Anything V2, VGGT, Amodal3R, TRELLIS, pix2gestalt, Open-World AMODAL, or Amodal Completion in the Wild. Where one of these is a runtime backend, its code, license, and weights remain separate; where it is listed as related work, AccessPath does not import or execute it.

2. Exact dependency status

Method/model	Status in this repository	Where in AccessPath	What a user must do
SAM 3	Used by the automatic mask-proposal stage.	tools/accessibility_mask_proposals.py	Install it in its own environment and request access to facebook/sam3.
Stable Diffusion inpainting	Used by the default GPU 2D completion stage.	tools/accessibility_2d_completion.py	Download an inpainting checkpoint under its own terms.
Depth Anything V2	Used by the default full-pipeline depth/geometry diagnostic.	accessibilityamodal/depth.py	Install the official repository and download the selected checkpoint.
Amodal3R	Used by the default visual 3D stage through a thin adapter.	tools/accessibility_3d_completion.py	Install the official runtime and obtain its model assets.
TRELLIS image large	Upstream dependency reported by the Amodal3R model card.	External Amodal3R runtime	Obtain it only following the upstream Amodal3R instructions.
VGGT	Optional depth/point-map engine; not selected by the default Slurm launcher.	accessibilityamodal/depth.py	Install/download only when running --depth-engine vggt.
pix2gestalt	Not executed. Kept only as a related-work/comparison URL in a baseline manifest.	tools/accessibility_fast_2d_baseline.py	No installation or checkpoint is required for AccessPath.
Open-World AMODAL	Not executed. Kept only as a related-work/comparison URL.	tools/accessibility_fast_2d_baseline.py	No installation or checkpoint is required for AccessPath.
Amodal Completion in the Wild	Not executed. Mentioned as a compatible external research backend only.	accessibilityamodal/pipeline.py	No installation or checkpoint is required for AccessPath.

Therefore, no pix2gestalt, Open-World AMODAL, or Amodal-Wild code/weights are needed for the commands in this repository. Their links are present so readers can distinguish the accessibility-adapted AccessPath workflow from related amodal-completion work.

3. Before starting

You need Linux, Git, Python, a CUDA-capable GPU, and a Slurm installation for the all-in-one pipeline command. The direct 2d and 3d commands can be run outside Slurm, but still need a compatible CUDA environment.

Clone the anonymous release:

git clone https://huggingface.co/anonymous-accesspath/AccessPath
cd AccessPath

Create a general utility environment. Install the PyTorch build that matches your CUDA driver using the official PyTorch selector, then install the Python packages used by the project wrappers:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
# Install a CUDA-compatible PyTorch build here, following pytorch.org.
python -m pip install -r requirements/runtime.txt
python -m pip install diffusers transformers accelerate safetensors huggingface_hub

Do not place downloaded checkpoints inside the Git clone. Keep them in a separate local models/ directory and pass their paths through environment variables. This avoids accidentally committing large or license-restricted files.

4. Download each runtime asset

4.1 SAM 3: mask proposals

SAM 3 is used only to propose a visible target mask and an obstacle mask. The proposal is not ground truth; inspect it before using it for an experiment.

1. Read and accept the access terms on facebook/sam3.
2. Follow the official SAM 3 installation guide. SAM 3 may require a newer Python/PyTorch/CUDA combination than the remaining pipeline, so a dedicated environment is recommended.
3. Authenticate and download the official files:

hf auth login
hf download facebook/sam3 sam3.pt config.json --local-dir /path/to/models/sam3

4. Record the environment and paths for AccessPath:

export SAM3_PYTHON=/path/to/sam3-environment/bin/python
export SAM3_REPO=/path/to/sam3-source
export SAM3_CHECKPOINT=/path/to/models/sam3/sam3.pt

4.2 Stable Diffusion: 2D inpainting

The 2D stage calls Diffusers' inpainting pipeline. Obtain an inpainting model from sd-legacy/stable-diffusion-inpainting or use another compatible checkpoint only after checking its model card and license. A typical download is:

hf download sd-legacy/stable-diffusion-inpainting \
  --local-dir /path/to/models/stable-diffusion-inpainting
export SD_MODEL=/path/to/models/stable-diffusion-inpainting

The 2D implementation changes only the reviewed completion envelope and saves its selected result plus metadata under the requested output directory.

4.3 Depth Anything V2: depth and geometry diagnostics

The default full launcher expects the large indoor metric checkpoint: Depth-Anything-V2-Metric-Hypersim-Large. Install source code from the official Depth Anything V2 repository, then download the exact checkpoint:

git clone https://github.com/DepthAnything/Depth-Anything-V2 /path/to/Depth-Anything-V2
hf download depth-anything/Depth-Anything-V2-Metric-Hypersim-Large \
  depth_anything_v2_metric_hypersim_vitl.pth \
  --local-dir /path/to/models/depth-anything-v2-metric-hypersim-large

export DEPTH_REPO=/path/to/Depth-Anything-V2
export DEPTH_CHECKPOINT=/path/to/models/depth-anything-v2-metric-hypersim-large/depth_anything_v2_metric_hypersim_vitl.pth

Depth output is a diagnostic estimate. It is not automatically calibrated metric ground truth for the input camera and must not be treated as a navigation-safety measurement.

4.4 Amodal3R and TRELLIS: visual 3D completion

AccessPath does not redistribute Amodal3R source code or weights. The adapter requires an external Python environment where this import succeeds:

python -c "from amodal3d.pipelines import Amodal3RImageTo3DPipeline; print('Amodal3R runtime ready')"

Use the official Amodal3R project page, model card, and paper for installation and license terms. The model card states that its implementation is built on TRELLIS and obtains pretrained assets from microsoft/TRELLIS-image-large. For a local snapshot of the published model card repository:

hf download Sm0kyWu/Amodal3R --local-dir /path/to/models/Amodal3R

Follow the upstream project instructions for any additional TRELLIS assets and CUDA rasterizer dependencies. Then configure AccessPath:

export AMODAL3D_PYTHON=/path/to/amodal3r-environment/bin/python
export AMODAL3D_MODEL=/path/to/models/Amodal3R
export AMODAL3D_TORCH_HOME=/path/to/model-cache/torch

The 3D result is a learned visual reconstruction. It is not guaranteed to be metric, watertight, scale-calibrated, or safe for path-planning decisions.

4.5 Optional VGGT geometry route

VGGT is implemented as an optional --depth-engine vggt route. It is useful for comparing a geometry/point-map estimate, but it is not the default AccessPath visual-3D output and it is not required for the default full run.

git clone https://github.com/facebookresearch/vggt /path/to/vggt
python -m pip install -r /path/to/vggt/requirements.txt
hf download facebook/VGGT-1B --local-dir /path/to/models/VGGT-1B

Then invoke the diagnostic engine with --depth-engine vggt, pass --vggt-repo /path/to/vggt, and use --vggt-model /path/to/models/VGGT-1B. See the official VGGT repository and model page for current access, license, and checkpoint conditions.

4.6 Project amodal-mask adapter checkpoint

AMODAL_CHECKPOINT is an AccessPath-trained mask-adapter checkpoint, not an upstream public dependency. It is deliberately withheld during anonymous review. Its release requires confirmation of training-data permissions, base-model terms, privacy review, and the paper's release policy.

Without this checkpoint, the all-in-one pipeline command stops at its preflight check. Readers can still run the direct 2D and visual-3D commands with their own reviewed masks. This limitation is intentional and explicit; the repository does not substitute a different checkpoint silently.

5. Run the three supported routes

Route A — direct 2D completion

Prepare an RGB image and three aligned binary masks: visible target, complete amodal target, and obstacle. All must have the same width and height as the RGB image.

python accesspath.py 2d -- \
  --image inputs/scene.jpg \
  --target-visible-mask inputs/target_visible.png \
  --target-amodal-mask inputs/target_amodal.png \
  --obstacle-mask inputs/obstacle.png \
  --category stairs \
  --model "$SD_MODEL" \
  --output-dir outputs/stairs_2d \
  --device cuda

Inspect the selected RGB and the JSON manifest under outputs/stairs_2d.

Route B — direct visual 3D completion

Create one three-value mask at the RGB resolution: 255 is background, 188 is visible target, and 0 is hidden target. Then run:

AMODAL3D_TORCH_HOME="$AMODAL3D_TORCH_HOME" \
"$AMODAL3D_PYTHON" tools/accessibility_3d_completion.py \
  --image inputs/scene.jpg \
  --mask inputs/amodal_3value.png \
  --model "$AMODAL3D_MODEL" \
  --output-dir outputs/stairs_3d

Inspect the generated manifest and image/mesh assets. Do not interpret a plausible-looking 3D asset as a physical measurement.

Route C — full Slurm pipeline

After configuring every required variable above, submit a finite one-image job:

python accesspath.py pipeline -- \
  --image inputs/scene.jpg \
  --category stairs \
  --output-dir outputs/stairs_full

The command submits one GPU job, waits by default, saves its results, and exits when the job terminates. Add --no-wait to return after submission. The output review order is documented in REPRODUCIBLE_PIPELINE.md.

6. Citation and related-work links

Use the official paper/model pages below when citing an external dependency.

@article{wu2025amodal3r,
  title={Amodal3R: Amodal 3D Reconstruction from Occluded 2D Images},
  author={Wu, Tianhao and Zheng, Chuanxia and Guan, Frank and Vedaldi, Andrea and Cham, Tat-Jen},
  journal={arXiv preprint arXiv:2503.13439},
  year={2025}
}

@inproceedings{wang2025vggt,
  title={VGGT: Visual Geometry Grounded Transformer},
  author={Wang, Jianyuan and Chen, Minghao and Karaev, Nikita and Vedaldi, Andrea and Rupprecht, Christian and Novotny, David},
  booktitle={CVPR},
  year={2025}
}

@inproceedings{ozguroglu2024pix2gestalt,
  title={pix2gestalt: Amodal Segmentation by Synthesizing Wholes},
  author={Ozguroglu, Ege and others},
  booktitle={CVPR},
  year={2024}
}

@inproceedings{ao2025open,
  title={Open-World Amodal Appearance Completion},
  author={Ao, Jiayang and Jiang, Yanbei and Ke, Qiuhong and Ehinger, Krista A.},
  booktitle={CVPR},
  year={2025}
}

@inproceedings{zhan2024amodal,
  title={Amodal Ground Truth and Completion in the Wild},
  author={Zhan, Guanqi and Zheng, Chuanxia and Xie, Weidi and Zisserman, Andrew},
  booktitle={CVPR},
  year={2024}
}

@article{yang2024depthanythingv2,
  title={Depth Anything V2},
  author={Yang, Lihe and others},
  journal={arXiv:2406.09414},
  year={2024}
}

For convenience, the canonical links are: Amodal3R paper, VGGT paper, pix2gestalt paper, Open-World AMODAL paper, Amodal Completion in the Wild paper, and Depth Anything V2 paper.