UrbanOmniDetect

Calibration-Free View-Agnostic Monocular 3D Object Detection for Urban Scenes

Mehmet Kerem Turkcan • Devika Gumaste • Zoran Kostic
Columbia University

This repository hosts the pretrained UrbanOmniDetect checkpoints. The full training and inference code (single-image BEV, real-time video, sliced high-resolution tracking, and evaluation) lives in the GitHub repository.

Highlights

• Calibration-free 3D detection. A single model predicts 3D bounding box keypoints from a raw RGB image without camera intrinsics, depth estimation, or ground-plane priors.
• View-agnostic. One unified architecture works across ego-vehicle, infrastructure, and aerial drone viewpoints.
• State-of-the-art on monocular KITTI. AP_3D = 30.71 and AP_BEV = 35.19 on the Moderate split at IoU >= 0.7, outperforming calibration-dependent baselines on Moderate and Hard.
• Real-time. Under 11 ms inference on an A100 GPU with TensorRT at 640x640.
• Robust. Calibration-dependent methods lose over 80% accuracy with a 5% focal-length error. Our method is invariant by construction.

KITTI Benchmark

Method	AP3D Easy	AP3D Mod.	AP3D Hard	APBEV Easy	APBEV Mod.	APBEV Hard
MonoDGP	30.76	22.34	19.02	39.40	28.20	24.42
MonoCon	26.33	19.01	15.98	34.65	25.39	21.93
MonoLSS	25.91	18.29	15.94	34.70	25.36	21.84
DEVIANT	24.63	16.54	14.52	32.60	23.04	19.99
Ours	29.61	30.71	27.76	33.86	35.19	31.38

Available Checkpoints

Checkpoints are named urbanomnidetect_<backbone><-head>_<imgsz>.pt and cover YOLOv8, YOLOv9, YOLO11, and YOLO12 across all scales (n / s / m / l / x) and head configurations (default, P1, P2, P6). The P2 feature-pyramid head is the most accurate; the YOLO11x-P2 family is recommended.

Checkpoint	Backbone	Input	Use case
checkpoints/urbanomnidetect_yolo11x-p2_1920.pt	YOLO11x + P2	1920	Highest accuracy (recommended)
checkpoints/urbanomnidetect_yolo11x-p2_1280.pt	YOLO11x + P2	1280	Accuracy / speed balance
checkpoints/urbanomnidetect_yolo11x-p2_640.pt	YOLO11x + P2	640	Real-time video and BEV
checkpoints/urbanomnidetect_yolo11n-p2_640.pt	YOLO11n + P2	640	Edge / lowest latency

See the Files and versions tab for the complete checkpoint list.

Quick Start

Install the dependencies:

pip install ultralytics scipy scikit-learn opencv-python matplotlib

Download the recommended checkpoint from this repository:

huggingface-cli download mehmetkeremturkcan/UrbanOmniDetect \
    checkpoints/urbanomnidetect_yolo11x-p2_1920.pt \
    --local-dir .

To download all checkpoints at once:

huggingface-cli download mehmetkeremturkcan/UrbanOmniDetect --local-dir .

Run Inference on a Single Image

The model follows the standard Ultralytics prediction API. Point it at any image, from any viewpoint, and it will predict 3D bounding box keypoints without requiring camera parameters.

from ultralytics import YOLO

model = YOLO("checkpoints/urbanomnidetect_yolo11x-p2_1920.pt")
results = model.predict("your_image.jpg", imgsz=1920, conf=0.1, device="cuda:0")

Each detected object has 8 ordered keypoints representing the 2D projections of its 3D bounding box corners. Indices 0 to 3 are the top corners and indices 4 to 7 are the ground-contact corners.

Bird's-Eye View and Real-Time Video

The bird's-eye view, real-time tracking, and sliced high-resolution pipelines (draw_bev.py, bev_realtime.py, sahi_tracker.py) are provided in the GitHub repository. The BEV head maps ground-contact keypoints to a top-down plane through an orthogonality-constrained homography, so no camera calibration is needed.

python bev_realtime.py \
    --input drone_manhattan.mp4 \
    --kp-model checkpoints/urbanomnidetect_yolo11x-p2_640.pt \
    --kp-imgsz 640 \
    --aux-model yolo26x.pt \
    --device cuda:0 \
    --export tensorrt \
    --output bev.mp4

Dataset

The models are trained on UrbanOmniView, a 37,446-frame dataset that combines ego-vehicle (KITTI), infrastructure (DAIR-V2X), and synthetic (UE5) viewpoints. The synthetic UE5 portion is released as part of this work.

Citation

If you use UrbanOmniDetect or the UrbanOmniView dataset in your research, please cite:

@inproceedings{turkcan2026urbanomnidetect,
  title     = {Calibration-Free View-Agnostic Monocular 3D Object Detection for Urban Scenes},
  author    = {Turkcan, Mehmet Kerem and Gumaste, Devika and Kostic, Zoran},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
  year      = {2026}
}

Acknowledgements

This work was supported by the NSF Engineering Research Center for Smart Streetscapes under Award EEC-2133516, NSF Grants CNS-2450567 and CNS-2038984, and by computing resources from the NVIDIA Academic Grant Program and the Empire AI Consortium.

License

This project is released under the GNU Affero General Public License v3.0.