Proof: Added local Docker build verification and logs

ff76809 5 months ago

24.2 kB

title: Drone Delivery Env — Drone Delivery OpenEnv
emoji: 🚁
colorFrom: indigo
colorTo: purple
sdk: docker
app_port: 8000
pinned: true
license: mit
tags:
  - reinforcement-learning
  - drone-navigation
  - openenv
  - deep-q-network
  - fastapi
  - pytorch
  - autonomous-agents
  - grid-world
short_description: Autonomous drone delivery RL environment.

🚁 Drone Delivery Env

Autonomous Neural Navigation Framework

A high-fidelity, end-to-end Reinforcement Learning environment for training and evaluating autonomous drone delivery agents in procedurally generated urban grids.

🌐 Live Demo · 📖 API Docs · 📦 PyPI · 🐛 Issues

Overview

Drone Delivery Env is a production-grade, OpenEnv-compatible simulation framework designed for research in deep reinforcement learning and autonomous decision-making. It provides a realistic urban delivery scenario where agents must navigate procedurally generated city grids, avoid obstacles, manage battery resources, and complete multi-waypoint delivery missions.

The framework supports three operational modes:

Mode	Description	Entry Point
Deep RL Training	Train a `PathQNet` DQN agent from scratch	`train.py`
LLM-Guided Inference	Drive the agent via any OpenAI-compatible LLM (e.g., Qwen, GPT-4)	`inference.py`
Interactive Server	REST API + browser-based dashboard	`server/app.py`

System Architecture

The codebase follows a clean separation-of-concerns architecture across four distinct layers:

drone_env/
│
├── core/                        # Physics & simulation engine
│   ├── drone.py                 # Movement kinematics, battery drain
│   ├── grid_generator.py        # Procedural city map generation (PyTorch RNG)
│   ├── obstacles.py             # Collision detection & terrain classification
│   ├── state_manager.py         # Episodic state initialization (UUID-based)
│   ├── graders.py               # Unified scoring functions per difficulty
│   └── tasks.py                 # Hyper-parameter configs: easy / medium / hard
│
├── rl/                          # Neural intelligence layer
│   ├── model.py                 # MapEncoder CNN + PathQNet DQN architecture
│   ├── policy.py                # ε-greedy policy with linear epsilon decay
│   └── trainer.py               # Experience replay, episode analytics, inference
│
├── server/                      # REST API + frontend
│   ├── app.py                   # FastAPI application, middleware, all endpoints
│   ├── grid_world_environment.py # DroneDeliveryEnvironment (OpenEnv interface)
│   ├── drone_env_environment.py # Legacy environment wrapper
│   ├── map_generator.py         # Map utility helpers
│   ├── Dockerfile               # Multi-stage production container
│   └── static/                  # Browser-based interactive dashboard
│       ├── index.html
│       ├── script.js
│       └── style.css
│
├── models.py                    # Pydantic schemas: DroneAction, DroneObservation, DroneState
├── train.py                     # Standalone DQN training loop
├── inference.py                 # LLM-agent inference runner (OpenAI-compatible)
├── client.py                    # Python SDK client for the REST API
├── openenv.yaml                 # OpenEnv Space manifest
├── pyproject.toml               # Package metadata and dependencies
└── validate-submission.sh       # Hugging Face submission validator

Component Interaction Flow

LLM / RL Agent
      │
      │  HTTP POST /step  {direction: "UP"}
      ▼
┌─────────────────────────────────┐
│   FastAPI Server  (app.py)      │
│   ┌──────────────────────────┐  │
│   │  DroneDeliveryEnvironment│  │
│   │  ┌────────┐ ┌──────────┐ │  │
│   │  │  grid_ │ │ core/*   │ │  │
│   │  │ world  │ │ physics  │ │  │
│   │  └────────┘ └──────────┘ │  │
│   └──────────────────────────┘  │
└─────────────────────────────────┘
      │
      │  DroneObservation (JSON)
      ▼
  Agent processes state → next action

Environment Mechanics

Grid World

Maps are procedurally generated using a seeded PyTorch Generator ensuring reproducibility. Each cell on the grid is one of seven types:

Emoji	Type	Effect
🚁	Drone	Agent's current position
🛣	Road	Safe traversal (no penalty)
🏢	Building	Passable with penalty (`r_building`)
🌳	Tree	Passable with penalty (`r_tree`)
🚧	Obstacle	Passable with penalty (`r_obstacle`)
📦	Delivery Target	Collect for delivery reward
✅	Delivered	Completed delivery waypoint

Action Space

The agent selects one discrete action per timestep:

UP | DOWN | LEFT | RIGHT | WAIT

Out-of-bound moves (hitting grid walls) are penalized but keep the agent in place.

Observation Space

Each DroneObservation returned after every step contains:

class DroneObservation(BaseModel):
    grid: List[str]              # Rendered emoji grid rows
    cell_types: List[List[str]]  # Raw cell type matrix (for neural input)
    grid_width: int
    grid_height: int
    drone_x: int                 # Current drone column
    drone_y: int                 # Current drone row
    battery: float               # Normalized battery 0.0–1.0
    battery_steps_remaining: int
    deliveries_total: int
    deliveries_done: int
    current_target: Optional[Tuple[int, int]]
    distance_to_target: Optional[float]  # Manhattan distance
    step_count: int
    max_steps: int
    reward_last: float
    reward_total: float
    score: float                 # Graded score 0–100
    done: bool
    message: str
    legend: Dict[str, str]

Difficulty Levels

Parameter	`easy_delivery`	`medium_delivery`	`hard_delivery`
Grid Size	10 × 10	14 × 14	18 × 18
Buildings	4	8	12
Trees	4	6	10
Obstacles	3	6	10
Deliveries	1	3	5
Max Steps	60	100	160
Battery	60	100	160
`r_delivery`	+1.0	+0.8	+0.6
`r_battery_dead`	−0.5	−0.5	−1.0

Neural Intelligence Layer

PathQNet Architecture

The neural model (rl/model.py) is a dual-input Deep Q-Network that fuses spatial map understanding with agent telemetry:

Input 1: cell_ids  (B, H×W)          Input 2: telemetry (B, 5)
         │                                      │
         ▼                                      │
  ┌──────────────────┐                          │
  │  MapEncoder CNN  │                          │
  │  Embedding(8)    │                          │
  │  Conv2d(8→16)    │                          │
  │  Conv2d(16→32)   │                          │
  │  AdaptiveAvgPool │                          │
  │  Linear → 64     │                          │
  └──────────────────┘                          │
         │  map_emb (B, 64)                     │
         └──────────────────────────────────────┘
                          │ concat (B, 69)
                          ▼
                  ┌───────────────┐
                  │  PathQNet MLP │
                  │  Linear(128)  │
                  │  LayerNorm    │
                  │  ReLU         │
                  │  Linear(128)  │
                  │  Linear(64)   │
                  │  Linear(5)    │  ← Q-values for 5 actions
                  └───────────────┘

Telemetry vector (5 dims):

drone_x / grid_width — normalized column position
drone_y / grid_height — normalized row position
battery — normalized battery level (0–1)
target_x / grid_width — normalized target column
target_y / grid_height — normalized target row

Epsilon-Greedy Policy

Linear epsilon decay from 1.0 → 0.05 over a configurable number of steps (rl/policy.py):

EpsilonGreedyPolicy(eps_start=1.0, eps_end=0.05, decay_steps=5000)

At each decision point, with probability ε the agent explores randomly; otherwise it selects argmax Q(s, a).

API Reference

The FastAPI server exposes the full OpenEnv-compatible interface. Access interactive docs at http://localhost:8000/docs.

Core Environment Endpoints

Method	Endpoint	Description
`POST`	`/reset`	Reset episode; optionally specify `task_name`
`POST`	`/step`	Execute one action; returns `DroneObservation`
`GET`	`/state`	Retrieve current `DroneState`
`GET`	`/grade/{task_name}`	Get graded score (0.0–1.0)

Analytics & Monitoring

Method	Endpoint	Description
`GET`	`/analyse/{task_name}`	Episode statistics from `memory.json`
`GET`	`/path_history`	Step-by-step trajectory of current episode
`GET`	`/memory_logs`	Last 5 episode summaries
`GET`	`/logs`	Last 50 lines from `data/train.log`
`GET`	`/terminal_logs`	Live HTTP request log stream

Utility

Method	Endpoint	Description
`GET`	`/tasks`	List all task configs
`POST`	`/predict`	Get next action from trained model
`GET`	`/health`	Health check → `{"status": "ok", "version": "0.2.1"}`
`GET`	`/`	Browser dashboard (interactive UI)

Quickstart

Prerequisites

Python ≥ 3.10
uv package manager (recommended) or pip
PyTorch ≥ 2.0

Local Installation

# Clone the repository
git clone https://huggingface.co/spaces/manikandan-n-07/drone_env

# Install with uv (recommended — uses uv.lock for reproducibility)
uv sync

# Or with pip
pip install -e ".[dev]"

Launch the Server

# Using uv (recommended)
uv run server --port 8000

# Or directly
python -m uvicorn server.app:app --host 0.0.0.0 --port 8000

Open http://localhost:8000 to access the interactive dashboard.

Training Manual

To train the drone agent, use the train.py script with the corresponding task and episode count:

# Easy: Train for basic navigation (1000 episodes)
python train.py --task easy_delivery --episodes 1000

# Medium: Train for longer routes and more targets (2000 episodes)
python train.py --task medium_delivery --episodes 2000

# Hard: Train for high-density obstacle navigation (5000 episodes)
python train.py --task hard_delivery --episodes 5000

Python SDK Client

from client import DroneEnvClient

with DroneEnvClient("http://localhost:8000") as client:
    # Check server health
    print(client.health())

    # Run a random episode for smoke-testing
    result = client.run_random_episode("easy_delivery", verbose=True)
    print(f"Score: {result['score']:.4f}")

    # Manual episode loop
    obs = client.reset("hard_delivery")
    while not obs["done"]:
        obs = client.step("RIGHT")   # or UP / DOWN / LEFT / WAIT
    
    analytics = client.analyse("hard_delivery")
    print(analytics)

Training

DQN Training Loop

Train a PathQNet agent with experience replay:

# Easy task — good for initial validation
python train.py --task easy_delivery --episodes 500

# Medium task — balanced challenge
python train.py --task medium_delivery --episodes 1000

# Hard task — full complexity, GPU recommended
python train.py --task hard_delivery --episodes 2000 --gpu

Hyperparameters (configurable in train.py):

Parameter	Value	Description
`GAMMA`	0.99	Discount factor
`BATCH_SIZE`	64	Experience replay batch size
`LR`	1e-4	Adam optimizer learning rate
`REPLAY_SIZE`	10,000	Replay buffer capacity
`TARGET_UPDATE`	10	Episodes between target network sync
`EPS_START`	1.0	Initial exploration rate
`EPS_END`	0.05	Minimum exploration rate
`EPS_DECAY`	0.995	Multiplicative decay per episode

Checkpointing & Resumption

Models are saved automatically every 50 episodes to data/{task_short}.pth:

data/easy.pth    ← easy_delivery checkpoint
data/medium.pth  ← medium_delivery checkpoint
data/hard.pth    ← hard_delivery checkpoint

Training automatically resumes from the latest checkpoint if one exists. To force fresh training, delete the corresponding .pth file.

Training Logs

Monitor training progress in real time:

# Live log stream
tail -f data/train.log

# Or via the API
curl http://localhost:8000/logs

LLM-Powered Inference

inference.py provides a fully OpenAI-compatible runner that drives the drone environment using any hosted LLM.

Configuration

Set the following environment variables (or edit .env):

# Option A: Hugging Face Inference Router (default — free tier)
HF_TOKEN=hf_your_token_here

# Option B: OpenAI-compatible endpoint
OPENAI_API_KEY=sk-your-key
API_BASE_URL=https://api.openai.com/v1

# Model selection (default: Qwen/Qwen2.5-7B-Instruct)
MODEL_NAME=Qwen/Qwen2.5-72B-Instruct

# Task difficulty
DRONE_TASK=easy_delivery

Supported Models

Model	Provider	Tier	Notes
`Qwen/Qwen2.5-7B-Instruct`	HF Router	Free	Fast, good baseline
`Qwen/Qwen2.5-72B-Instruct`	HF Router	Credits	High capability
`Qwen/QwQ-32B-Preview`	HF Router	Credits	Reasoning-optimized
`gpt-4o`	OpenAI	Paid	Reference performance

Run Inference

# Using HF token (set in .env)
python inference.py

# Override model at runtime
MODEL_NAME=Qwen/Qwen2.5-72B-Instruct python inference.py

Output Format

The runner emits structured benchmark-compatible log lines:

[START] task=easy_delivery env=drone_env_v1 model=Qwen/Qwen2.5-7B-Instruct
[STEP] step=1 action=RIGHT reward=-0.05 done=false error=null
[STEP] step=2 action=DOWN reward=-0.05 done=false error=null
...
[END] success=true steps=23 score=0.847 rewards=-0.05,-0.05,1.00,...

System Prompt

The LLM receives a minimal, action-focused system prompt:

You are a drone navigation AI. Your goal is to deliver all packages.
Actions: UP, DOWN, LEFT, RIGHT, WAIT.
Respond with exactly ONE action name in uppercase.

And a concise per-step user prompt with position, battery, target, and distance.

Docker Deployment

Build & Run Locally

# Build from the server/ directory
docker build -t drone-env -f server/Dockerfile .

# Run with health check
docker run -p 8000:8000 \
  -e HF_TOKEN=hf_your_token \
  drone-env

Local Build Verification

This repository's Docker environment has been verified locally on desktop-linux.

Metric	Value
Status	✅ Completed
Duration	29m 38s
Revision	`b958aaf`
Platform	linux/amd64

data/docker_build.log contains the full verification history.

Multi-Stage Build Details

The server/Dockerfile uses a two-stage build:

Builder stage — installs all Python dependencies via uv sync with layer caching
Runtime stage — copies only the virtual environment and application code

# Health check built in
HEALTHCHECK --interval=30s --timeout=3s \
  CMD curl -f http://localhost:8000/health || exit 1

# Entrypoint
CMD ["sh", "-c", "cd /app/env && uvicorn server.app:app --host 0.0.0.0 --port 8000"]

Hugging Face Submission

Space Manifest (`openenv.yaml`)

spec_version: 1
name: drone-env
type: space
runtime: fastapi
app: server.app:app
port: 8000

Validate Before Submission

The included validator script checks three things end-to-end:

HF Space is live — pings your Space's /reset endpoint
Docker build succeeds — runs a local docker build
OpenEnv validation passes — runs openenv validate

chmod +x validate-submission.sh

# Usage
./validate-submission.sh https://your-space.hf.space [./repo-dir]

# Example
./validate-submission.sh https://manikandan-n-07-drone-env.hf.space .

Windows (PowerShell) Validation

If you are on Windows, run these steps manually to validate your Space:

# 1. Ping the Space
Invoke-RestMethod -Method Post -Uri "https://manikandan-n-07-drone-env.hf.space/reset" -ContentType "application/json" -Body '{}'

# 2. Local Docker Build
docker build .

# 3. OpenEnv Validate
openenv validate

A passing run produces: ```

All 3/3 checks passed! Your submission is ready to submit.


### Push to Hugging Face Hub

```bash
# Install the HF CLI
pip install huggingface_hub

# Login
huggingface-cli login

# Create a new Space (Docker SDK)
huggingface-cli repo create drone-env --type space --space-sdk docker

# Add the HF remote and push
git remote add hf https://huggingface.co/spaces/manikandan-n-07/drone-env
git push hf main

Reward Engineering

The environment uses a composite reward signal combining sparse terminal rewards and dense shaping:

$R_t = r_{\text{step}} + r_{\text{shaping}} + r_{\text{terminal}}$

Component	Formula	Purpose
$r_{\text{step}}$	$-0.05$ (constant)	Temporal pressure — discourages lingering
$r_{\text{shaping}}$	$\Delta d \times 0.05$	Manhattan-distance potential — dense guidance toward target
$r_{\text{wall}}$	$-0.20$	Out-of-bounds penalty
$r_{\text{obstacle}}$	$-0.10$ to $-0.20$	Terrain avoidance signal
$r_{\text{delivery}}$	$+1.0$ to $+0.6$	Sparse reward — scales with difficulty
$r_{\text{battery_dead}}$	$-0.5$ to $-1.0$	Terminal failure penalty

Reward shaping uses the potential-based function:

$r_{\text{shaping}} = (d_{\text{before}} - d_{\text{after}}) \times 0.05$

Grading & Evaluation

Scores are computed by core/graders.py using a unified formula:

$\text{score} = 0.8 \times \underbrace{\frac{\text{deliveries\_done}}{\text{deliveries\_total}}}_{\text{delivery ratio}} + 0.2 \times \underbrace{\left( 0.5 \cdot \text{battery} + 0.5 \cdot \left(1 - \frac{\text{steps}}{\text{max\_steps}}\right) \right)}_{\text{efficiency}}$

Project Structure

drone_env/
├── core/
│   ├── drone.py               # compute_next_pos(), drain_battery()
│   ├── graders.py             # grade_easy/medium/hard(), GRADERS dict
│   ├── grid_generator.py      # generate_city_map(), EMOJI, LEGEND
│   ├── obstacles.py           # check_move() → outcome, cell_type
│   ├── state_manager.py       # new_episode_state() → DroneState
│   └── tasks.py               # TASK_CONFIG dict (all difficulty params)
├── rl/
│   ├── model.py               # MapEncoder, PathQNet, ACTIONS, CELL2IDX
│   ├── policy.py              # EpsilonGreedyPolicy
│   └── trainer.py             # record_episode(), PathLearner, get_action_from_policy()
├── server/
│   ├── app.py                 # FastAPI app, all routes, TerminalLogManager
│   ├── grid_world_environment.py  # DroneDeliveryEnvironment (OpenEnv base)
│   ├── Dockerfile             # Multi-stage production image
│   └── static/                # Browser dashboard (HTML/JS/CSS)
├── data/
│   ├── memory.json            # Persisted episode history (last 100 episodes)
│   └── train.log              # Training progress log
├── tests/
│   ├── test_api.py            # API integration tests
│   └── test_env.py            # Environment unit tests
├── models.py                  # DroneAction, DroneObservation, DroneState (Pydantic)
├── train.py                   # DQN training entry point
├── inference.py               # LLM inference runner
├── client.py                  # Python HTTP client SDK
├── openenv.yaml               # HF Space manifest
├── pyproject.toml             # Package config & dependencies
└── validate-submission.sh     # Pre-submission validation script

Configuration Reference

`pyproject.toml` Dependencies

[project]
name = "drone-env"
version = "0.2.0"
requires-python = ">=3.10"
dependencies = [
    "openenv-core[core]>=0.2.1",
    "torch>=2.0.0",
    "openai>=1.0.0",
    "python-multipart>=0.0.9",
]

Environment Variables

Variable	Default	Description
`HF_TOKEN`	—	Hugging Face API token for LLM inference
`OPENAI_API_KEY`	—	OpenAI API key (alternative to HF)
`API_BASE_URL`	HF Router URL	Override LLM endpoint
`MODEL_NAME`	`Qwen/Qwen2.5-7B-Instruct`	LLM model identifier
`DRONE_TASK`	`easy_delivery`	Default task for inference runner
`LOCAL_IMAGE_NAME`	`drone-inference-v1`	Local Docker image tag

🧪 Testing

# Run all tests
uv run pytest tests/ -v

# With coverage report
uv run pytest tests/ --cov=. --cov-report=html

# Specific test files
uv run pytest tests/test_env.py -v
uv run pytest tests/test_api.py -v

🤝 Contributing

Fork the repository on Hugging Face Hub
Create a feature branch: git checkout -b feat/your-feature
Commit your changes with descriptive messages
Run the test suite and validator before submitting
Open a Pull Request against main

📄 License

This project is licensed under the MIT License. See LICENSE for details.

Build system uses Meta's BSD-licensed setuptools configuration template.

Built with 🚁 for the OpenEnv ecosystem

Advancing autonomous agent research through high-fidelity simulation