Hayula Gateway: A Unified OpenAI-Compatible API Layer for Multi-Model Local Inference — Hayula Research [Hayula Research](/) [Papers](https://research.hayula.xyz) [Blog](https://blog.hayula.xyz) [Git](https://git.hayula.xyz) [HF](https://huggingface.co/HayulaLabs) Infrastructure · Paper H3 # Hayula Gateway: A Unified OpenAI-Compatible API Layer for Multi-Model Local Inference Yahya Saqban, Hayula AI Lab · June 2026 · Hayula AI Lab ## Hayula Gateway: A Unified OpenAI-Compatible API Layer for Multi-Model Local Inference ## Abstract We present **Hayula Gateway**, a unified API routing layer that provides OpenAI-compatible endpoints for multiple locally-hosted large language models running on a single Apple M2 Ultra workstation (192GB unified memory). The gateway orchestrates three distinct model services—**SAIF** (a multi-expert cybersecurity suite on port 8443), **Averroes-Q** (a bilingual Arabic-English model on port 8080), and **DeepSeek R1** (a reasoning-optimized model on port 8083)—exposing them through a single `/v1/chat/completions` interface. Hayula Gateway implements intelligent request routing, load-aware dispatching, LRU-based model lifecycle management with hot-swap capability, and configurable fallback chains (SAIF → DeepSeek R1 → OpenRouter) for reliability. The system achieves zero cloud dependency for primary inference while maintaining sub-200ms routing overhead on the M2 Ultra's 76-core GPU. We describe the architecture, routing strategies, memory management under the 192GB unified memory constraint, and production deployment via launchd on macOS. Hayula Gateway is the central inference fabric of the Hayula ecosystem, demonstrating that a single consumer workstation can replace multiple cloud API subscriptions for diverse LLM workloads. ## 1 Introduction The proliferation of large language models has created a fragmented landscape where practitioners must manage multiple API endpoints, authentication schemes, and provider-specific formats [1]. Cloud-based solutions offer convenience but introduce recurring costs, data privacy concerns, and dependency on external infrastructure. Meanwhile, local inference on consumer hardware has matured significantly—models from 7B to 70B parameters now run efficiently on systems like the Apple M2 Ultra with its 192GB unified memory [2]. However, running multiple specialized models locally introduces a new challenge: how to coordinate diverse model services behind a single, consistent interface. **Hayula Gateway** addresses this challenge by providing an OpenAI-compatible API layer that routes requests to three locally-hosted model services: - **SAIF** (port 8443): A multi-expert cybersecurity suite comprising a router model and eight domain specialists, exposed via a custom `/v1/completions` format. Only 4 of 9 total models are loaded simultaneously using an LRU-based hot-swap cache due to memory constraints. - **Averroes-Q** (port 8080): A bilingual (Arabic/English) instruction-tuned model for general-purpose conversation and domain-agnostic tasks. - **DeepSeek R1** (port 8083): A reasoning-optimized model for complex chain-of-thought and analytical tasks. The gateway unifies these services behind a single interface, implements intelligent routing policies, and provides fallback chains (SAIF → DeepSeek R1 → OpenRouter) that ensure reliability even when local models are reloading or at capacity. All inference occurs locally with zero cloud dependency—the OpenRouter fallback is purely an emergency measure. The name **Hayula** (هيula, Arabic for "matter" or "substance") reflects the system's role as the foundational substance from which all Hayula AI services are formed. The gateway is the material infrastructure that enables the entire ecosystem. ## 2 Related Work ### 2.1 Model Serving Infrastructure Several frameworks provide model serving infrastructure for LLMs. **vLLM** [3] offers high-throughput serving with PagedAttention for efficient KV-cache management. **Llama.cpp** [4] provides efficient CPU and GPU inference with server mode. **Ollama** [5] simplifies local model deployment with a user-friendly API. **LocalAI** [6] implements OpenAI-compatible APIs for local models. However, these systems are primarily designed for single-model serving or simple model switching, lacking sophisticated multi-model orchestration with heterogeneous service interfaces. ### 2.2 API Gateways for LLMs Commercial solutions like **OpenRouter** [7] and **Portkey** [8] provide unified API gateways for cloud-based LLMs, offering load balancing, fallback, and cost management. However, they introduce latency, require cloud connectivity, and expose all traffic to third-party infrastructure. **LiteLLM** [9] provides a Python library for unified cloud LLM access but does not address local model coordination. ### 2.3 Local Inference on Apple Silicon The Apple M2 Ultra's unified memory architecture [10] enables running large models that would exceed the VRAM of discrete GPUs. Prior work has demonstrated successful deployment of 7B–70B parameter models on M-series hardware [11]. The mlx-lm framework [12] provides optimized inference on Apple Silicon, leveraging the Metal Performance Shaders backend. Hayula Gateway extends this capability by coordinating multiple simultaneous inference services on a single machine. ### 2.4 LRU-Based Model Management Memory-constrained deployment of multiple large models has been explored in systems like **FlexGen** [13] and **S-LoRA** [14], which use offloading and dynamic loading strategies. **Petals** [15] distributes model layers across a peer-to-peer network. Hayula Gateway applies LRU-cache principles to model lifecycle management, keeping only the most recently used SAIF experts in GPU memory while swapping others from SSD storage. ## 3 System Architecture ### 3.1 Overview Hayula Gateway adopts a **multi-service proxy** architecture. It accepts requests via a single OpenAI-compatible `/v1/chat/completions` endpoint, classifies or selects the target model based on routing policy, and proxies the request to the appropriate backend service. The architecture consists of four layers: - **API Layer**: Exposes OpenAI-compatible endpoints, handles authentication, and validates request schemas. - **Routing Layer**: Determines which backend model to invoke based on request parameters, model availability, and configured policies. - **Service Layer**: Manages the lifecycle of backend model processes (SAIF, Averroes-Q, DeepSeek R1), including health checks, restart, and hot-swap. - **Fallback Layer**: Implements cascading fallback chains for reliability. ``` `┌─────────────────────────────────────────────┐ │ Client Application │ └──────────────────┬──────────────────────────┘ │ POST /v1/chat/completions ▼ ┌─────────────────────────────────────────────┐ │ Hayula Gateway (Python) │ │ ┌─────────┐ ┌──────────┐ ┌────────────┐ │ │ │ API │->│ Routing │->│ Service │ │ │ │ Layer │ │ Layer │ │ Manager │ │ │ └─────────┘ └──────────┘ └─────┬──────┘ │ │ ┌────────┴───────┐ │ │ │ Fallback Chain │ │ │ └────────────────┘ │ └──────────────────────────┬───────────────────┘ │ ┌──────────────────┼──────────────────┐ ▼ ▼ ▼ ┌───────────────┐ ┌──────────────┐ ┌──────────────┐ │ SAIF :8443 │ │ Averroes-Q │ │ DeepSeek R1 │ │ (9 models, │ │ :8080 │ │ :8083 │ │ 4 hot-loaded)│ │ (bilingual) │ │ (reasoning) │ └───────┬───────┘ └──────────────┘ └──────────────┘ │ │ ▼ ▼ ┌────────────────┐ ┌─────────────────┐ │ 8 Specialists │ │ OpenRouter API │ │ + Router │ │ (emergency only)│ └────────────────┘ └─────────────────┘` ``` ### 3.2 API Layer The API layer exposes a single endpoint: **`POST /v1/chat/completions`** The request schema is fully OpenAI-compatible: ``` `{ "model": "saif" | "averroes-q" | "deepseek-r1" | "auto", "messages": [ {"role": "system", "content": "..."}, {"role": "user", "content": "..."} ], "temperature": 0.7, "max_tokens": 2048, "stream": false }` ``` The `model` parameter determines routing behavior: - `"saif"`: Route directly to the SAIF service for cybersecurity tasks. - `"averroes-q"`: Route to the bilingual general-purpose model. - `"deepseek-r1"`: Route to the reasoning model for complex analysis. - `"auto"`: Let the gateway select the best model based on heuristic analysis of the input. Additional Hayula-specific parameters are supported: - `x-fallback-on`: When `true`, enables fallback chain if primary model fails. - `x-route-hint`: Optional routing hint (e.g., `"rev_eng"`, `"webapp"` for SAIF specialist selection). ### 3.3 Routing Layer 3.3.1 Explicit Routing When the client specifies a model name, the routing layer maps it directly to the corresponding backend service: Model NameBackendPortService Type `saif`SAIF Gateway8443Cybersecurity multi-expert `averroes-q`Averroes-Q8080Bilingual instruction `deepseek-r1`DeepSeek R18083Reasoning/Chain-of-thought 3.3.2 Auto-Routing When `model: "auto"` is specified, the routing layer employs lightweight heuristic classification using a fast text classifier (a small ONNX-based model or regex-based keyword matching) to dispatch the request: - **Keyword-based dispatch**: Queries containing security-related terms (CVE, exploit, vulnerability, malware, buffer overflow, kernel, patch, reverse engineering, etc.) are routed to SAIF. - **Reasoning indicator**: Queries containing analytical prompts (explain step by step, reason about, compare and contrast, logical deduction) are routed to DeepSeek R1. - **Default**: All other queries, particularly those in Arabic or mixed Arabic-English, are routed to Averroes-Q. The auto-router achieves approximately 85% accuracy in preliminary evaluation and can be overridden by the client. 3.3.3 SAIF Specialist Routing When routing to SAIF, the gateway may optionally select a specific specialist within the SAIF multi-expert suite. If a `x-route-hint` is provided, it maps to the appropriate specialist directly. Otherwise, the SAIF service's internal Router model (Router 7B) performs classification as described in [16]. ### 3.4 Model Lifecycle Management 3.4.1 Service Registry All backend services are registered with the gateway at startup via a configuration file: ``` `services: saif: host: localhost port: 8443 health_endpoint: /v1/health max_retries: 3 timeout: 120 fallback: deepseek-r1 models: 9 # 1 router + 8 specialists concurrent_load: 4 # LRU cache size averroes-q: host: localhost port: 8080 health_endpoint: /health max_retries: 2 timeout: 60 fallback: null # no fallback deepseek-r1: host: localhost port: 8083 health_endpoint: /health max_retries: 2 timeout: 120 fallback: openrouter` ``` 3.4.2 Health Monitoring The gateway performs periodic health checks against each backend at configurable intervals (default: 30 seconds). A service is marked as: - **Healthy**: Responds within 5 seconds with HTTP 200. - **Degraded**: Responds but with elevated latency (>10 seconds). - **Unavailable**: Fails to respond after 3 consecutive checks. - **Loading**: Model is being hot-swapped; accepts requests but returns a 503 with `Retry-After` header. 3.4.3 LRU Hot-Swap for SAIF Experts SAIF consists of 9 models (1 Router + 8 Specialists), each requiring approximately 14.2GB in fp16. Loading all 9 simultaneously would require ~128GB, which is feasible on the 192GB M2 Ultra but leaves insufficient headroom for concurrent services and system processes. Hayula Gateway addresses this through **LRU-based hot-swap caching**: - Only **4 SAIF models** are loaded concurrently in GPU memory. - The Router model is always kept resident (essential for dispatch). - The remaining 3 slots are allocated to the most recently used specialists. - When a request targets an unloaded specialist, the least recently used specialist is offloaded to SSD, and the target specialist is loaded from disk. - Hot-swap latency averages 2.5 seconds (SSD read of ~14GB at 3.5GB/s). ``` `Memory Layout (192GB Unified Memory): ┌────────────────────────────────────────────┐ │ System + Other Processes ~16 GB │ ├────────────────────────────────────────────┤ │ SAIF Router (always loaded) ~14 GB │ ├────────────────────────────────────────────┤ │ LRU Cache Slot 1 (recently used) ~14 GB │ ├────────────────────────────────────────────┤ │ LRU Cache Slot 2 (recently used) ~14 GB │ ├────────────────────────────────────────────┤ │ LRU Cache Slot 3 (recently used) ~14 GB │ ├────────────────────────────────────────────┤ │ Averroes-Q (resident) ~14 GB │ ├────────────────────────────────────────────┤ │ DeepSeek R1 (resident) ~30-70 GB │ ├────────────────────────────────────────────┤ │ Free Memory ~36-76 GB │ └────────────────────────────────────────────┘` ``` The LRU eviction policy ensures that the most frequently requested specialists remain in memory, minimizing hot-swap overhead during typical usage patterns. ### 3.5 Fallback Chain Hayula Gateway implements a configurable fallback chain for reliability. The primary chain is: **SAIF → DeepSeek R1 → OpenRouter** When a request to the primary service fails (due to timeouts, loading state, or errors), the gateway automatically retries with the next service in the chain: ``` `Request → SAIF(:8443) ├── Success → Return Response └── Failure → DeepSeek R1(:8083) ├── Success → Return Response └── Failure → OpenRouter(cloud) ├── Success → Return Response └── Failure → Return 503 Error` ``` Fallback is transparent to the client—the response always conforms to the OpenAI chat completions schema, with an additional `x-hayula-chain` header indicating the service that actually handled the request: ``` `x-hayula-chain: saif→deepseek-r1 (served by deepseek-r1)` ``` Fallback behavior is controlled per-request via the `x-fallback-on` parameter, allowing clients to opt in or out. ### 3.6 Request/Response Transformation Each backend service exposes a different API format. Hayula Gateway performs transparent request/response transformation: **SAIF (custom format):** - Request: Standard OpenAI format → transformed to SAIF's custom `/v1/completions` JSON schema, which includes specialist routing metadata. - Response: SAIF's JSON output → transformed to OpenAI chat completions response format with `choices`, `usage`, etc. **Averroes-Q (custom format):** - Request: Standard OpenAI format → transformed to Averroes-Q's server protocol. - Response: Transformed to standard OpenAI schema. **DeepSeek R1 (custom format):** - Request: Standard OpenAI format → transformed to DeepSeek R1 server protocol. - Response: Transformed to standard OpenAI schema, with reasoning content preserved in the response metadata. ## 4 Implementation ### 4.1 Hardware Platform Hayula Gateway runs on a single **Apple M2 Ultra** workstation: ComponentSpecification CPU24-core (16 performance + 8 efficiency) GPU76-core Apple GPU Memory192GB unified memory (800 GB/s bandwidth) Storage2TB SSD (~3.5 GB/s sequential read) NetworkLocalhost loopback (multi-Gbps) The unified memory architecture is critical for this deployment, as it allows all model services to coexist in the same memory pool without VRAM constraints. ### 4.2 Software Stack ComponentTechnology RuntimePython 3.11 Web FrameworkASGI (Uvicorn + Starlette/FastAPI) Async HTTP Clienthttpx (for proxying to backends) Model ServingCustom launchd-managed services (SAIF, Averroes-Q, DeepSeek R1) Process Managementlaunchd (macOS service management) ConfigurationYAML-based config file CachingLRU implementation using Python's `lru_cache` + file-level locking ### 4.3 Gateway Core The gateway is implemented as an ASGI application (approximately 1,500 lines of Python) with the following architecture: ``` `# Simplified gateway structure from fastapi import FastAPI, HTTPException from httpx import AsyncClient import asyncio from enum import Enum class ModelType(Enum): SAIF = "saif" AVVERROES_Q = "averroes-q" DEEPSEEK_R1 = "deepseek-r1" # Service registry with fallback chains SERVICE_REGISTRY = { ModelType.SAIF: { "url": "http://localhost:8443/v1/completions", "timeout": 120, "fallback": ModelType.DEEPSEEK_R1, }, ModelType.AVVERROES_Q: { "url": "http://localhost:8080/chat", "timeout": 60, "fallback": None, }, ModelType.DEEPSEEK_R1: { "url": "http://localhost:8083/chat", "timeout": 120, "fallback": ModelType.OPENROUTER, }, } app = FastAPI() client = AsyncClient(timeout=120.0) @app.post("/v1/chat/completions") async def chat_completions(request: ChatRequest): model = resolve_model(request.model) result = await dispatch_with_fallback(model, request) return result async def dispatch_with_fallback(model, request, chain=None): chain = chain or [] service = SERVICE_REGISTRY[model] transformed = transform_request(request, model) try: response = await client.post( service["url"], json=transformed ) response.raise_for_status() return transform_response(response.json(), model, chain) except Exception as e: if service["fallback"] and request.fallback_on: chain.append(model.value) return await dispatch_with_fallback( service["fallback"], request, chain ) raise HTTPException(status_code=503, detail=str(e))` ``` ### 4.4 Launchd Service Management Each backend model service is managed by a separate launchd plist, ensuring automatic restart on failure: ``` ` Label com.hayula.gateway ProgramArguments /usr/local/bin/python3.11 /opt/hayula/gateway/main.py --config /opt/hayula/gateway/config.yaml KeepAlive RunAtLoad WorkingDirectory /opt/hayula/gateway StandardOutPath /opt/hayula/gateway/logs/stdout.log StandardErrorPath /opt/hayula/gateway/logs/stderr.log ` ``` The gateway depends on the backend services being available. A separate launchd plist is configured for each backend, with the gateway configured to start after its dependencies are verified healthy. ### 4.5 SAIF LRU Hot-Swap Implementation The LRU hot-swap mechanism for SAIF experts is implemented at the SAIF service level, but the gateway is aware of the loading state and can trigger pre-warming or retry routing based on current cache state: ``` `class SAIFLRUCache: """LRU cache for SAIF experts - hot-swap on cache miss.""" def __init__(self, max_slots: int = 3, router_path: str = "/models/router"): self.max_slots = max_slots self.cache = OrderedDict() # expert_name -> loaded path self.router_loaded = False self.router_path = router_path self.lock = asyncio.Lock() async def load_expert(self, expert_name: str) -> str: """Load an expert, evicting LRU if necessary.""" async with self.lock: if expert_name in self.cache: # Cache hit - move to most recently used self.cache.move_to_end(expert_name) return self.cache[expert_name] # Cache miss - need to load and potentially evict if len(self.cache) >= self.max_slots: # Evict LRU (first item in OrderedDict) lru_name, lru_path = self.cache.popitem(last=False) await self._unload_model(lru_path) # Load the new expert expert_path = f"/models/{expert_name}" await self._load_model(expert_path) self.cache[expert_name] = expert_path return expert_path async def _load_model(self, path: str) -> None: """Load model from disk into GPU memory (~2.5s for 14GB).""" # Implementation uses mlx-lm load with Metal backend ... async def _unload_model(self, path: str) -> None: """Offload model from GPU memory.""" # Implementation releases Metal buffers ...` ``` ## 5 Evaluation ### 5.1 Routing Latency We measured end-to-end latency introduced by the gateway (excluding backend inference time): OperationP50P95P99 Request parsing + validation0.3 ms0.5 ms0.8 ms Explicit routing lookup0.1 ms0.2 ms0.4 ms Auto-routing (heuristic)2.1 ms4.5 ms8.2 ms Request transformation0.4 ms0.7 ms1.1 ms Response transformation0.3 ms0.6 ms0.9 ms **Total overhead (explicit)****1.1 ms****2.0 ms****3.2 ms** **Total overhead (auto)****3.2 ms****6.3 ms****11.4 ms** The gateway adds negligible latency relative to model inference times, which range from seconds to minutes depending on model size and generation length. ### 5.2 Fallback Performance ScenarioPrimaryFallback 1Fallback 2Total TimeChain Header All healthySAIF——12.3 s`saif` SAIF down—DeepSeek R1—18.7 s`saif→deepseek-r1` SAIF + DeepSeek down——OpenRouter14.2 s + network`saif→deepseek-r1→openrouter` SAIF loading (hot-swap)—DeepSeek R1—16.1 s`saif→deepseek-r1` Fallback to OpenRouter adds network latency (typically 1-3 seconds for the API call) but ensures service continuity. ### 5.3 SAIF LRU Cache Performance We evaluated the LRU hot-swap mechanism over a trace of 1,000 sequential requests across the 8 SAIF specialists: MetricValue Cache Hit Rate72.4% Cache Miss Rate27.6% Average Hot-Swap Time2.47 s Total Hot-Swap Time (1K requests)681 s Estimated Time Without LRU (all 8 loaded)0 s (pre-loaded) Memory Saved~70 GB (5 experts × 14 GB) The 72.4% hit rate indicates that most workloads exhibit temporal locality—users tend to query a subset of specialists repeatedly. The 27.6% miss rate incurs a 2.47-second penalty per miss, which is acceptable for non-real-time cybersecurity analysis tasks. ### 5.4 Memory Utilization We measured peak memory usage under different load scenarios: ScenarioMemory UsedFree MemoryNotes Idle (all services loaded)116 GB76 GBRouter + 3 LRU slots + Averroes-Q + DeepSeek R1 Heavy SAIF (hot-swap thrashing)130 GB62 GBPeak during cross-specialist transitions SAIF + Averroes-Q concurrent130 GB62 GBBoth services active simultaneously All services + system overhead142 GB50 GBWith macOS and background processes The system operates comfortably within the 192GB limit with approximately 50-76GB of headroom for future expansion. ## 6 Deployment ### 6.1 Service Topology All services run on a single M2 Ultra workstation, communicating over localhost loopback: ``` `┌─────────────────────────────────┐ │ M2 Ultra (localhost only) │ │ │ │ ┌─────────┐ ┌───────────────┐ │ │ │ Gateway │ │ SAIF (:8443) │ │ │ │ :8000 │ │ - Router 7B │ │ │ └────┬────┘ │ - 8 Specialists│ │ │ │ │ - LRU Cache │ │ │ ├──────┤ (4 loaded) │ │ │ │ └───────────────┘ │ │ │ ┌───────────────┐ │ │ ├──────┤ Averroes-Q │ │ │ │ │ (:8080) │ │ │ │ └───────────────┘ │ │ │ ┌───────────────┐ │ │ └──────┤ DeepSeek R1 │ │ │ │ (:8083) │ │ │ └───────────────┘ │ │ │ │ ┌──────────────────────────┐ │ │ │ launchd supervision │ │ │ │ (auto-restart on crash) │ │ │ └──────────────────────────┘ │ └─────────────────────────────────┘ │ │ (emergency only) ▼ ┌──────────────┐ │ OpenRouter │ │ (cloud API) │ └──────────────┘` ``` The gateway is exposed on port 8000, listening only on localhost (127.0.0.1) for security. Applications access it either directly (if running on the same machine) or through an SSH tunnel from remote clients. ### 6.2 Startup Sequence On system boot, launchd starts services in dependency order: - **SAIF service** (`com.hayula.saif`): Loads Router model and initializes LRU cache with the 3 most recently used specialists from persistent state. - **Averroes-Q service** (`com.hayula.averroes-q`): Loads the bilingual model on port 8080. - **DeepSeek R1 service** (`com.hayula.deepseek-r1`): Loads the reasoning model on port 8083. - **Gateway** (`com.hayula.gateway`): Performs health checks against all backends, then starts accepting requests on port 8000. Total startup time: approximately 45 seconds (dominated by model loading). ### 6.3 Client Integration Applications consume Hayula Gateway as a drop-in replacement for the OpenAI API: ``` `from openai import OpenAI # Point to local gateway instead of api.openai.com client = OpenAI( base_url="http://localhost:8000/v1", api_key="local" # gateway validates but doesn't require cloud keys ) # Route to specific model response = client.chat.completions.create( model="saif", messages=[ {"role": "user", "content": "Analyze this kernel exploit: CVE-2024-..." }] ) # Auto-route response = client.chat.completions.create( model="auto", messages=[ {"role": "user", "content": "ما هي نقاط الضعف في تطبيقات الويب؟"} ] # Auto-routed to Averroes-Q (Arabic detected) )` ``` No code changes are required beyond updating the `base_url` parameter, making Hayula Gateway a transparent migration path from cloud API subscriptions to local inference. ## 7 Limitations and Future Work ### 7.1 Limitations - **Single-Node Bottleneck**: All services run on a single M2 Ultra. While the machine is powerful, it represents a single point of failure and cannot scale horizontally. A hardware failure brings down all inference services. - **SAIF LRU Cache Miss Penalty**: The 2.47-second hot-swap time is noticeable during rapid cross-specialist queries. This penalty accumulates during workloads that rapidly cycle through many specialists. - **Auto-Routing Accuracy**: The heuristic-based auto-router achieves approximately 85% accuracy, which may route queries to suboptimal models. A trained classifier would improve this significantly. - **No Streaming Support**: The current implementation does not support OpenAI-compatible streaming (SSE). Streaming is proxied as complete responses only. - **No Rate Limiting**: The gateway does not implement request rate limiting or quality-of-service guarantees, which could lead to resource contention under high load. - **OpenRouter Dependency for Fallback**: While primary inference is entirely local, the fallback chain depends on OpenRouter (a cloud service) as the final tier. A fully local fallback would require additional redundant local hardware. ### 7.2 Future Work - **Multi-Node Distributed Gateway**: Extend the gateway to coordinate across multiple M-series workstations or Mac mini clusters for horizontal scaling and redundancy. - **Learned Auto-Router**: Replace the heuristic classifier with a lightweight trained model (e.g., DistilBERT or a small LoRA adapter) for improved auto-routing accuracy. - **Streaming Support**: Implement full SSE streaming proxying, allowing clients to receive token-by-token responses from any backend. - **Intelligent Pre-Warming**: Use request pattern analysis to predictively load specialists into the LRU cache before they are requested, reducing hot-swap miss rate. - **Quantized Model Support**: Integrate 4-bit and 8-bit quantization (GGUF, EXL2, or MLX quantization) to fit more models in memory simultaneously. - **Observability and Metrics**: Add Prometheus metrics, structured logging, and a health dashboard for monitoring gateway performance and service health. - **Plugin Architecture**: Allow third-party model services to register with the gateway dynamically, enabling a plug-and-play model ecosystem. - **Request Caching**: Implement semantic caching for identical or similar requests to reduce redundant inference for common queries. ## 8 Conclusion We introduced **Hayula Gateway**, a unified API routing layer that provides OpenAI-compatible endpoints for multiple locally-hosted large language models on a single Apple M2 Ultra workstation. The gateway orchestrates three diverse model services—SAIF (cybersecurity multi-expert), Averroes-Q (bilingual instruction), and DeepSeek R1 (reasoning)—behind a single `/v1/chat/completions` interface. Our contributions include: - **Multi-model orchestration**: A single interface for three heterogeneous model services with automatic request/response transformation. - **Intelligent routing**: Both explicit model selection and heuristic auto-routing with approximately 85% accuracy. - **LRU-based model lifecycle management**: Hot-swap caching for SAIF specialists that saves approximately 70GB of memory while maintaining 72.4% cache hit rate, with only 2.47 seconds penalty on cache misses. - **Configurable fallback chains**: Cascading fallback (SAIF → DeepSeek R1 → OpenRouter) ensuring reliability without compromising local-first operation. - **Zero cloud dependency**: Primary inference runs entirely on local hardware, replacing the need for multiple cloud API subscriptions. The gateway introduces sub-200ms overhead (typically 1-11ms depending on routing mode), operates within the M2 Ultra's 192GB memory envelope with 50-76GB headroom, and is managed through launchd for production reliability. Hayula Gateway serves as the central inference fabric of the Hayula ecosystem, demonstrating that a single consumer workstation can replace an array of cloud API subscriptions for diverse LLM workloads while maintaining OpenAI API compatibility for frictionless client integration. All code and configuration are released under the Hayula ecosystem at https://github.com/BinSaqban/hayula-gateway. ## Acknowledgments This work was conducted at Hayula AI Lab. The SAIF multi-expert cybersecurity suite [16], Averroes-Q-Instruct [17], and DeepSeek R1 [18] are the foundational model services that Hayula Gateway orchestrates. We thank the open-source communities behind FastAPI, httpx, and the Apple MLX framework for their foundational tools. Deployment and testing were performed on Apple M2 Ultra hardware provided by Hayula AI Lab. ## References [1] Zheng, L., et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." NeurIPS, 2023. [2] Apple Inc. "M2 Ultra: Apple's Most Powerful Chip." Apple Hardware Documentation, 2023. [3] Kwon, W., et al. "Efficient Memory Management for Large Language Model Serving with PagedAttention." SOSP, 2023. [4] Gerganov, G., et al. "llama.cpp: Inference of LLaMA Model in Pure C/C++." GitHub, 2023. [5] Ollama. "Get up and running with large language models locally." GitHub, 2024. [6] LocalAI. "Free, Open Source OpenAI Alternative." GitHub, 2023. [7] OpenRouter. "Unified API for LLMs." https://openrouter.ai, 2024. [8] Portkey AI. "AI Gateway and Observability." https://portkey.ai, 2024. [9] LiteLLM. "Call all LLM APIs using the OpenAI format." GitHub, 2024. [10] Apple Inc. "Unified Memory Architecture." Apple Silicon Documentation, 2022. [11] Han, D., et al. "MLX: Efficient Machine Learning on Apple Silicon." Apple Machine Learning Research, 2024. [12] mlx-lm community. "mlx-lm: LLM Inference on Apple Silicon." GitHub, 2024. [13] Sheng, Y., et al. "FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU." ICML, 2023. [14] Sheng, Y., et al. "S-LoRA: Serving Thousands of Concurrent LoRA Adapters." arXiv, 2023. [15] Borzunov, A., et al. "Petals: Collaborative Inference and Fine-tuning of Large Models." arXiv, 2023. [16] Saqban, Y. "Saif: A Multi-Expert 7B Parameter Cybersecurity Suite with Arabic-Capable Router Architecture." Hayula AI Lab, 2026. [17] Saqban, Y. "Averroes-Q-Instruct: A Bilingual Arabic-English Instruction-Tuned Model." Hayula AI Lab, 2026. [18] DeepSeek Team. "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning." arXiv, 2025. [← Back to Papers](https://research.hayula.xyz)