{"id": "release-eval-001", "prompt": "I need a webhook-based lead enrichment pipeline that enriches incoming leads, stores research notes, and asks a human before outreach. Recommend an orchestration architecture with tradeoffs.", "metadata": {"source": "heldout_model_eval", "source_id": "eval-001"}}
{"id": "release-eval-002", "prompt": "Compare n8n and LangGraph for a long-running document review workflow that needs retries, human approval, and durable state.", "metadata": {"source": "heldout_model_eval", "source_id": "eval-002"}}
{"id": "release-eval-003", "prompt": "Design a local-first personal knowledge assistant over a markdown vault. It needs citations, semantic search, and optional local model inference.", "metadata": {"source": "heldout_model_eval", "source_id": "eval-003"}}
{"id": "release-eval-004", "prompt": "I want model routing across local and cloud LLMs based on latency, cost, privacy, and context length. What should the orchestration layer look like?", "metadata": {"source": "heldout_model_eval", "source_id": "eval-004"}}
{"id": "release-eval-005", "prompt": "Build a GitHub issue triage assistant that labels issues, drafts replies, and escalates security-sensitive reports to a human.", "metadata": {"source": "heldout_model_eval", "source_id": "eval-005"}}
{"id": "release-eval-006", "prompt": "What orchestration pattern fits a multi-agent product research workflow with researcher, analyst, and strategist roles, but strict quality gates?", "metadata": {"source": "heldout_model_eval", "source_id": "eval-006"}}
{"id": "release-eval-007", "prompt": "I need an AI workflow observability layer that records prompts, outputs, latency, token cost, errors, and user feedback.", "metadata": {"source": "heldout_model_eval", "source_id": "eval-007"}}
{"id": "release-eval-008", "prompt": "Recommend a safe architecture for a browser automation agent that can fill forms but must never submit or send messages without approval.", "metadata": {"source": "heldout_model_eval", "source_id": "eval-008"}}
{"id": "release-eval-009", "prompt": "I need an orchestration design for a lead enrichment pipeline that ingests webhook data, enriches it with public APIs, stores the results in a vector store, and triggers notifications when a lead is ready for outreach.", "metadata": {"source": "dpo_seed", "source_id": "dpo-001"}}
{"id": "release-eval-010", "prompt": "Design a cost-conscious local workflow to build an AI developer assistant that can answer questions about my tool stack and recommend integrations.", "metadata": {"source": "dpo_seed", "source_id": "dpo-002"}}
{"id": "release-eval-011", "prompt": "Compare an n8n-based workflow versus a graph-agent approach for automating document review, extraction, and summary generation.", "metadata": {"source": "dpo_seed", "source_id": "dpo-003"}}
{"id": "release-eval-012", "prompt": "I want to route different tasks to different models based on latency and cost; how should I implement model routing for a mixed local/cloud orchestration pipeline?", "metadata": {"source": "dpo_seed", "source_id": "dpo-004"}}
{"id": "release-eval-013", "prompt": "Build a pipeline for a multi-agent finance research assistant that keeps long-term memory, creates summaries, and follows GDPR-safe data handling.", "metadata": {"source": "dpo_seed", "source_id": "dpo-005"}}
{"id": "release-eval-014", "prompt": "How should I combine a vector database, an LLM, and a webhook-based trigger system to automate customer support ticket categorization and escalation?", "metadata": {"source": "dpo_seed", "source_id": "dpo-006"}}
{"id": "release-eval-015", "prompt": "Recommend the best architecture for a self-hosted automation platform that integrates LangChain, Qdrant, and local model inference for internal engineering knowledge retrieval.", "metadata": {"source": "dpo_seed", "source_id": "dpo-007"}}
{"id": "release-eval-016", "prompt": "I need a high-reliability orchestration system for a production agent workflow with retries, observability, and failure recovery.", "metadata": {"source": "dpo_seed", "source_id": "dpo-008"}}
{"id": "release-eval-017", "prompt": "What is the simplest way to connect Hugging Face model inference, Discord webhooks, and a vector memory layer for a community support bot?", "metadata": {"source": "dpo_seed", "source_id": "dpo-009"}}
{"id": "release-eval-018", "prompt": "Design an orchestration plan for a tool that monitors GitHub repos, identifies new release notes, and generates update summaries automatically.", "metadata": {"source": "dpo_seed", "source_id": "dpo-010"}}
{"id": "release-eval-019", "prompt": "How can I create a hybrid architecture that uses local models for sensitive data and cloud models for expensive reasoning tasks?", "metadata": {"source": "dpo_seed", "source_id": "dpo-011"}}
{"id": "release-eval-020", "prompt": "Suggest the best pattern for building an automation pipeline that quickly prototypes new tools while being easy to transition into production.", "metadata": {"source": "dpo_seed", "source_id": "dpo-012"}}
{"id": "release-eval-021", "prompt": "I want to build an AI orchestration advisor for internal teams; what should the data flow look like from user request to recommended tool chain?", "metadata": {"source": "dpo_seed", "source_id": "dpo-013"}}
{"id": "release-eval-022", "prompt": "Compare using LangGraph versus a custom agent framework for a process that requires stateful decision-making across multiple steps.", "metadata": {"source": "dpo_seed", "source_id": "dpo-014"}}
{"id": "release-eval-023", "prompt": "Describe a federated orchestration approach where different teams can plug in their own microservices and local tools under a shared workflow layer.", "metadata": {"source": "dpo_seed", "source_id": "dpo-015"}}
{"id": "release-eval-024", "prompt": "Propose an architecture for a workflow that ingests Slack messages, performs semantic search over team docs, and answers follow-up questions.", "metadata": {"source": "dpo_seed", "source_id": "dpo-016"}}
{"id": "release-eval-025", "prompt": "How should I design a data pipeline that keeps the vector knowledge base current with new tool releases and changelog updates?", "metadata": {"source": "dpo_seed", "source_id": "dpo-017"}}
{"id": "release-eval-026", "prompt": "What is the best orchestration stack for a desktop agent that can launch local processes, access files, and report status through a simple UI?", "metadata": {"source": "dpo_seed", "source_id": "dpo-018"}}
{"id": "release-eval-027", "prompt": "Recommend an orchestration strategy for a tool that automates API-based SaaS workflows with conditional branching and error handling.", "metadata": {"source": "dpo_seed", "source_id": "dpo-019"}}
{"id": "release-eval-028", "prompt": "Build a plan for a personal productivity agent that uses memory, tool calling, and scheduling to keep a developer on track.", "metadata": {"source": "dpo_seed", "source_id": "dpo-020"}}
{"id": "release-eval-029", "prompt": "How can I blend RAG and fine-tuned local inference to answer detailed engineering questions with both recency and accuracy?", "metadata": {"source": "dpo_seed", "source_id": "dpo-021"}}
{"id": "release-eval-030", "prompt": "Suggest an architecture for a developer onboarding assistant that ingests internal docs, code examples, and process guides.", "metadata": {"source": "dpo_seed", "source_id": "dpo-022"}}
{"id": "release-eval-031", "prompt": "I need a pipeline to detect and alert on urgent issues across multiple monitoring sources, using AI to prioritize triage.", "metadata": {"source": "dpo_seed", "source_id": "dpo-023"}}
{"id": "release-eval-032", "prompt": "Compare an event-driven automation solution with a polling-based workflow for a system that must react to external triggers quickly.", "metadata": {"source": "dpo_seed", "source_id": "dpo-024"}}
{"id": "release-eval-033", "prompt": "What orchestration approach should I use for a model that needs to access external tools via MCP servers safely and efficiently?", "metadata": {"source": "dpo_seed", "source_id": "dpo-025"}}
{"id": "release-eval-034", "prompt": "Design an architecture for a low-latency query assistant that uses an on-device model for immediate responses and a vector store for long-term context.", "metadata": {"source": "dpo_seed", "source_id": "dpo-026"}}
{"id": "release-eval-035", "prompt": "How can I implement a knowledge update pipeline that tracks new AI tool releases and automatically refreshes the vector database?", "metadata": {"source": "dpo_seed", "source_id": "dpo-027"}}
{"id": "release-eval-036", "prompt": "Suggest a workflow for a proof-of-concept automation MVP that can later scale into a robust production orchestration platform.", "metadata": {"source": "dpo_seed", "source_id": "dpo-028"}}
{"id": "release-eval-037", "prompt": "Recommend the best orchestration pattern for integrating plugin-style tool calls into a conversational agent with stateful context.", "metadata": {"source": "dpo_seed", "source_id": "dpo-029"}}
{"id": "release-eval-038", "prompt": "Which architecture is best for a high-signal developer advisory assistant that balances actionable recommendations with safe, conservative defaults?", "metadata": {"source": "dpo_seed", "source_id": "dpo-030"}}
{"id": "release-eval-039", "prompt": "Which tool should I use for webhook-based workflow automation?", "metadata": {"source": "retrieval_smoke", "source_id": "retrieval-001"}}
{"id": "release-eval-040", "prompt": "How do I build a stateful multi-step agent?", "metadata": {"source": "retrieval_smoke", "source_id": "retrieval-002"}}
{"id": "release-eval-041", "prompt": "Compare n8n vs LangGraph for orchestration.", "metadata": {"source": "retrieval_smoke", "source_id": "retrieval-003"}}
{"id": "release-eval-042", "prompt": "Compare AutoGen and Semantic Kernel for multi-agent orchestration.", "metadata": {"source": "retrieval_smoke", "source_id": "retrieval-004"}}
{"id": "release-eval-043", "prompt": "Should I use Chroma or Weaviate as a vector database?", "metadata": {"source": "retrieval_smoke", "source_id": "retrieval-005"}}
{"id": "release-eval-044", "prompt": "What vector database setup should I use for semantic search?", "metadata": {"source": "retrieval_smoke", "source_id": "retrieval-006"}}
{"id": "release-eval-045", "prompt": "How should I evaluate RAG quality with Promptfoo or Ragas?", "metadata": {"source": "retrieval_smoke", "source_id": "retrieval-007"}}
{"id": "release-eval-046", "prompt": "What should I use for observability and traces in LLM applications?", "metadata": {"source": "retrieval_smoke", "source_id": "retrieval-008"}}
{"id": "release-eval-047", "prompt": "Should I use vLLM or llama.cpp for local inference serving?", "metadata": {"source": "retrieval_smoke", "source_id": "retrieval-009"}}
{"id": "release-eval-048", "prompt": "How do I run local models for a private developer assistant?", "metadata": {"source": "retrieval_smoke", "source_id": "retrieval-010"}}
{"id": "release-eval-049", "prompt": "Which tool can route model calls across providers by cost and latency?", "metadata": {"source": "retrieval_smoke", "source_id": "retrieval-011"}}
{"id": "release-eval-050", "prompt": "Which MCP server should expose GitHub and Playwright tools safely?", "metadata": {"source": "retrieval_smoke", "source_id": "retrieval-012"}}