docs/workflow_notes (1).md

[AGENTARIUM_ASSET]
Name: Workflow Notes — Implementation Guide
Version: v1.1.1
Status: Draft

Purpose
This document explains how to implement the Gardenier package in a real environment:
- core files (system prompt, reasoning template, personality fingerprint, guardrails)
- memory
- datasets + optional knowledge map
- vector database ingestion (embed + upsert)
- execution flow using either LangChain or n8n

-------------------------------------------------------------------------------

1) Package Assembly (Core Files)

1.1 Files and roles
- core/system_prompt.md:
  The identity + operating rules of Gardenier.
- core/reasoning_template.md:
  The deterministic pipeline Gardenier follows.
- core/personality_fingerprint.md:
  The voice/behavior constraints (professional, neutral, structured).
- guardrails/guardrails.md:
  Hard safety constraints and non-execution rules.

1.2 How to combine them at runtime
When you run Gardenier, assemble a single “Gardenier Runtime Prompt” as:
A) System Prompt
B) Guardrails (hard constraints)
C) Reasoning Template (pipeline rules)
D) Personality Fingerprint (tone/behavior dial)
E) Output Format Enforcement (SPO structure)

Implementation rule:
- System + Guardrails must be treated as highest priority.
- Personality never overrides Guardrails.
- Reasoning template governs the compiler loop and output structure.

-------------------------------------------------------------------------------

2) Datasets and Knowledge Map (RAG Layer)

2.1 Required datasets (minimum)
- domain_type_catalog.csv
- latent_constraints_signals.csv
- prompt_template_catalog.csv
- tone_policy.csv
- validation_rules.csv

2.2 Optional knowledge map
A knowledge map is a lightweight entity graph describing package primitives.
Use it if you want better recall and safer expansions.
Typical entities:
- domain_type
- template_id
- tone_id
- validation_rule_id
- latent_constraint_type

Typical relations:
- domain_type -> uses_template -> template_id
- domain_type -> recommended_tone -> tone_id
- domain_type -> requires_validation -> rule_id
- latent_signal -> implies_constraint -> constraint_rule

2.3 Document normalization before embedding
Before embedding, convert each dataset row into a canonical text record.

Example record format:
[ROW]
dataset=validation_rules
rule_id=VAL_REQUIRED_SECTIONS
rule_type=completeness
severity=critical
description=...
fix_hint=...

Store metadata alongside each record:
- dataset_name
- primary_id (rule_id/template_id/tone_id)
- domain_type (if present)
- severity (if present)
- version

-------------------------------------------------------------------------------

3) Vector Database Upsert (Embed + Index)

3.1 Choose a vector DB
Any vector store works (Pinecone, Qdrant, Weaviate, Chroma, pgvector).
You need:
- an embeddings model
- a vector index/collection
- metadata filters (recommended)

3.2 Step-by-step upsert procedure
Step 1: Load CSV files
- Read each CSV row.
- Validate required columns exist (schema check).

Step 2: Convert each row to a document
- Use the canonical record format (Section 2.3).

Step 3: Generate embeddings
- For each document text, generate an embedding vector.

Step 4: Upsert into the vector DB
- Use a stable ID: {dataset_name}:{primary_id}
- Store metadata (dataset_name, domain_type, severity, version).

Step 5: Verify retrieval
- Query examples:
  - required SPO sections
  - tone policy for executive brief
  - high severity safety constraints
- Confirm top hits match expected rows.

3.3 Index strategy (recommended)
- One index/collection for the package: gardenier_knowledge
- Use metadata filtering by dataset_name to retrieve targeted signals.
- Retrieve at least:
  - templates + domain types (routing)
  - validation rules (integrity)
  - latent constraint signals (inference)
  - tone policies (style)

-------------------------------------------------------------------------------

4) Memory Implementation (Session Memory)

4.1 Memory scope rule
- Default: session-only memory (recommended).
- Store only what improves compilation accuracy.

4.2 Memory fields (minimum)
- session_id
- last_seed
- last_domain_type
- latent_constraints (carryover)
- constraints_carryover (carryover)
- tone_preference (optional)

4.3 Memory usage rule
- Memory must not override new explicit user requirements.
- Memory can only:
  - suggest default tone
  - carry constraints like no hype, strict format
  - maintain continuity across turns

-------------------------------------------------------------------------------

5) Running Gardenier with RAG (Compilation Logic)

5.1 Retrieval plan (what to fetch)
Given seed text:
A) Retrieve domain routing hints:
- domain_type_catalog + prompt_template_catalog
B) Retrieve latent constraint patterns:
- latent_constraints_signals
C) Retrieve tone options:
- tone_policy
D) Retrieve validation constraints:
- validation_rules

5.2 Compilation loop
- Parse seed -> infer candidate domain_type.
- Retrieve top-k rows per dataset (k small: 3–8).
- Compile a draft SPO using the selected template.
- Run validation checks (based on retrieved rules).
- If invalid, rewrite and re-check.
- Output exactly one SPO.

-------------------------------------------------------------------------------

6) Implementation in LangChain (Reference Build)

6.1 Components
- LLM: the model you use for Gardenier
- Retriever: vectorstore.as_retriever()
- Prompt assembly: merge core files + retrieved snippets
- Memory: session store (ConversationBuffer or custom store)
- Output parser: ensure SPO structure (regex/section checks)

6.2 Minimal steps
Step 1: Load core files as strings.
Step 2: Load vector store retriever (gardenier_knowledge).
Step 3: On each request:
  - retrieve relevant rows (filters by dataset_name)
  - assemble Gardenier Runtime Prompt (core + retrieved context)
  - call LLM
  - validate structure (required sections)
  - if fail, retry once with repair instruction
Step 4: return SPO.

6.3 Guardrail enforcement
- Hard-code a post-check that rejects outputs containing:
  - tool use claims (I searched, I emailed, I executed)
  - missing required headings
- If violated: re-run with repair to comply instruction.

-------------------------------------------------------------------------------

7) Implementation in n8n (Reference Build)

7.1 High-level workflow
Workflow: Gardenier Compiler
1) Trigger (Webhook / Chat input)
2) Load core files (static text nodes or file read)
3) Retrieve knowledge (Vector DB query node / HTTP request)
4) Assemble prompt (Set/Function node)
5) Call LLM (OpenAI/LLM node)
6) Validate output (IF node + Function validator)
7) If invalid -> Repair call (LLM node once) -> Validate again
8) Return SPO (Webhook response)

7.2 Step-by-step
Step 1: Trigger node receives {seed, optional context, session_id}.
Step 2: Retrieve from vector DB:
- Query = seed
- Filter dataset_name in batches:
  - domain_type_catalog + prompt_template_catalog
  - latent_constraints_signals
  - tone_policy
  - validation_rules
Step 3: Assemble a single prompt:
- System: system_prompt + guardrails
- Developer/message body: reasoning_template + personality + retrieved snippets
- User message: seed + context
Step 4: LLM node generates SPO.
Step 5: Validator function checks:
- required headings exist
- directives count 5–9
- no tool/action claims
Step 6: If fail -> Repair SPO to comply LLM node -> Validate again.
Step 7: Return SPO.

7.3 What to store in n8n
- session memory in a DB (Supabase/Postgres) or simple store:
  - session_id, last_domain_type, constraints_carryover, tone_preference

-------------------------------------------------------------------------------

8) How to Start Using It (Operational)

8.1 First run checklist
- Core files loaded and concatenated correctly.
- Vector DB index exists and contains embedded dataset rows.
- Retrieval returns relevant rows.
- SPO validator is active.
- Output is pasteable into Worker agent.

8.2 Example user request
Input seed:
Here’s a messy feature list for my app… make it a clean spec with milestones.

Expected output:
- domain_type = project_spec
- SPO with required sections
- directives 5–9
- output format includes a spec template

8.3 Common failure modes
- Too much retrieval noise:
  - fix by filtering by dataset_name and lowering top-k.
- SPO missing headings:
  - enforce validator + repair loop.
- Over-assumptions:
  - require more Inputs Required.

-------------------------------------------------------------------------------

9) Recommended Version Discipline
- When you expand datasets, bump dataset version and package version.
- Keep schemas stable; expand rows, not columns, unless major version bump.
- Treat templates and validation rules as core intelligence assets.