README.md

metadata

title: Zetema
emoji: 🏺
colorFrom: yellow
colorTo: red
sdk: gradio
sdk_version: 6.19.0
app_file: app.py
pinned: false
short_description: Semantic search over Perseus and First1KGreek

Zetema

Semantic search over the Perseus Digital Library and First1KGreek Greek corpora using local open-weight language models. Ask questions in English (or Greek) and find which ancient authors write about a given topic.

What it does

• Cross-lingual retrieval: type a query in English and retrieve matching passages in ancient Greek — the embedding model maps both into the same semantic space
• Author ranking: see at a glance which authors discuss a topic most
• Read in context: every result links to the passage in the Scaife Viewer via its CTS URN, with the English translation opened side by side when the corpus has one
• Reranked results: dense retrieval fetches a candidate pool, then a cross-encoder reranker (Qwen/Qwen3-Reranker-0.6B) orders the final results
• Century and genre filters: narrow the search to particular centuries or genres (history, tragedy, philosophy, …), both derived from the TLG Canon; result cards show each author's dates
• LLM synthesis: optionally send top passages to a local LLM (Gemma 3 by default, via Hugging Face transformers, streamed) to get a summary in English with key phrase translations
• 364 authors, 1,644 Greek works: Perseus and First1KGreek together — Plato, Aristotle, Thucydides, Homer, Aeschylus, Sophocles, Euripides, Herodotus, and many more

Requirements

• Python 3.11+
• CUDA GPU strongly recommended (reranking and synthesis run locally on it)
• ~30 GB free disk space (models + index; the default Gemma 3 12B synthesis model alone is ~24 GB)
• A Hugging Face account with the Gemma license accepted, logged in via huggingface-cli login (only needed for LLM synthesis/chat)
• 16 GB RAM recommended for the in-memory vector search

Setup

1. Clone and install dependencies

git clone <this-repo>
cd zetema
pip install -r requirements.txt
pip install torch sentence-transformers transformers numpy pandas

Install a torch build that matches your CUDA driver — a mismatched build silently falls back to CPU (check torch.cuda.is_available()). See the PyTorch install selector for the right index URL.

2. Point to the corpus

Two symlinks in the project root point at your local corpus checkouts. Both are optional individually — ingest.py skips a missing one — but the index only covers what it can find:

# example — change the targets to match your actual paths
ln -sf /path/to/perseus/xml perseus
ln -sf /path/to/First1KGreek/data first1k

Each corpus should contain directories named tlg0001, tlg0003, … each with __cts__.xml metadata and TEI XML text files. Where the two corpora carry the same work, Perseus takes precedence and the duplicate is skipped.

3. Build the index (one-time, ~10–20 min on GPU, longer on CPU)

python ingest.py

This will:

1. Download the Qwen/Qwen3-Embedding-0.6B embedding model (~1.2 GB, cached after first run)
2. Parse all Greek TEI XML files and extract sections
3. Embed every section and store the result in data/perseus_rag_qwen.duckdb

Progress is shown with a progress bar. Re-run anytime to rebuild the index from scratch.

4. Launch the app

python app.py

Opens at http://localhost:7861.

Running on Hugging Face Spaces (this branch)

This hf-space branch is directly pushable to a Space. One-time setup:

1. Upload the index to a dataset repo (private is fine — the Space authenticates with your token):

   hf repo create zetema-index --repo-type dataset --private
   hf upload zetema-index data/perseus_rag_qwen.duckdb perseus_rag_qwen.duckdb --repo-type dataset
   

2. Create the Space at huggingface.co/new-space: SDK Gradio, hardware ZeroGPU (included with Pro). Then push this branch to it:

   git remote add space https://huggingface.co/spaces/<user>/zetema
   git push space hf-space:main
   

3. Add an HF_TOKEN secret in the Space settings (a token from an account that has accepted the Gemma license) — needed both for the gated synthesis model and for the private index dataset.

At startup the app downloads the DuckDB index from the dataset repo (override the default Jacobo/zetema-index with a ZETEMA_INDEX_REPO env var) and preloads the models; GPU-heavy calls (retrieval + reranking, LLM generation) run inside ZeroGPU's @spaces.GPU windows. Locally, none of this changes anything — the app runs exactly as on main.

Usage

Query	What you find
immortality of the soul	Plato, Pythagorean fragments, Plotinus
rhetoric and persuasion	Aristotle, Gorgias, Isocrates
hubris and divine punishment	Aeschylus, Sophocles, Herodotus
democracy and tyranny	Thucydides, Xenophon, Plato
ψυχή	Query directly in Greek

Controls:

• Number of results — how many passages to retrieve (5–50)
• Filter by author / century / genre — narrow the search; the three filters combine, and century and genre come from the TLG Canon (authors without canon data land in an "Unknown" bucket)
• Synthesize results (on by default) — sends top passages to a local LLM for a streamed thematic summary with English translations of key phrases; a chat panel lets you ask follow-up questions grounded in the retrieved passages
• Color scheme — switch the UI theme live (Parchment by default)

How it works

Query (English or Greek)
        │
        ▼
  Qwen3-Embedding-0.6B           ← same model used at ingest time
  encodes query as vector
        │
        ▼
  cosine similarity over          ← all section embeddings loaded
  ~133k Greek passages              in memory at startup
        │
        ▼
  author / century / genre        ← masks from TLG Canon data
  filter masks applied              (data/tlg_dates.json)
        │
        ▼
  top-50 candidates reranked by
  Qwen3-Reranker-0.6B (cross-encoder)
        │
        ▼
  top-k passages returned
  with author / work / section
        │
        ▼  (optional)
  LLM synthesis, streamed
  (Gemma 3 12B via transformers)
        │
        ▼
  Gradio web UI

Corpus parsing: TEI div[type=textpart] elements are the primary indexing unit for prose works. For verse texts (Homer, Pindar, the tragedians), individual <l> lines are grouped into 20-line chunks so each indexed unit carries enough semantic content.

Cross-lingual retrieval: Qwen/Qwen3-Embedding-0.6B is a multilingual embedding model that maps texts from different languages into a shared embedding space. An English query and an ancient Greek passage about the same concept will have a high cosine similarity without any translation step. A cross-encoder reranker then rescores the top candidates jointly with the query for much sharper final ordering.

Author metadata: data/tlg_dates.json maps each TLG author ID to a name, date range (signed centuries, e.g. −5 for 5th c. BC), and genre epithets, extracted from the TLG Canon by extract_tlg_dates.py. This drives the century and genre filters and the dates shown on result cards.

Project structure

zetema/
├── ingest.py                    # corpus ingestion: parse → embed → store
├── app.py                       # Gradio web UI
├── extract_tlg_dates.py         # derives data/tlg_dates.json from the TLG Canon
├── extract_eng_translations.py  # derives data/eng_translations.json from the corpora
├── requirements.txt
├── perseus -> ...               # symlink to Perseus XML corpus
├── first1k -> ...               # symlink to First1KGreek XML corpus
└── data/
    ├── tlg_dates.json           # author dates + genres (checked in)
    ├── eng_translations.json    # work → English version label (checked in)
    └── perseus_rag_qwen.duckdb  # built by ingest.py

References

• Perseus Digital Library, Tufts University
• First1KGreek, Open Greek and Latin
• Thesaurus Linguae Graecae Canon (author dates and genres)
• Qwen/Qwen3-Embedding-0.6B embedding model
• Qwen/Qwen3-Reranker-0.6B reranker
• google/gemma-3-12b-it synthesis/chat model (default)