---
title: Zetema
emoji: 🏺
colorFrom: yellow
colorTo: red
sdk: gradio
sdk_version: 6.19.0
app_file: app.py
pinned: false
short_description: Semantic search over Perseus and First1KGreek
---

# Zetema

Semantic search over the [Perseus Digital Library](http://www.perseus.tufts.edu/) and [First1KGreek](https://github.com/OpenGreekAndLatin/First1K-Greek) Greek corpora using local open-weight language models. Ask questions in English (or Greek) and find which ancient authors write about a given topic.

## What it does

- **Cross-lingual retrieval**: type a query in English and retrieve matching passages in ancient Greek — the embedding model maps both into the same semantic space
- **Author ranking**: see at a glance which authors discuss a topic most
- **Read in context**: every result links to the passage in the [Scaife Viewer](https://scaife.perseus.org) via its CTS URN, with the English translation opened side by side when the corpus has one
- **Reranked results**: dense retrieval fetches a candidate pool, then a cross-encoder reranker (`Qwen/Qwen3-Reranker-0.6B`) orders the final results
- **Century and genre filters**: narrow the search to particular centuries or genres (history, tragedy, philosophy, …), both derived from the TLG Canon; result cards show each author's dates
- **LLM synthesis**: optionally send top passages to a local LLM (Gemma 3 by default, via Hugging Face `transformers`, streamed) to get a summary in English with key phrase translations
- **364 authors, 1,644 Greek works**: Perseus and [First1KGreek](https://github.com/OpenGreekAndLatin/First1K-Greek) together — Plato, Aristotle, Thucydides, Homer, Aeschylus, Sophocles, Euripides, Herodotus, and many more

## Requirements

- Python 3.11+
- CUDA GPU strongly recommended (reranking and synthesis run locally on it)
- ~30 GB free disk space (models + index; the default Gemma 3 12B synthesis model alone is ~24 GB)
- A Hugging Face account with the [Gemma license](https://huggingface.co/google/gemma-3-12b-it) accepted, logged in via `huggingface-cli login` (only needed for LLM synthesis/chat)
- 16 GB RAM recommended for the in-memory vector search

## Setup

### 1. Clone and install dependencies

```bash
git clone <this-repo>
cd zetema
pip install -r requirements.txt
pip install torch sentence-transformers transformers numpy pandas
```

Install a `torch` build that matches your CUDA driver — a mismatched build silently falls back to CPU (check `torch.cuda.is_available()`). See the [PyTorch install selector](https://pytorch.org/get-started/locally/) for the right index URL.

### 2. Point to the corpus

Two symlinks in the project root point at your local corpus checkouts. Both are optional individually — `ingest.py` skips a missing one — but the index only covers what it can find:

```bash
# example — change the targets to match your actual paths
ln -sf /path/to/perseus/xml perseus
ln -sf /path/to/First1KGreek/data first1k
```

Each corpus should contain directories named `tlg0001`, `tlg0003`, … each with `__cts__.xml` metadata and TEI XML text files. Where the two corpora carry the same work, Perseus takes precedence and the duplicate is skipped.

### 3. Build the index (one-time, ~10–20 min on GPU, longer on CPU)

```bash
python ingest.py
```

This will:
1. Download the `Qwen/Qwen3-Embedding-0.6B` embedding model (~1.2 GB, cached after first run)
2. Parse all Greek TEI XML files and extract sections
3. Embed every section and store the result in `data/perseus_rag_qwen.duckdb`

Progress is shown with a progress bar. Re-run anytime to rebuild the index from scratch.

### 4. Launch the app

```bash
python app.py
```

Opens at [http://localhost:7861](http://localhost:7861).

## Running on Hugging Face Spaces (this branch)

This `hf-space` branch is directly pushable to a Space. One-time setup:

1. **Upload the index** to a dataset repo (private is fine — the Space authenticates with your token):

   ```bash
   hf repo create zetema-index --repo-type dataset --private
   hf upload zetema-index data/perseus_rag_qwen.duckdb perseus_rag_qwen.duckdb --repo-type dataset
   ```

2. **Create the Space** at huggingface.co/new-space: SDK **Gradio**, hardware **ZeroGPU** (included with Pro). Then push this branch to it:

   ```bash
   git remote add space https://huggingface.co/spaces/<user>/zetema
   git push space hf-space:main
   ```

3. **Add an `HF_TOKEN` secret** in the Space settings (a token from an account that has accepted the [Gemma license](https://huggingface.co/google/gemma-3-12b-it)) — needed both for the gated synthesis model and for the private index dataset.

At startup the app downloads the DuckDB index from the dataset repo (override the default `Jacobo/zetema-index` with a `ZETEMA_INDEX_REPO` env var) and preloads the models; GPU-heavy calls (retrieval + reranking, LLM generation) run inside ZeroGPU's `@spaces.GPU` windows. Locally, none of this changes anything — the app runs exactly as on `main`.

## Usage

| Query | What you find |
|---|---|
| `immortality of the soul` | Plato, Pythagorean fragments, Plotinus |
| `rhetoric and persuasion` | Aristotle, Gorgias, Isocrates |
| `hubris and divine punishment` | Aeschylus, Sophocles, Herodotus |
| `democracy and tyranny` | Thucydides, Xenophon, Plato |
| `ψυχή` | Query directly in Greek |

**Controls:**
- **Number of results** — how many passages to retrieve (5–50)
- **Filter by author / century / genre** — narrow the search; the three filters combine, and century and genre come from the TLG Canon (authors without canon data land in an "Unknown" bucket)
- **Synthesize results** (on by default) — sends top passages to a local LLM for a streamed thematic summary with English translations of key phrases; a chat panel lets you ask follow-up questions grounded in the retrieved passages
- **Color scheme** — switch the UI theme live (Parchment by default)

## How it works

```
Query (English or Greek)
        │
        ▼
  Qwen3-Embedding-0.6B           ← same model used at ingest time
  encodes query as vector
        │
        ▼
  cosine similarity over          ← all section embeddings loaded
  ~133k Greek passages              in memory at startup
        │
        ▼
  author / century / genre        ← masks from TLG Canon data
  filter masks applied              (data/tlg_dates.json)
        │
        ▼
  top-50 candidates reranked by
  Qwen3-Reranker-0.6B (cross-encoder)
        │
        ▼
  top-k passages returned
  with author / work / section
        │
        ▼  (optional)
  LLM synthesis, streamed
  (Gemma 3 12B via transformers)
        │
        ▼
  Gradio web UI
```

**Corpus parsing:** TEI `div[type=textpart]` elements are the primary indexing unit for prose works. For verse texts (Homer, Pindar, the tragedians), individual `<l>` lines are grouped into 20-line chunks so each indexed unit carries enough semantic content.

**Cross-lingual retrieval:** `Qwen/Qwen3-Embedding-0.6B` is a multilingual embedding model that maps texts from different languages into a shared embedding space. An English query and an ancient Greek passage about the same concept will have a high cosine similarity without any translation step. A cross-encoder reranker then rescores the top candidates jointly with the query for much sharper final ordering.

**Author metadata:** `data/tlg_dates.json` maps each TLG author ID to a name, date range (signed centuries, e.g. −5 for 5th c. BC), and genre epithets, extracted from the TLG Canon by `extract_tlg_dates.py`. This drives the century and genre filters and the dates shown on result cards.

## Project structure

```
zetema/
├── ingest.py                    # corpus ingestion: parse → embed → store
├── app.py                       # Gradio web UI
├── extract_tlg_dates.py         # derives data/tlg_dates.json from the TLG Canon
├── extract_eng_translations.py  # derives data/eng_translations.json from the corpora
├── requirements.txt
├── perseus -> ...               # symlink to Perseus XML corpus
├── first1k -> ...               # symlink to First1KGreek XML corpus
└── data/
    ├── tlg_dates.json           # author dates + genres (checked in)
    ├── eng_translations.json    # work → English version label (checked in)
    └── perseus_rag_qwen.duckdb  # built by ingest.py
```

## References

- [Perseus Digital Library](http://www.perseus.tufts.edu/), Tufts University
- [First1KGreek](https://github.com/OpenGreekAndLatin/First1K-Greek), Open Greek and Latin
- [Thesaurus Linguae Graecae](https://www.tlg.uci.edu/) Canon (author dates and genres)
- [Qwen/Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) embedding model
- [Qwen/Qwen3-Reranker-0.6B](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B) reranker
- [google/gemma-3-12b-it](https://huggingface.co/google/gemma-3-12b-it) synthesis/chat model (default)