Spaces:
Running on Zero
Running on Zero
Commit ·
04721fb
1
Parent(s): 2f18d90
Add opt-in usage telemetry to a private HF dataset
Browse filesLogs upload, translate, edit, download, navigate, resume, and prompt
events to billingsmoore/stui-usage, gated by a per-session opt-out
checkbox in Settings and disclosed on the landing page. Also stops
tracking CLAUDE.md and any .claude/ directory — those are local
development aids now, not shared repo content.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- .gitignore +2 -0
- CLAUDE.md +0 -121
- app.py +16 -4
- engine/__init__.py +1 -0
- engine/usage_store.py +63 -0
- handlers.py +80 -5
- requirements.txt +1 -0
- tests/test_handlers.py +194 -6
- tests/test_usage_store.py +68 -0
.gitignore
CHANGED
|
@@ -4,3 +4,5 @@ __pycache__/
|
|
| 4 |
.venv/
|
| 5 |
venv/
|
| 6 |
.pytest_cache/
|
|
|
|
|
|
|
|
|
| 4 |
.venv/
|
| 5 |
venv/
|
| 6 |
.pytest_cache/
|
| 7 |
+
CLAUDE.md
|
| 8 |
+
.claude/
|
CLAUDE.md
DELETED
|
@@ -1,121 +0,0 @@
|
|
| 1 |
-
# CLAUDE.md
|
| 2 |
-
|
| 3 |
-
This file guides Claude Code (or any future session) working in this
|
| 4 |
-
repository. This project is a **standalone, simplified fork** of
|
| 5 |
-
`TranslationUI` (from the `Garchen` monorepo) — it is intentionally
|
| 6 |
-
detached: no submodule relationship, no shared git history, no HuggingFace
|
| 7 |
-
dataset dependencies. Everything needed to understand and run it lives here.
|
| 8 |
-
|
| 9 |
-
## What this app is
|
| 10 |
-
|
| 11 |
-
A minimal Gradio app for translating Tibetan Buddhist texts into English.
|
| 12 |
-
Upload a `.txt`, `.docx`, or `.pdf` file → it's split into segments →
|
| 13 |
-
translate each segment (or all at once) → edit inline → download the result.
|
| 14 |
-
|
| 15 |
-
Compared to the original `TranslationUI` this app deliberately drops:
|
| 16 |
-
- Video/audio playback and Google Drive import (no media, ever)
|
| 17 |
-
- Subtitle formats (SRT/VTT) and timestamp-based segments — plain text in, plain text out
|
| 18 |
-
- Multi-language target selection (English only)
|
| 19 |
-
- Per-language prompt files and a separate post-edit pass (one prompt, no post-edit)
|
| 20 |
-
- HuggingFace-dataset-backed project storage and versioned prompt history
|
| 21 |
-
(no `HF_TOKEN`, no `datasets`/`huggingface_hub` write dependency — state
|
| 22 |
-
lives only in the browser session; prompt edits are a local file)
|
| 23 |
-
|
| 24 |
-
## Architecture
|
| 25 |
-
|
| 26 |
-
```
|
| 27 |
-
app.py Gradio UI (Blocks), wires handlers to components
|
| 28 |
-
handlers.py Event handlers + in-memory session state (gr.State)
|
| 29 |
-
styles.py Minimal CSS (no floating video player JS anymore)
|
| 30 |
-
translation_prompt.txt The one editable translation prompt (Garchen Rinpoche voice/glossary)
|
| 31 |
-
engine/
|
| 32 |
-
chunking.py Plain-text -> segment list (paragraph, then shad/whitespace splitting)
|
| 33 |
-
formats.py Read .txt/.docx/.pdf; write .txt/.docx/.json
|
| 34 |
-
prompt.py Read/write/reset translation_prompt.txt (local file only)
|
| 35 |
-
translate.py Backend dispatcher (OpenRouter vs local CPU model) + batching/progress
|
| 36 |
-
openrouter_backend.py OpenRouter API calls, retries, server-side model fallback, batch prompt building
|
| 37 |
-
local_backend.py CPU fallback: AutoModelForSeq2SeqLM "billingsmoore/mlotsawa-ground-base"
|
| 38 |
-
```
|
| 39 |
-
|
| 40 |
-
## Translation backend selection
|
| 41 |
-
|
| 42 |
-
`engine/translate.py::_resolve_key()` decides per-call:
|
| 43 |
-
- An OpenRouter API key was typed into the UI, **or** `OPENROUTER_API_KEY` is set (env var / `.env`)
|
| 44 |
-
→ use OpenRouter, with the prompt in `translation_prompt.txt` (editable in Settings) and the model
|
| 45 |
-
picked in the dropdown. The dropdown is populated by `engine/openrouter_backend.py::list_model_choices()`,
|
| 46 |
-
which live-fetches OpenRouter's `/models` catalog and puts `CURATED_MODELS` (a small hand-picked,
|
| 47 |
-
known-good-for-translation set) at the top, followed by the rest of the text-output catalog
|
| 48 |
-
alphabetically — falls back to just `CURATED_MODELS` if the fetch fails (offline, OpenRouter down).
|
| 49 |
-
`CURATED_MODELS` also doubles as the `models` fallback array OpenRouter is given per request, so it
|
| 50 |
-
retries server-side against another good model if the chosen one errors out.
|
| 51 |
-
- No key anywhere → use the local CPU model `billingsmoore/mlotsawa-ground-base` via
|
| 52 |
-
`AutoModelForSeq2SeqLM.generate()` directly (see `engine/local_backend.py`). This is a plain finetuned
|
| 53 |
-
T5 seq2seq model — it does **not** read `translation_prompt.txt` at all. First call downloads the
|
| 54 |
-
model (~a few hundred MB) and is slow on CPU; it stays cached in memory for the life of the process.
|
| 55 |
-
This model was introduced in ["Optimizing T5 for Lightweight Tibetan-English Translation"](https://doi.org/10.21203/rs.3.rs-7409829/v1)
|
| 56 |
-
(Moore & Lauren, 2025); its training/eval code is at
|
| 57 |
-
[optimizing-t5-tibetan-english-mt](https://github.com/billingsmoore/optimizing-t5-tibetan-english-mt).
|
| 58 |
-
|
| 59 |
-
**Why not `pipeline("translation", ...)`:** that's the model card's documented usage, but the
|
| 60 |
-
`"translation"` pipeline task was removed in `transformers>=5` (raises `Unknown task translation`).
|
| 61 |
-
Instead `local_backend.py` reads the model's own `model.config.task_specific_params["translation_bo_to_en"]`
|
| 62 |
-
(prefix `"translate Tibetan to English: "`, `num_beams=4`, `max_length=300`) and calls `generate()`
|
| 63 |
-
directly — verified to reproduce the model card's example output exactly, and works on any transformers
|
| 64 |
-
version that ships `AutoModelForSeq2SeqLM`/`AutoTokenizer`. Don't "fix" this back to `pipeline(...)`
|
| 65 |
-
without first checking what transformers version is actually installed.
|
| 66 |
-
|
| 67 |
-
## Segmentation
|
| 68 |
-
|
| 69 |
-
`engine/chunking.py::split_into_segments()`: splits on blank-line paragraph breaks, then (for any
|
| 70 |
-
paragraph longer than `max_chars`, default 400) on Tibetan sentence-ending shad marks (`།` `༎`), then
|
| 71 |
-
falls back to plain whitespace word-chunking for any single "sentence" still too long. There is no
|
| 72 |
-
timestamp concept anywhere in this app — segments are just `{"source": str, "target": str}` dicts.
|
| 73 |
-
|
| 74 |
-
## Running it
|
| 75 |
-
|
| 76 |
-
```bash
|
| 77 |
-
pip install -r requirements.txt
|
| 78 |
-
python app.py
|
| 79 |
-
```
|
| 80 |
-
Opens on `http://localhost:7860`. Put `OPENROUTER_API_KEY=...` in a `.env` file to default to
|
| 81 |
-
OpenRouter without typing a key into the UI each time.
|
| 82 |
-
|
| 83 |
-
## Testing
|
| 84 |
-
|
| 85 |
-
```bash
|
| 86 |
-
pip install -r requirements-dev.txt
|
| 87 |
-
pytest
|
| 88 |
-
```
|
| 89 |
-
|
| 90 |
-
`tests/` covers chunking, file I/O (txt/docx/pdf load+export, JSON resume),
|
| 91 |
-
prompt storage, the OpenRouter backend (model curation/fetch-fallback, request
|
| 92 |
-
retry/fallback logic, batch prompt building/parsing — all with `requests`
|
| 93 |
-
mocked, no real network calls or API key needed), the backend dispatcher, the
|
| 94 |
-
local CPU backend (with `torch`/`transformers` stubbed via `sys.modules` so
|
| 95 |
-
the suite doesn't require installing those heavy deps), the Gradio event
|
| 96 |
-
handlers, and a smoke test that the Blocks graph in `app.py` builds without
|
| 97 |
-
error. `samples/sample_tibetan.{txt,docx,pdf}` are real Tibetan-script fixture
|
| 98 |
-
files (a well-known refuge/bodhicitta verse, the four immeasurables, and a
|
| 99 |
-
dedication verse) used by the format-loading tests and useful for manually
|
| 100 |
-
exercising the upload flow.
|
| 101 |
-
|
| 102 |
-
A `pre-push` git hook (`.githooks/pre-push`) runs the full suite and blocks
|
| 103 |
-
the push if anything fails. It's already installed into `.git/hooks/pre-push`
|
| 104 |
-
in this working copy; on a fresh clone, install it with either
|
| 105 |
-
`git config core.hooksPath .githooks` or `cp .githooks/pre-push .git/hooks/`.
|
| 106 |
-
|
| 107 |
-
## Conventions to keep
|
| 108 |
-
|
| 109 |
-
- No comments explaining *what* code does — only *why*, when non-obvious (mirrors the parent repo's style).
|
| 110 |
-
- Don't reintroduce HuggingFace dataset persistence, subtitle timing, or multi-language prompt files —
|
| 111 |
-
those were removed on purpose to keep this app simple. If a future ask needs them, treat it as a
|
| 112 |
-
deliberate scope expansion, not a bug to fix.
|
| 113 |
-
- This repo has no relationship to `Garchen`'s git history — it's fine to `git init` here and commit
|
| 114 |
-
independently.
|
| 115 |
-
|
| 116 |
-
## Remotes
|
| 117 |
-
|
| 118 |
-
Two remotes, same dual-remote convention as `Garchen`: `origin` = GitHub
|
| 119 |
-
(https://github.com/billingsmoore/SimpleTranslationUI, source of truth) and `space` = the deployed
|
| 120 |
-
Hugging Face Space (https://huggingface.co/spaces/billingsmoore/SimpleTranslationUI). Push to `origin`
|
| 121 |
-
for normal commits; push to `space` (`git push space main:main`) to actually redeploy the Space.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
app.py
CHANGED
|
@@ -9,6 +9,7 @@ from engine import DEFAULT_MODEL, list_model_choices
|
|
| 9 |
from styles import CSS
|
| 10 |
from handlers import (
|
| 11 |
MAX_SLOTS,
|
|
|
|
| 12 |
_handle_download_click,
|
| 13 |
_handle_get_files,
|
| 14 |
_handle_next,
|
|
@@ -20,6 +21,7 @@ from handlers import (
|
|
| 20 |
_read_prompt,
|
| 21 |
_reset_prompt,
|
| 22 |
_save_prompt,
|
|
|
|
| 23 |
_translate_all,
|
| 24 |
_translate_one,
|
| 25 |
)
|
|
@@ -43,7 +45,9 @@ def build_app() -> gr.Blocks:
|
|
| 43 |
"the translation prompt below) and generally lower translation quality. For higher-quality, instruction-"
|
| 44 |
"following translation, add an **OpenRouter API key** in Settings and pick a model "
|
| 45 |
"from the dropdown — this calls OpenRouter's API (usage is billed to your OpenRouter "
|
| 46 |
-
"account) and uses the editable translation prompt."
|
|
|
|
|
|
|
| 47 |
)
|
| 48 |
|
| 49 |
with gr.Row():
|
|
@@ -59,6 +63,10 @@ def build_app() -> gr.Blocks:
|
|
| 59 |
)
|
| 60 |
|
| 61 |
with gr.Accordion("Settings", open=False):
|
|
|
|
|
|
|
|
|
|
|
|
|
| 62 |
with gr.Row():
|
| 63 |
openrouter_api_key = gr.Textbox(
|
| 64 |
label="OpenRouter API Key (optional)",
|
|
@@ -142,7 +150,7 @@ def build_app() -> gr.Blocks:
|
|
| 142 |
inputs=[app_state, openrouter_api_key, openrouter_model, *slot_sources, *slot_targets],
|
| 143 |
outputs=translate_all_outputs,
|
| 144 |
)
|
| 145 |
-
cancel_btn.click(
|
| 146 |
|
| 147 |
save_btn.click(_handle_save, inputs=edit_inputs, outputs=[app_state, status_output])
|
| 148 |
download_btn.click(
|
|
@@ -171,8 +179,12 @@ def build_app() -> gr.Blocks:
|
|
| 171 |
outputs=[app_state, slot_targets[i]],
|
| 172 |
)
|
| 173 |
|
| 174 |
-
save_prompt_btn.click(_save_prompt, inputs=[prompt_box], outputs=[prompt_status])
|
| 175 |
-
reset_prompt_btn.click(_reset_prompt, inputs=[], outputs=[prompt_box, prompt_status])
|
|
|
|
|
|
|
|
|
|
|
|
|
| 176 |
|
| 177 |
return demo
|
| 178 |
|
|
|
|
| 9 |
from styles import CSS
|
| 10 |
from handlers import (
|
| 11 |
MAX_SLOTS,
|
| 12 |
+
_handle_cancel_translate,
|
| 13 |
_handle_download_click,
|
| 14 |
_handle_get_files,
|
| 15 |
_handle_next,
|
|
|
|
| 21 |
_read_prompt,
|
| 22 |
_reset_prompt,
|
| 23 |
_save_prompt,
|
| 24 |
+
_set_usage_tracking,
|
| 25 |
_translate_all,
|
| 26 |
_translate_one,
|
| 27 |
)
|
|
|
|
| 45 |
"the translation prompt below) and generally lower translation quality. For higher-quality, instruction-"
|
| 46 |
"following translation, add an **OpenRouter API key** in Settings and pick a model "
|
| 47 |
"from the dropdown — this calls OpenRouter's API (usage is billed to your OpenRouter "
|
| 48 |
+
"account) and uses the editable translation prompt.\n\n"
|
| 49 |
+
"Usage of this space may be monitored for research purposes. You can opt out of this "
|
| 50 |
+
"in Settings."
|
| 51 |
)
|
| 52 |
|
| 53 |
with gr.Row():
|
|
|
|
| 63 |
)
|
| 64 |
|
| 65 |
with gr.Accordion("Settings", open=False):
|
| 66 |
+
usage_tracking_checkbox = gr.Checkbox(
|
| 67 |
+
label="Allow usage tracking for research purposes",
|
| 68 |
+
value=True,
|
| 69 |
+
)
|
| 70 |
with gr.Row():
|
| 71 |
openrouter_api_key = gr.Textbox(
|
| 72 |
label="OpenRouter API Key (optional)",
|
|
|
|
| 150 |
inputs=[app_state, openrouter_api_key, openrouter_model, *slot_sources, *slot_targets],
|
| 151 |
outputs=translate_all_outputs,
|
| 152 |
)
|
| 153 |
+
cancel_btn.click(_handle_cancel_translate, inputs=[app_state], outputs=[app_state], cancels=[translate_event])
|
| 154 |
|
| 155 |
save_btn.click(_handle_save, inputs=edit_inputs, outputs=[app_state, status_output])
|
| 156 |
download_btn.click(
|
|
|
|
| 179 |
outputs=[app_state, slot_targets[i]],
|
| 180 |
)
|
| 181 |
|
| 182 |
+
save_prompt_btn.click(_save_prompt, inputs=[app_state, prompt_box], outputs=[app_state, prompt_status])
|
| 183 |
+
reset_prompt_btn.click(_reset_prompt, inputs=[app_state], outputs=[app_state, prompt_box, prompt_status])
|
| 184 |
+
|
| 185 |
+
usage_tracking_checkbox.change(
|
| 186 |
+
_set_usage_tracking, inputs=[app_state, usage_tracking_checkbox], outputs=[app_state],
|
| 187 |
+
)
|
| 188 |
|
| 189 |
return demo
|
| 190 |
|
engine/__init__.py
CHANGED
|
@@ -12,6 +12,7 @@ from .formats import (
|
|
| 12 |
parse_json,
|
| 13 |
)
|
| 14 |
from .prompt import read_prompt, save_prompt, reset_prompt, PROMPT_PATH
|
|
|
|
| 15 |
from .translate import (
|
| 16 |
CURATED_MODELS,
|
| 17 |
DEFAULT_MODEL,
|
|
|
|
| 12 |
parse_json,
|
| 13 |
)
|
| 14 |
from .prompt import read_prompt, save_prompt, reset_prompt, PROMPT_PATH
|
| 15 |
+
from .usage_store import log_event
|
| 16 |
from .translate import (
|
| 17 |
CURATED_MODELS,
|
| 18 |
DEFAULT_MODEL,
|
engine/usage_store.py
ADDED
|
@@ -0,0 +1,63 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Usage telemetry, written as one event per file to the billingsmoore/stui-usage
|
| 2 |
+
HuggingFace dataset (private). Tracks uploads, translations, and edits so we can
|
| 3 |
+
study how the app is actually used; users can opt out per-session in Settings.
|
| 4 |
+
|
| 5 |
+
Each event is an independent commit (no shared index to read-modify-write),
|
| 6 |
+
so concurrent sessions never race each other. Fails open (returns False /
|
| 7 |
+
no-op and logs a warning) if HF_TOKEN isn't set, the dataset repo doesn't
|
| 8 |
+
exist yet, or the network call fails — usage logging must never break the
|
| 9 |
+
translation workflow.
|
| 10 |
+
"""
|
| 11 |
+
|
| 12 |
+
import json
|
| 13 |
+
import os
|
| 14 |
+
import uuid
|
| 15 |
+
from datetime import datetime, timezone
|
| 16 |
+
|
| 17 |
+
DATASET_REPO_ID = "billingsmoore/stui-usage"
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
def _hf_token() -> str | None:
|
| 21 |
+
return os.environ.get("HF_TOKEN") or None
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def _ensure_dataset(token: str) -> None:
|
| 25 |
+
from huggingface_hub import HfApi
|
| 26 |
+
HfApi(token=token).create_repo(
|
| 27 |
+
repo_id=DATASET_REPO_ID, repo_type="dataset", private=True, exist_ok=True,
|
| 28 |
+
)
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
def log_event(event_type: str, session_id: str, payload: dict) -> bool:
|
| 32 |
+
"""Write one usage event. Returns True on success, False if skipped/failed."""
|
| 33 |
+
token = _hf_token()
|
| 34 |
+
if not token or not session_id:
|
| 35 |
+
return False
|
| 36 |
+
try:
|
| 37 |
+
from huggingface_hub import CommitOperationAdd, HfApi
|
| 38 |
+
_ensure_dataset(token)
|
| 39 |
+
|
| 40 |
+
now = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
| 41 |
+
record = {
|
| 42 |
+
"event": event_type,
|
| 43 |
+
"session_id": session_id,
|
| 44 |
+
"timestamp": now,
|
| 45 |
+
**payload,
|
| 46 |
+
}
|
| 47 |
+
path_in_repo = f"events/{session_id}/{now}_{event_type}_{uuid.uuid4().hex[:8]}.json"
|
| 48 |
+
|
| 49 |
+
HfApi(token=token).create_commit(
|
| 50 |
+
repo_id=DATASET_REPO_ID,
|
| 51 |
+
repo_type="dataset",
|
| 52 |
+
operations=[
|
| 53 |
+
CommitOperationAdd(
|
| 54 |
+
path_in_repo=path_in_repo,
|
| 55 |
+
path_or_fileobj=json.dumps(record, ensure_ascii=False, indent=2).encode("utf-8"),
|
| 56 |
+
),
|
| 57 |
+
],
|
| 58 |
+
commit_message=f"{event_type} event ({session_id})",
|
| 59 |
+
)
|
| 60 |
+
return True
|
| 61 |
+
except Exception as e:
|
| 62 |
+
print(f"[WARN] Could not log usage event {event_type!r}: {e}")
|
| 63 |
+
return False
|
handlers.py
CHANGED
|
@@ -4,6 +4,7 @@ import math
|
|
| 4 |
import os
|
| 5 |
import tempfile
|
| 6 |
import threading
|
|
|
|
| 7 |
import zipfile
|
| 8 |
from pathlib import Path
|
| 9 |
|
|
@@ -14,6 +15,7 @@ from engine import (
|
|
| 14 |
export_to_json,
|
| 15 |
export_to_txt,
|
| 16 |
load_source_file,
|
|
|
|
| 17 |
parse_json,
|
| 18 |
read_prompt,
|
| 19 |
reset_prompt,
|
|
@@ -29,7 +31,25 @@ MAX_SLOTS = 25
|
|
| 29 |
# ── State ─────────────────────────────────────────────────────────────────────
|
| 30 |
|
| 31 |
def _make_state() -> dict:
|
| 32 |
-
return {
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
|
| 34 |
|
| 35 |
def _save_page_edits(state: dict, sources: list, targets: list) -> dict:
|
|
@@ -115,6 +135,13 @@ def _load_source(source_file, state):
|
|
| 115 |
state["label"] = label
|
| 116 |
state["has_translation"] = False
|
| 117 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 118 |
return (*_load_page(state), gr.update(visible=True))
|
| 119 |
|
| 120 |
|
|
@@ -140,6 +167,16 @@ def _load_resume_json(resume_file, state):
|
|
| 140 |
|
| 141 |
state["has_translation"] = any(seg.get("target") for seg in state["segments"])
|
| 142 |
state["page"] = 0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 143 |
return _load_page(state)
|
| 144 |
|
| 145 |
|
|
@@ -189,6 +226,13 @@ def _handle_save(state, *slot_values):
|
|
| 189 |
sources = list(slot_values[:MAX_SLOTS])
|
| 190 |
targets = list(slot_values[MAX_SLOTS:])
|
| 191 |
state = _save_page_edits(state, sources, targets)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 192 |
return state, gr.update(value="Saved.", visible=True)
|
| 193 |
|
| 194 |
|
|
@@ -223,6 +267,15 @@ def _handle_get_files(state, selected_labels):
|
|
| 223 |
for path in chosen:
|
| 224 |
zf.write(path, arcname=os.path.basename(path))
|
| 225 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 226 |
return gr.update(value=zip_path, visible=True), gr.update(visible=False), gr.update(visible=False)
|
| 227 |
|
| 228 |
|
|
@@ -240,6 +293,13 @@ def _translate_one(state: dict, slot_idx: int, source: str, api_key: str, model:
|
|
| 240 |
segments[actual_idx]["target"] = text
|
| 241 |
state["segments"] = segments
|
| 242 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 243 |
return state, gr.update(value=text)
|
| 244 |
|
| 245 |
|
|
@@ -289,23 +349,36 @@ def _translate_all(state, api_key, model, *slot_values):
|
|
| 289 |
if errors:
|
| 290 |
status += f" {len(errors)} error(s)."
|
| 291 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 292 |
yield (*_load_page(state), gr.update(value=status, visible=True))
|
| 293 |
|
| 294 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 295 |
# ── Prompt management ─────────────────────────────────────────────────────────
|
| 296 |
|
| 297 |
def _read_prompt() -> str:
|
| 298 |
return read_prompt()
|
| 299 |
|
| 300 |
|
| 301 |
-
def _save_prompt(text: str):
|
| 302 |
save_prompt(text)
|
| 303 |
-
|
|
|
|
| 304 |
|
| 305 |
|
| 306 |
-
def _reset_prompt():
|
| 307 |
text = reset_prompt()
|
| 308 |
-
|
|
|
|
| 309 |
|
| 310 |
|
| 311 |
# ── Navigation ��───────────────────────────────────────────────────────────────
|
|
@@ -316,6 +389,7 @@ def _handle_prev(state, *slot_values):
|
|
| 316 |
state = _save_page_edits(state, sources, targets)
|
| 317 |
if state["page"] > 0:
|
| 318 |
state["page"] -= 1
|
|
|
|
| 319 |
return _load_page(state)
|
| 320 |
|
| 321 |
|
|
@@ -327,4 +401,5 @@ def _handle_next(state, *slot_values):
|
|
| 327 |
total_pages = math.ceil(total / MAX_SLOTS) if total else 1
|
| 328 |
if state["page"] < total_pages - 1:
|
| 329 |
state["page"] += 1
|
|
|
|
| 330 |
return _load_page(state)
|
|
|
|
| 4 |
import os
|
| 5 |
import tempfile
|
| 6 |
import threading
|
| 7 |
+
import uuid
|
| 8 |
import zipfile
|
| 9 |
from pathlib import Path
|
| 10 |
|
|
|
|
| 15 |
export_to_json,
|
| 16 |
export_to_txt,
|
| 17 |
load_source_file,
|
| 18 |
+
log_event,
|
| 19 |
parse_json,
|
| 20 |
read_prompt,
|
| 21 |
reset_prompt,
|
|
|
|
| 31 |
# ── State ─────────────────────────────────────────────────────────────────────
|
| 32 |
|
| 33 |
def _make_state() -> dict:
|
| 34 |
+
return {
|
| 35 |
+
"page": 0,
|
| 36 |
+
"segments": [],
|
| 37 |
+
"has_translation": False,
|
| 38 |
+
"label": "output",
|
| 39 |
+
"session_id": uuid.uuid4().hex,
|
| 40 |
+
"usage_tracking_enabled": True,
|
| 41 |
+
}
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
def _log(state: dict, event_type: str, payload: dict) -> None:
|
| 45 |
+
if not state.get("usage_tracking_enabled", True):
|
| 46 |
+
return
|
| 47 |
+
log_event(event_type, state.get("session_id", ""), payload)
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
def _set_usage_tracking(state: dict, enabled: bool):
|
| 51 |
+
state["usage_tracking_enabled"] = enabled
|
| 52 |
+
return state
|
| 53 |
|
| 54 |
|
| 55 |
def _save_page_edits(state: dict, sources: list, targets: list) -> dict:
|
|
|
|
| 135 |
state["label"] = label
|
| 136 |
state["has_translation"] = False
|
| 137 |
|
| 138 |
+
_log(state, "upload", {
|
| 139 |
+
"filename": Path(path).name,
|
| 140 |
+
"file_type": Path(path).suffix.lstrip("."),
|
| 141 |
+
"segment_count": len(segments),
|
| 142 |
+
"segments": [{"source": seg.get("source", "")} for seg in segments],
|
| 143 |
+
})
|
| 144 |
+
|
| 145 |
return (*_load_page(state), gr.update(visible=True))
|
| 146 |
|
| 147 |
|
|
|
|
| 167 |
|
| 168 |
state["has_translation"] = any(seg.get("target") for seg in state["segments"])
|
| 169 |
state["page"] = 0
|
| 170 |
+
|
| 171 |
+
_log(state, "resume", {
|
| 172 |
+
"filename": Path(path).name,
|
| 173 |
+
"segment_count": len(state["segments"]),
|
| 174 |
+
"segments": [
|
| 175 |
+
{"source": seg.get("source", ""), "target": seg.get("target", "")}
|
| 176 |
+
for seg in state["segments"]
|
| 177 |
+
],
|
| 178 |
+
})
|
| 179 |
+
|
| 180 |
return _load_page(state)
|
| 181 |
|
| 182 |
|
|
|
|
| 226 |
sources = list(slot_values[:MAX_SLOTS])
|
| 227 |
targets = list(slot_values[MAX_SLOTS:])
|
| 228 |
state = _save_page_edits(state, sources, targets)
|
| 229 |
+
|
| 230 |
+
n = min(MAX_SLOTS, len(state["segments"]) - state["page"] * MAX_SLOTS)
|
| 231 |
+
_log(state, "edit", {
|
| 232 |
+
"segment_count": max(n, 0),
|
| 233 |
+
"segments": [{"source": sources[i], "target": targets[i]} for i in range(max(n, 0))],
|
| 234 |
+
})
|
| 235 |
+
|
| 236 |
return state, gr.update(value="Saved.", visible=True)
|
| 237 |
|
| 238 |
|
|
|
|
| 267 |
for path in chosen:
|
| 268 |
zf.write(path, arcname=os.path.basename(path))
|
| 269 |
|
| 270 |
+
_log(state, "download", {
|
| 271 |
+
"formats": list(selected_labels or []),
|
| 272 |
+
"segment_count": len(segments),
|
| 273 |
+
"segments": [
|
| 274 |
+
{"source": seg.get("source", ""), "target": seg.get("target", "")}
|
| 275 |
+
for seg in segments
|
| 276 |
+
],
|
| 277 |
+
})
|
| 278 |
+
|
| 279 |
return gr.update(value=zip_path, visible=True), gr.update(visible=False), gr.update(visible=False)
|
| 280 |
|
| 281 |
|
|
|
|
| 293 |
segments[actual_idx]["target"] = text
|
| 294 |
state["segments"] = segments
|
| 295 |
|
| 296 |
+
backend = f"openrouter:{model}" if using_openrouter(api_key) else "local_cpu"
|
| 297 |
+
_log(state, "translate", {
|
| 298 |
+
"backend": backend,
|
| 299 |
+
"segment_count": 1,
|
| 300 |
+
"segments": [{"source": source, "target": text}],
|
| 301 |
+
})
|
| 302 |
+
|
| 303 |
return state, gr.update(value=text)
|
| 304 |
|
| 305 |
|
|
|
|
| 349 |
if errors:
|
| 350 |
status += f" {len(errors)} error(s)."
|
| 351 |
|
| 352 |
+
_log(state, "translate", {
|
| 353 |
+
"backend": backend,
|
| 354 |
+
"segment_count": count,
|
| 355 |
+
"segments": [{"source": seg.get("source", ""), "target": seg.get("target", "")} for seg in segments],
|
| 356 |
+
})
|
| 357 |
+
|
| 358 |
yield (*_load_page(state), gr.update(value=status, visible=True))
|
| 359 |
|
| 360 |
|
| 361 |
+
def _handle_cancel_translate(state: dict):
|
| 362 |
+
_log(state, "translate_cancel", {})
|
| 363 |
+
return state
|
| 364 |
+
|
| 365 |
+
|
| 366 |
# ── Prompt management ─────────────────────────────────────────────────────────
|
| 367 |
|
| 368 |
def _read_prompt() -> str:
|
| 369 |
return read_prompt()
|
| 370 |
|
| 371 |
|
| 372 |
+
def _save_prompt(state: dict, text: str):
|
| 373 |
save_prompt(text)
|
| 374 |
+
_log(state, "prompt_save", {"prompt": text})
|
| 375 |
+
return state, gr.update(value="Prompt saved.")
|
| 376 |
|
| 377 |
|
| 378 |
+
def _reset_prompt(state: dict):
|
| 379 |
text = reset_prompt()
|
| 380 |
+
_log(state, "prompt_reset", {})
|
| 381 |
+
return state, gr.update(value=text), gr.update(value="Prompt reset to default.")
|
| 382 |
|
| 383 |
|
| 384 |
# ── Navigation ��───────────────────────────────────────────────────────────────
|
|
|
|
| 389 |
state = _save_page_edits(state, sources, targets)
|
| 390 |
if state["page"] > 0:
|
| 391 |
state["page"] -= 1
|
| 392 |
+
_log(state, "navigate", {"direction": "prev", "page": state["page"]})
|
| 393 |
return _load_page(state)
|
| 394 |
|
| 395 |
|
|
|
|
| 401 |
total_pages = math.ceil(total / MAX_SLOTS) if total else 1
|
| 402 |
if state["page"] < total_pages - 1:
|
| 403 |
state["page"] += 1
|
| 404 |
+
_log(state, "navigate", {"direction": "next", "page": state["page"]})
|
| 405 |
return _load_page(state)
|
requirements.txt
CHANGED
|
@@ -2,6 +2,7 @@ gradio>=6.9.0
|
|
| 2 |
spaces
|
| 3 |
requests
|
| 4 |
python-dotenv
|
|
|
|
| 5 |
transformers
|
| 6 |
torch
|
| 7 |
sentencepiece
|
|
|
|
| 2 |
spaces
|
| 3 |
requests
|
| 4 |
python-dotenv
|
| 5 |
+
huggingface_hub
|
| 6 |
transformers
|
| 7 |
torch
|
| 8 |
sentencepiece
|
tests/test_handlers.py
CHANGED
|
@@ -10,6 +10,13 @@ import handlers as h
|
|
| 10 |
SAMPLES_DIR = Path(__file__).parent.parent / "samples"
|
| 11 |
|
| 12 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
def _segments(*sources):
|
| 14 |
return [{"source": s, "target": ""} for s in sources]
|
| 15 |
|
|
@@ -18,7 +25,24 @@ def _segments(*sources):
|
|
| 18 |
|
| 19 |
def test_make_state_defaults():
|
| 20 |
state = h._make_state()
|
| 21 |
-
assert state
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
|
| 23 |
|
| 24 |
def test_save_page_edits_writes_current_page_slots():
|
|
@@ -157,6 +181,29 @@ def test_load_source_txt_populates_segments():
|
|
| 157 |
assert result[-1]["visible"] is True
|
| 158 |
|
| 159 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 160 |
def test_load_source_bad_file_hides_editor_and_does_not_raise(tmp_path):
|
| 161 |
bad_path = tmp_path / "bad.rtf"
|
| 162 |
bad_path.write_text("nope", encoding="utf-8")
|
|
@@ -259,24 +306,61 @@ def test_handle_save_persists_edits_and_shows_status():
|
|
| 259 |
state = h._make_state()
|
| 260 |
state["segments"] = _segments("a")
|
| 261 |
slot_values = ["edited"] + [""] * (h.MAX_SLOTS - 1) + ["translated"] + [""] * (h.MAX_SLOTS - 1)
|
| 262 |
-
|
|
|
|
| 263 |
assert new_state["segments"][0] == {"source": "edited", "target": "translated"}
|
| 264 |
assert status["value"] == "Saved."
|
| 265 |
|
| 266 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 267 |
# ── translation handlers ──────────────────────────────────────────────────
|
| 268 |
|
| 269 |
def test_translate_one_updates_state_and_returns_text():
|
| 270 |
state = h._make_state()
|
| 271 |
state["segments"] = _segments("source text")
|
| 272 |
state["page"] = 0
|
| 273 |
-
with patch.object(h, "_engine_translate_one", return_value="translated!") as mock_translate
|
|
|
|
| 274 |
new_state, update = h._translate_one(state, 0, "source text", "api-key", "model-x")
|
| 275 |
assert update["value"] == "translated!"
|
| 276 |
assert new_state["segments"][0]["target"] == "translated!"
|
| 277 |
mock_translate.assert_called_once_with("source text", "api-key", "model-x")
|
| 278 |
|
| 279 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 280 |
def test_translate_one_records_error_in_target():
|
| 281 |
state = h._make_state()
|
| 282 |
state["segments"] = _segments("source text")
|
|
@@ -311,7 +395,8 @@ def test_translate_all_reports_completion_status():
|
|
| 311 |
return segments, []
|
| 312 |
|
| 313 |
with patch.object(h, "translate_segments", side_effect=fake_translate_segments), \
|
| 314 |
-
patch.object(h, "using_openrouter", return_value=True)
|
|
|
|
| 315 |
results = list(h._translate_all(state, "api-key", "model-x", *slot_values))
|
| 316 |
|
| 317 |
final = results[-1]
|
|
@@ -320,6 +405,11 @@ def test_translate_all_reports_completion_status():
|
|
| 320 |
final_state = final[0]
|
| 321 |
assert [s["target"] for s in final_state["segments"]] == ["A", "B"]
|
| 322 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 323 |
|
| 324 |
def test_translate_all_reports_errors_in_status():
|
| 325 |
state = h._make_state()
|
|
@@ -371,10 +461,108 @@ def test_read_save_reset_prompt_roundtrip(monkeypatch, tmp_path):
|
|
| 371 |
|
| 372 |
assert h._read_prompt() == "default"
|
| 373 |
|
| 374 |
-
|
|
|
|
| 375 |
assert status["value"] == "Prompt saved."
|
| 376 |
assert h._read_prompt() == "edited"
|
| 377 |
|
| 378 |
-
box_update, status_update = h._reset_prompt()
|
| 379 |
assert box_update["value"] == "default"
|
| 380 |
assert status_update["value"] == "Prompt reset to default."
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
SAMPLES_DIR = Path(__file__).parent.parent / "samples"
|
| 11 |
|
| 12 |
|
| 13 |
+
@pytest.fixture(autouse=True)
|
| 14 |
+
def no_hf_token(monkeypatch):
|
| 15 |
+
"""Usage logging fails open without HF_TOKEN, so tests that don't
|
| 16 |
+
explicitly mock log_event still can't make a real network call."""
|
| 17 |
+
monkeypatch.delenv("HF_TOKEN", raising=False)
|
| 18 |
+
|
| 19 |
+
|
| 20 |
def _segments(*sources):
|
| 21 |
return [{"source": s, "target": ""} for s in sources]
|
| 22 |
|
|
|
|
| 25 |
|
| 26 |
def test_make_state_defaults():
|
| 27 |
state = h._make_state()
|
| 28 |
+
assert state["page"] == 0
|
| 29 |
+
assert state["segments"] == []
|
| 30 |
+
assert state["has_translation"] is False
|
| 31 |
+
assert state["label"] == "output"
|
| 32 |
+
assert state["usage_tracking_enabled"] is True
|
| 33 |
+
assert state["session_id"]
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def test_make_state_generates_unique_session_ids():
|
| 37 |
+
assert h._make_state()["session_id"] != h._make_state()["session_id"]
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
def test_set_usage_tracking_toggles_flag():
|
| 41 |
+
state = h._make_state()
|
| 42 |
+
state = h._set_usage_tracking(state, False)
|
| 43 |
+
assert state["usage_tracking_enabled"] is False
|
| 44 |
+
state = h._set_usage_tracking(state, True)
|
| 45 |
+
assert state["usage_tracking_enabled"] is True
|
| 46 |
|
| 47 |
|
| 48 |
def test_save_page_edits_writes_current_page_slots():
|
|
|
|
| 181 |
assert result[-1]["visible"] is True
|
| 182 |
|
| 183 |
|
| 184 |
+
def test_load_source_logs_upload_event():
|
| 185 |
+
state = h._make_state()
|
| 186 |
+
path = str(SAMPLES_DIR / "sample_tibetan.txt")
|
| 187 |
+
with patch.object(h, "log_event") as mock_log:
|
| 188 |
+
h._load_source(path, state)
|
| 189 |
+
mock_log.assert_called_once()
|
| 190 |
+
event_type, session_id, payload = mock_log.call_args[0]
|
| 191 |
+
assert event_type == "upload"
|
| 192 |
+
assert session_id == state["session_id"]
|
| 193 |
+
assert payload["filename"] == "sample_tibetan.txt"
|
| 194 |
+
assert payload["file_type"] == "txt"
|
| 195 |
+
assert payload["segment_count"] == len(state["segments"])
|
| 196 |
+
|
| 197 |
+
|
| 198 |
+
def test_load_source_skips_logging_when_tracking_disabled():
|
| 199 |
+
state = h._make_state()
|
| 200 |
+
state["usage_tracking_enabled"] = False
|
| 201 |
+
path = str(SAMPLES_DIR / "sample_tibetan.txt")
|
| 202 |
+
with patch.object(h, "log_event") as mock_log:
|
| 203 |
+
h._load_source(path, state)
|
| 204 |
+
mock_log.assert_not_called()
|
| 205 |
+
|
| 206 |
+
|
| 207 |
def test_load_source_bad_file_hides_editor_and_does_not_raise(tmp_path):
|
| 208 |
bad_path = tmp_path / "bad.rtf"
|
| 209 |
bad_path.write_text("nope", encoding="utf-8")
|
|
|
|
| 306 |
state = h._make_state()
|
| 307 |
state["segments"] = _segments("a")
|
| 308 |
slot_values = ["edited"] + [""] * (h.MAX_SLOTS - 1) + ["translated"] + [""] * (h.MAX_SLOTS - 1)
|
| 309 |
+
with patch.object(h, "log_event"):
|
| 310 |
+
new_state, status = h._handle_save(state, *slot_values)
|
| 311 |
assert new_state["segments"][0] == {"source": "edited", "target": "translated"}
|
| 312 |
assert status["value"] == "Saved."
|
| 313 |
|
| 314 |
|
| 315 |
+
def test_handle_save_logs_edit_event():
|
| 316 |
+
state = h._make_state()
|
| 317 |
+
state["segments"] = _segments("a")
|
| 318 |
+
slot_values = ["edited"] + [""] * (h.MAX_SLOTS - 1) + ["translated"] + [""] * (h.MAX_SLOTS - 1)
|
| 319 |
+
with patch.object(h, "log_event") as mock_log:
|
| 320 |
+
h._handle_save(state, *slot_values)
|
| 321 |
+
event_type, session_id, payload = mock_log.call_args[0]
|
| 322 |
+
assert event_type == "edit"
|
| 323 |
+
assert session_id == state["session_id"]
|
| 324 |
+
assert payload["segments"] == [{"source": "edited", "target": "translated"}]
|
| 325 |
+
|
| 326 |
+
|
| 327 |
+
def test_handle_save_skips_logging_when_tracking_disabled():
|
| 328 |
+
state = h._make_state()
|
| 329 |
+
state["usage_tracking_enabled"] = False
|
| 330 |
+
state["segments"] = _segments("a")
|
| 331 |
+
slot_values = [""] * h.MAX_SLOTS * 2
|
| 332 |
+
with patch.object(h, "log_event") as mock_log:
|
| 333 |
+
h._handle_save(state, *slot_values)
|
| 334 |
+
mock_log.assert_not_called()
|
| 335 |
+
|
| 336 |
+
|
| 337 |
# ── translation handlers ──────────────────────────────────────────────────
|
| 338 |
|
| 339 |
def test_translate_one_updates_state_and_returns_text():
|
| 340 |
state = h._make_state()
|
| 341 |
state["segments"] = _segments("source text")
|
| 342 |
state["page"] = 0
|
| 343 |
+
with patch.object(h, "_engine_translate_one", return_value="translated!") as mock_translate, \
|
| 344 |
+
patch.object(h, "log_event"):
|
| 345 |
new_state, update = h._translate_one(state, 0, "source text", "api-key", "model-x")
|
| 346 |
assert update["value"] == "translated!"
|
| 347 |
assert new_state["segments"][0]["target"] == "translated!"
|
| 348 |
mock_translate.assert_called_once_with("source text", "api-key", "model-x")
|
| 349 |
|
| 350 |
|
| 351 |
+
def test_translate_one_logs_translate_event():
|
| 352 |
+
state = h._make_state()
|
| 353 |
+
state["segments"] = _segments("source text")
|
| 354 |
+
with patch.object(h, "_engine_translate_one", return_value="translated!"), \
|
| 355 |
+
patch.object(h, "using_openrouter", return_value=True), \
|
| 356 |
+
patch.object(h, "log_event") as mock_log:
|
| 357 |
+
h._translate_one(state, 0, "source text", "api-key", "model-x")
|
| 358 |
+
event_type, session_id, payload = mock_log.call_args[0]
|
| 359 |
+
assert event_type == "translate"
|
| 360 |
+
assert payload["backend"] == "openrouter:model-x"
|
| 361 |
+
assert payload["segments"] == [{"source": "source text", "target": "translated!"}]
|
| 362 |
+
|
| 363 |
+
|
| 364 |
def test_translate_one_records_error_in_target():
|
| 365 |
state = h._make_state()
|
| 366 |
state["segments"] = _segments("source text")
|
|
|
|
| 395 |
return segments, []
|
| 396 |
|
| 397 |
with patch.object(h, "translate_segments", side_effect=fake_translate_segments), \
|
| 398 |
+
patch.object(h, "using_openrouter", return_value=True), \
|
| 399 |
+
patch.object(h, "log_event") as mock_log:
|
| 400 |
results = list(h._translate_all(state, "api-key", "model-x", *slot_values))
|
| 401 |
|
| 402 |
final = results[-1]
|
|
|
|
| 405 |
final_state = final[0]
|
| 406 |
assert [s["target"] for s in final_state["segments"]] == ["A", "B"]
|
| 407 |
|
| 408 |
+
event_type, session_id, payload = mock_log.call_args[0]
|
| 409 |
+
assert event_type == "translate"
|
| 410 |
+
assert payload["segment_count"] == 2
|
| 411 |
+
assert payload["segments"] == [{"source": "a", "target": "A"}, {"source": "b", "target": "B"}]
|
| 412 |
+
|
| 413 |
|
| 414 |
def test_translate_all_reports_errors_in_status():
|
| 415 |
state = h._make_state()
|
|
|
|
| 461 |
|
| 462 |
assert h._read_prompt() == "default"
|
| 463 |
|
| 464 |
+
state = h._make_state()
|
| 465 |
+
_, status = h._save_prompt(state, "edited")
|
| 466 |
assert status["value"] == "Prompt saved."
|
| 467 |
assert h._read_prompt() == "edited"
|
| 468 |
|
| 469 |
+
_, box_update, status_update = h._reset_prompt(state)
|
| 470 |
assert box_update["value"] == "default"
|
| 471 |
assert status_update["value"] == "Prompt reset to default."
|
| 472 |
+
|
| 473 |
+
|
| 474 |
+
def test_save_prompt_logs_prompt_save_event():
|
| 475 |
+
state = h._make_state()
|
| 476 |
+
with patch.object(h, "log_event") as mock_log:
|
| 477 |
+
h._save_prompt(state, "new prompt text")
|
| 478 |
+
event_type, session_id, payload = mock_log.call_args[0]
|
| 479 |
+
assert event_type == "prompt_save"
|
| 480 |
+
assert session_id == state["session_id"]
|
| 481 |
+
assert payload["prompt"] == "new prompt text"
|
| 482 |
+
|
| 483 |
+
|
| 484 |
+
def test_reset_prompt_logs_prompt_reset_event():
|
| 485 |
+
state = h._make_state()
|
| 486 |
+
with patch.object(h, "log_event") as mock_log:
|
| 487 |
+
h._reset_prompt(state)
|
| 488 |
+
event_type, session_id, payload = mock_log.call_args[0]
|
| 489 |
+
assert event_type == "prompt_reset"
|
| 490 |
+
assert session_id == state["session_id"]
|
| 491 |
+
|
| 492 |
+
|
| 493 |
+
# ── other interaction logging ───────────────────────────────────────────────
|
| 494 |
+
|
| 495 |
+
def test_load_resume_json_logs_resume_event():
|
| 496 |
+
state = h._make_state()
|
| 497 |
+
state["segments"] = _segments("a", "b", "c")
|
| 498 |
+
resume_content = json.dumps([
|
| 499 |
+
{"source": "a", "target": "A"},
|
| 500 |
+
{"source": "b", "target": ""},
|
| 501 |
+
{"source": "c", "target": "C"},
|
| 502 |
+
])
|
| 503 |
+
import tempfile
|
| 504 |
+
with tempfile.NamedTemporaryFile(suffix=".json", mode="w", delete=False) as f:
|
| 505 |
+
f.write(resume_content)
|
| 506 |
+
path = f.name
|
| 507 |
+
|
| 508 |
+
with patch.object(h, "log_event") as mock_log:
|
| 509 |
+
h._load_resume_json(path, state)
|
| 510 |
+
|
| 511 |
+
event_type, session_id, payload = mock_log.call_args[0]
|
| 512 |
+
assert event_type == "resume"
|
| 513 |
+
assert payload["filename"] == Path(path).name
|
| 514 |
+
assert payload["segment_count"] == 3
|
| 515 |
+
assert payload["segments"] == [
|
| 516 |
+
{"source": "a", "target": "A"},
|
| 517 |
+
{"source": "b", "target": ""},
|
| 518 |
+
{"source": "c", "target": "C"},
|
| 519 |
+
]
|
| 520 |
+
|
| 521 |
+
|
| 522 |
+
def test_handle_get_files_logs_download_event():
|
| 523 |
+
state = h._make_state()
|
| 524 |
+
state["segments"] = [{"source": "src", "target": "tgt"}]
|
| 525 |
+
state["label"] = "lbl"
|
| 526 |
+
with patch.object(h, "log_event") as mock_log:
|
| 527 |
+
h._handle_get_files(state, ["Translation (TXT)"])
|
| 528 |
+
event_type, session_id, payload = mock_log.call_args[0]
|
| 529 |
+
assert event_type == "download"
|
| 530 |
+
assert payload["formats"] == ["Translation (TXT)"]
|
| 531 |
+
assert payload["segments"] == [{"source": "src", "target": "tgt"}]
|
| 532 |
+
|
| 533 |
+
|
| 534 |
+
def test_handle_get_files_no_selection_does_not_log():
|
| 535 |
+
state = h._make_state()
|
| 536 |
+
state["segments"] = [{"source": "src", "target": "tgt"}]
|
| 537 |
+
with patch.object(h, "log_event") as mock_log:
|
| 538 |
+
h._handle_get_files(state, [])
|
| 539 |
+
mock_log.assert_not_called()
|
| 540 |
+
|
| 541 |
+
|
| 542 |
+
def test_handle_prev_and_next_log_navigate_events():
|
| 543 |
+
state = h._make_state()
|
| 544 |
+
state["segments"] = _segments(*[f"s{i}" for i in range(h.MAX_SLOTS + 3)])
|
| 545 |
+
state["page"] = 0
|
| 546 |
+
slot_values = [""] * h.MAX_SLOTS * 2
|
| 547 |
+
|
| 548 |
+
with patch.object(h, "log_event") as mock_log:
|
| 549 |
+
result = h._handle_next(state, *slot_values)
|
| 550 |
+
new_state = result[0]
|
| 551 |
+
event_type, session_id, payload = mock_log.call_args[0]
|
| 552 |
+
assert event_type == "navigate"
|
| 553 |
+
assert payload == {"direction": "next", "page": 1}
|
| 554 |
+
|
| 555 |
+
with patch.object(h, "log_event") as mock_log:
|
| 556 |
+
h._handle_prev(new_state, *slot_values)
|
| 557 |
+
event_type, session_id, payload = mock_log.call_args[0]
|
| 558 |
+
assert event_type == "navigate"
|
| 559 |
+
assert payload == {"direction": "prev", "page": 0}
|
| 560 |
+
|
| 561 |
+
|
| 562 |
+
def test_handle_cancel_translate_logs_event():
|
| 563 |
+
state = h._make_state()
|
| 564 |
+
with patch.object(h, "log_event") as mock_log:
|
| 565 |
+
h._handle_cancel_translate(state)
|
| 566 |
+
event_type, session_id, payload = mock_log.call_args[0]
|
| 567 |
+
assert event_type == "translate_cancel"
|
| 568 |
+
assert session_id == state["session_id"]
|
tests/test_usage_store.py
ADDED
|
@@ -0,0 +1,68 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Unit tests for engine.usage_store — usage telemetry written to the
|
| 2 |
+
billingsmoore/stui-usage HuggingFace dataset (private). All huggingface_hub
|
| 3 |
+
calls are mocked; no test in this file talks to the network.
|
| 4 |
+
"""
|
| 5 |
+
|
| 6 |
+
import json
|
| 7 |
+
from unittest.mock import MagicMock, patch
|
| 8 |
+
|
| 9 |
+
import pytest
|
| 10 |
+
|
| 11 |
+
from engine import usage_store
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
@pytest.fixture(autouse=True)
|
| 15 |
+
def no_token(monkeypatch):
|
| 16 |
+
monkeypatch.delenv("HF_TOKEN", raising=False)
|
| 17 |
+
|
| 18 |
+
|
| 19 |
+
@pytest.fixture
|
| 20 |
+
def with_token(monkeypatch):
|
| 21 |
+
monkeypatch.setenv("HF_TOKEN", "fake-token")
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
class TestNoToken:
|
| 25 |
+
def test_log_event_returns_false_without_network_call(self):
|
| 26 |
+
with patch("huggingface_hub.HfApi") as mock_api:
|
| 27 |
+
assert usage_store.log_event("upload", "session-1", {"foo": "bar"}) is False
|
| 28 |
+
mock_api.assert_not_called()
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
class TestLogEvent:
|
| 32 |
+
def test_no_session_id_returns_false_without_network_call(self, with_token):
|
| 33 |
+
with patch("huggingface_hub.HfApi") as mock_api:
|
| 34 |
+
assert usage_store.log_event("upload", "", {"foo": "bar"}) is False
|
| 35 |
+
mock_api.assert_not_called()
|
| 36 |
+
|
| 37 |
+
def test_writes_single_commit_with_expected_fields(self, with_token):
|
| 38 |
+
mock_api_instance = MagicMock()
|
| 39 |
+
with patch("huggingface_hub.HfApi", return_value=mock_api_instance):
|
| 40 |
+
ok = usage_store.log_event("translate", "session-1", {"backend": "local_cpu", "segment_count": 2})
|
| 41 |
+
|
| 42 |
+
assert ok is True
|
| 43 |
+
mock_api_instance.create_repo.assert_called_once_with(
|
| 44 |
+
repo_id=usage_store.DATASET_REPO_ID, repo_type="dataset", private=True, exist_ok=True,
|
| 45 |
+
)
|
| 46 |
+
mock_api_instance.create_commit.assert_called_once()
|
| 47 |
+
_, kwargs = mock_api_instance.create_commit.call_args
|
| 48 |
+
assert kwargs["repo_id"] == usage_store.DATASET_REPO_ID
|
| 49 |
+
assert kwargs["repo_type"] == "dataset"
|
| 50 |
+
|
| 51 |
+
operations = kwargs["operations"]
|
| 52 |
+
assert len(operations) == 1
|
| 53 |
+
op = operations[0]
|
| 54 |
+
assert op.path_in_repo.startswith("events/session-1/")
|
| 55 |
+
assert op.path_in_repo.endswith(".json")
|
| 56 |
+
|
| 57 |
+
payload = json.loads(op.path_or_fileobj.decode("utf-8"))
|
| 58 |
+
assert payload["event"] == "translate"
|
| 59 |
+
assert payload["session_id"] == "session-1"
|
| 60 |
+
assert payload["backend"] == "local_cpu"
|
| 61 |
+
assert payload["segment_count"] == 2
|
| 62 |
+
assert "timestamp" in payload
|
| 63 |
+
|
| 64 |
+
def test_network_error_fails_open(self, with_token):
|
| 65 |
+
mock_api_instance = MagicMock()
|
| 66 |
+
mock_api_instance.create_commit.side_effect = Exception("network error")
|
| 67 |
+
with patch("huggingface_hub.HfApi", return_value=mock_api_instance):
|
| 68 |
+
assert usage_store.log_event("edit", "session-1", {}) is False
|