Initial public release: Approach 1, Approach 2, Baseline + sample data

.gitignore

@@ -0,0 +1,26 @@
+# Python
+__pycache__/
+*.pyc
+*.pyo
+.venv/
+venv/
+
+# Streamlit
+.streamlit/secrets.toml
+
+# Jupyter
+.ipynb_checkpoints/
+
+# OS / editor
+.DS_Store
+Thumbs.db
+.vscode/
+.idea/
+
+# Anaconda envs
+*.conda
+*.egg-info/
+
+# Temp
+~WRL*.tmp
+*.tmp

README.md

@@ -0,0 +1,184 @@
+# Metadata Hierarchy Construction — TFM
+
+Master's thesis prototype: automatic hierarchy construction from data-dictionary metadata.
+Three algorithms are implemented for comparison.
+
+## Approaches
+
+- **Baseline** — Pure clustering baseline. Plain TF-IDF / Word2Vec embeddings + hierarchical
+  clustering. Documented in `README_baseline.md`.
+
+- **Approach 1** — Global embedding pipeline. Uses SBERT + N×M concept-table alignment
+  (Gonçalves 2019) + HiExpan refinement (Shen et al. KDD 2018) + Castanet parallel facets.
+  Optionally retrieves concept context from Wikidata / Wikipedia / WordNet / BioPortal.
+
+- **Approach 2** — Dataset-constrained multi-aspect hierarchy. Algorithmic pipeline with no
+  domain hardcoding:
+  1. Group-anchored L1/L2 (from detected metadata column structure — BISE 2026)
+  2. Phrase-slot mining (IE-style slot induction) for descriptions with regular structure
+  3. **FASTopic** semantic aspect discovery (Wu et al. NeurIPS 2024) — replaces NMF
+  4. NMF lexical fallback for small groups
+  5. GMM + BIC for small clusters, MiniBatchKMeans + silhouette for large ones
+  6. Deterministic 5-stage label generation (description prefix → group anchor → IDF filter
+     → bigram-preferred TF-IDF → optional LLM refinement)
+  7. **Optional local-LLM label refinement** via Ollama + Qwen 2.5 (TopicTag pattern, DocEng
+     2024). Strict grounding check rejects labels not derived from CSV evidence. Per-node
+     provenance recorded.
+  8. TraCo-inspired hierarchy diagnostics (AAAI 2024)
+
+  No facet trees — single coherent LoD tree.
+
+See each script's "Method" tab in the running app for the full algorithm and paper references.
+
+## Paper stack
+
+| Component | Paper |
+|---|---|
+| Multi-aspect taxonomy scaffold | Zhu et al. 2025, EMNLP |
+| Canonical metadata text objects | Gonçalves et al. 2019, ESWC |
+| Semantic aspect discovery | Wu et al. 2024 (FASTopic), NeurIPS, arXiv:2405.17978 |
+| Phrase-slot mining | IE / slot-induction literature (ACM CSUR 2022) |
+| LLM label refinement pattern | Eren et al. 2024 (TopicTag), DocEng, arXiv:2407.19616 |
+| Local LLM (used for refinement) | Qwen Team 2024 (Qwen 2.5), arXiv:2412.15115 |
+| Hierarchy quality diagnostics | Wu et al. 2024 (TraCo), AAAI, arXiv:2401.14113 |
+| Group-anchored entry strategy | Motamedi, Novalija, Rei 2026, Springer BISE |
+| Multidimensional taxonomy motivation | Kargupta et al. 2025 (TaxoAdapt), ACL |
+| Future-work semantic consistency | SC-Taxo 2026, arXiv:2605.00620 |
+| Concept-label evaluation framework | Kejriwal et al. 2022 (TICL), EAAI |
+
+## Project layout
+
+```
+Hierarchy tool/
+├── baseline.py          # Pure clustering baseline (Streamlit app)
+├── approach_1.py        # Approach 1 (Streamlit app)
+├── approach_2.py        # Approach 2 (Streamlit app)
+├── approach_1.ipynb     # Approach 1 reproducible notebook
+├── approach_2.ipynb     # Approach 2 reproducible notebook
+├── baseline.ipynb       # Baseline reproducible notebook
+├── launcher.py          # Run all three apps simultaneously on different ports
+├── data/                # Sample input CSVs (AI-MIND, HCP, etc.)
+├── outputs/             # Generated hierarchies (JSON)
+└── requirements.txt
+```
+
+## Running locally
+
+### 1. Install Python dependencies
+
+```bash
+pip install -r requirements.txt
+```
+
+Python 3.10 or 3.11 recommended.
+
+### 2. (Approach 2 only) Install Ollama for the local-LLM label refinement layer
+
+**This is optional — Approach 2 produces deterministic labels without it.**  If you want
+the optional TopicTag-style LLM label refinement:
+
+1. Download and install Ollama from https://ollama.com/download
+2. Open Ollama once so the background service starts (icon in the system tray)
+3. Pull the recommended model:
+   ```bash
+   ollama pull qwen2.5:3b-instruct
+   ```
+   (For higher quality at higher RAM cost: `ollama pull qwen2.5:7b-instruct`.)
+4. Verify the server is reachable:
+   - In a browser open `http://localhost:11434/api/tags`
+   - Or run `ollama list`
+
+When Approach 2 starts it auto-detects Ollama and the "Refine labels with LLM" checkbox
+defaults to ON. Uncheck any time. The deterministic pipeline is the canonical thesis
+result; the LLM is an optional re-phraser of evidence already in the CSV.
+
+To override the default URL or model:
+
+```bash
+# Optional environment variables
+set OLLAMA_URL=http://localhost:11434/v1
+set OLLAMA_MODEL=qwen2.5:3b-instruct
+```
+
+Or change them live in the Approach 2 sidebar.
+
+### 3. Run one app at a time
+
+```bash
+streamlit run baseline.py
+# or
+streamlit run approach_1.py
+# or
+streamlit run approach_2.py
+```
+
+Each opens at http://localhost:8501 by default.
+
+### 4. Run all three apps simultaneously (for side-by-side comparison)
+
+```bash
+python launcher.py
+```
+
+This opens three browser tabs:
+
+- http://localhost:8501 — Baseline
+- http://localhost:8502 — Approach 1
+- http://localhost:8503 — Approach 2
+
+Press **Enter** in the launcher terminal to stop all servers.
+
+## Using the apps
+
+1. Upload one or more metadata CSV / TSV / XLSX / JSON files in the sidebar.
+2. Confirm the auto-detected column roles (leaf / group / text / meta).
+3. Click **Build hierarchy**.
+4. Inspect the LoD tree, evaluation metrics, label provenance (Approach 2), and export JSON.
+
+Sample data is in `data/`:
+- `ai-mind-variable-descriptions(in).csv`
+- `HCP_S1200_DataDictionary_Oct_30_2023.csv`
+
+## Outputs
+
+- **Baseline / Approach 1** export two JSON files compatible with the VIANNA viewer:
+  - `*_lod.json` — primary LoD tree
+  - `*_facets.json` — parallel Castanet facet trees
+
+- **Approach 2** exports a single LoD JSON:
+  - `*_approach2_lod.json` — primary LoD tree (every aggregation node carries
+    `label_provenance` with source stage, confidence, and evidence terms)
+
+Filenames are derived from the uploaded CSV file name, so different CSVs export under
+different filenames into `outputs/approach 2/`.
+
+Existing output examples are in `outputs/approach 1/` and `outputs/approach 2/`.
+
+## Defensibility highlights for Approach 2
+
+- **No domain hardcoding.** Slot names, group anchors, and labels are all derived from the
+  detected metadata columns + the uploaded CSV — no hand-curated domain vocabulary.
+- **Deterministic by default.** Tree topology and all five label-generation stages are
+  reproducible from the input CSV alone. Local LLM is opt-in.
+- **Grounded LLM refinement.** Every LLM-proposed label must pass a strict grounding
+  check — every word in the label must appear in the extracted evidence. Failed proposals
+  are rejected and the deterministic label is used instead. Per-node provenance lets
+  you answer "did the LLM invent this?" with hard evidence.
+- **Local-only LLM.** Qwen 2.5 runs on the thesis machine via Ollama. No external API
+  calls, no third-party data sharing, no key management.
+
+## Troubleshooting
+
+| Symptom | Fix |
+|---|---|
+| `FASTopic not installed` warning | `pip install fastopic` (also installs `torch`) |
+| `openai` package missing | `pip install openai` |
+| `Ollama not reachable` in sidebar | Open the Ollama app from Start menu; the service runs in the system tray |
+| Model not found | `ollama pull qwen2.5:3b-instruct` |
+| Build very slow with LLM on | Expected for HCP — ~15–40 min on CPU with a 3B model. Disable LLM for fast iteration. |
+| `LLM-labeled nodes: 0/N` after build | The grounding check rejected every LLM proposal. Check the **🔍 Label Provenance** tab — counts under `llm_rejected = True` show what happened. |
+| Hierarchy too shallow | Increase `Max LoD tree depth` slider (top of sidebar in Approach 2) |
+
+## License
+
+For thesis evaluation only.

baseline.py

@@ -0,0 +1,760 @@
+# baseline.py — Metadata Hierarchy Builder — Baseline (Taxonomizer)
+#
+# Pure Taxonomizer baseline — NO hardcoded, domain-specific patterns.
+# The only lexical resource is a generic English stop-word list (standard
+# information-retrieval practice, not dataset-specific).
+#
+# Pipeline (dataset-only, no external APIs, no sentence-transformers):
+#   1. Load metadata file (CSV / TSV / XLSX / JSON)
+#   2. Detect column roles (leaf / group / text / meta)
+#   3. Build canonical schema (_leaf_id, _leaf_label, _group_path, _text)
+#   4. Represent each variable as a TF-IDF text object
+#   5. Recursively cluster variables (agglomerative, cosine distance) into an
+#      abstract-to-concrete taxonomy; internal-node labels are the most
+#      discriminative terms of each cluster — derived from the data, not hardcoded
+#   6. Visualise (Sunburst / Treemap)
+#   7. Export VIANNA-compatible JSON + canonical CSV
+#
+# Papers:
+#   [TAX] Taxonomizer (Sultanum et al.) — leaf=attribute, internal=abstract group
+#         built bottom-up by recursively clustering item feature vectors and
+#         labelling each internal node with its members' shared/discriminative terms
+#   [GON] Goncalves et al. — TF-IDF text objects + cosine distance
+#   [HIE] HiExpan (adapted) — discriminative-term node labelling
+
+from __future__ import annotations
+import csv, json, re, warnings
+from collections import defaultdict
+from pathlib import Path
+import tempfile
+
+import numpy as np
+import pandas as pd
+import plotly.graph_objects as go
+import streamlit as st
+from sklearn.cluster import AgglomerativeClustering
+from sklearn.feature_extraction.text import TfidfVectorizer
+from sklearn.metrics import normalized_mutual_info_score, adjusted_rand_score, silhouette_score
+from sklearn.metrics.pairwise import cosine_distances
+from sklearn.preprocessing import LabelEncoder
+
+warnings.filterwarnings('ignore')
+
+st.set_page_config(page_title='Metadata Hierarchy — Baseline', page_icon='🌿', layout='wide')
+st.title('Metadata Hierarchy Builder — Baseline (Taxonomizer)')
+st.caption(
+    'Pure Taxonomizer baseline: TF-IDF text objects + recursive agglomerative '
+    'clustering into an abstract-to-concrete taxonomy, with internal-node labels '
+    'derived from each cluster’s discriminative terms. No hardcoded domain '
+    'patterns, no external APIs, no sentence embeddings — works on any dataset.'
+)
+
+# ─────────────────────────────────────────────────────────────────────────────
+# CONSTANTS
+# ─────────────────────────────────────────────────────────────────────────────
+LEAF_KEYS  = 'variable var field column attribute name code id item indicator question measure concept'.split()
+GROUP_KEYS = 'task category domain module section table dataset assessment test variant group topic instrument form subscale construct'.split()
+TEXT_KEYS  = 'description definition desc label title question meaning note notes text display full details explanation comment'.split()
+META_KEYS  = 'type dtype data_type datatype unit units format decimal precision values value coding codebook range min max scale'.split()
+
+# ─────────────────────────────────────────────────────────────────────────────
+# FILE LOADING
+# ─────────────────────────────────────────────────────────────────────────────
+def safe_name(name: str) -> str:
+    return ''.join(ch if ch.isalnum() or ch in '-_.' else '_' for ch in name)
+
+def try_read_csv(path: Path) -> pd.DataFrame:
+    best, best_score = None, -1
+    for enc in ['utf-8-sig', 'utf-8', 'latin1']:
+        for sep in [None, ',', '\t', ';', '|']:
+            try:
+                df = pd.read_csv(path, sep=sep, engine='python', encoding=enc)
+                score = df.shape[1] * 10 - float(df.isna().mean().mean())
+                if score > best_score:
+                    best, best_score = df, score
+            except Exception:
+                pass
+    if best is None:
+        raise ValueError(f'Could not read {path.name}')
+    best.columns = [str(c).strip().replace(';', '') for c in best.columns]
+    # Repair comma-packed rows (AI-Mind format)
+    if len(best) > 0:
+        first = best.iloc[:, 0].astype(str)
+        other_null = best.iloc[:, 1:].isna().mean().mean() if best.shape[1] > 1 else 1.0
+        if first.str.contains(',').mean() > 0.50 and other_null > 0.70:
+            lines = path.read_text(encoding='utf-8-sig', errors='replace').splitlines()
+            if lines:
+                header = [h.strip().replace(';', '') for h in lines[0].split(',')]
+                rows = []
+                for line in lines[1:]:
+                    line = line.strip().rstrip(';')
+                    if not line:
+                        continue
+                    if line.startswith('"') and line.endswith('"'):
+                        line = line[1:-1]
+                    try:
+                        parts = next(csv.reader([line], quotechar='"'))
+                    except Exception:
+                        continue
+                    if len(parts) >= len(header):
+                        rows.append(parts[:len(header)])
+                if rows:
+                    best = pd.DataFrame(rows, columns=header)
+    best.columns = [str(c).strip().replace(';', '') for c in best.columns]
+    return best
+
+def load_any(path: Path) -> pd.DataFrame:
+    s = path.suffix.lower()
+    if s in ['.csv', '.tsv', '.txt']:
+        return try_read_csv(path)
+    if s in ['.xlsx', '.xls']:
+        return pd.read_excel(path)
+    if s == '.json':
+        obj = json.loads(path.read_text(encoding='utf-8', errors='replace'))
+        if isinstance(obj, list):
+            return pd.json_normalize(obj)
+        if isinstance(obj, dict):
+            for v in obj.values():
+                if isinstance(v, list):
+                    return pd.json_normalize(v)
+    raise ValueError(f'Unsupported file type: {s}')
+
+def save_upload(f) -> Path:
+    tmp = Path(tempfile.mkdtemp(prefix='baseline_'))
+    p = tmp / safe_name(f.name)
+    p.write_bytes(f.getbuffer())
+    return p
+
+# ─────────────────────────────────────────────────────────────────────────────
+# ROLE DETECTION  [GON]
+# ─────────────────────────────────────────────────────────────────────────────
+def norm(c: str) -> str:
+    return re.sub(r'[^a-z0-9]+', '_', str(c).strip().lower()).strip('_')
+
+def kscore(c: str, keys: list) -> int:
+    nc = norm(c)
+    return sum(1 for k in keys if k in nc)
+
+def profile_columns(df: pd.DataFrame) -> pd.DataFrame:
+    out = []
+    n = max(len(df), 1)
+    for col in df.columns:
+        s = df[col]
+        non = float(s.notna().mean())
+        nun = int(s.nunique(dropna=True))
+        ur  = nun / n
+        avg = float(s.dropna().astype(str).map(len).mean()) if s.notna().any() else 0
+        out.append({
+            'column':         str(col),
+            'non_null':       round(non, 3),
+            'unique_values':  nun,
+            'unique_ratio':   round(ur, 3),
+            'avg_length':     round(avg, 1),
+            'leaf_score':     4*kscore(col, LEAF_KEYS)  + (3 if 0.5 <= ur <= 1 else 0) + (1 if avg < 80 else 0),
+            'group_score':    4*kscore(col, GROUP_KEYS) + (3 if 1 < nun < min(n*0.5, 80) else 0) + (1 if avg < 60 else 0),
+            'text_score':     5*kscore(col, TEXT_KEYS)  + (4 if avg > 50 else 0) + (1 if non > 0.5 else 0),
+            'metadata_score': 4*kscore(col, META_KEYS)  + (2 if 1 < nun < min(n*0.8, 100) else 0),
+        })
+    return pd.DataFrame(out)
+
+def detect_roles(df: pd.DataFrame) -> tuple:
+    prof  = profile_columns(df)
+    leaf  = prof.sort_values(['leaf_score', 'unique_ratio'], ascending=False).head(1)['column'].tolist()
+    text  = (prof[(prof.text_score >= 4) | (prof.avg_length > 80)]
+             .sort_values('text_score', ascending=False)['column'].tolist()) or leaf.copy()
+    group = (prof[(prof.group_score >= 4) & (~prof.column.isin(leaf)) & (prof.unique_values > 1)]
+             .sort_values('group_score', ascending=False)['column'].head(3).tolist())
+    meta  = (prof[(prof.metadata_score >= 4) & (~prof.column.isin(text + leaf + group))]
+             .sort_values('metadata_score', ascending=False)['column'].head(5).tolist())
+    return {'leaf_cols': leaf, 'group_cols': group, 'text_cols': text, 'metadata_cols': meta}, prof
+
+# ─────────────────────────────────────────────────────────────────────────────
+# CANONICAL SCHEMA  [GON]
+# ─────────────────────────────────────────────────────────────────────────────
+def sv(x) -> str:
+    return '' if pd.isna(x) else str(x).strip()
+
+def build_canonical(df: pd.DataFrame, cfg: dict, source: str) -> pd.DataFrame:
+    leaf_cols  = cfg.get('leaf_cols', [])
+    group_cols = cfg.get('group_cols', [])
+    text_cols  = cfg.get('text_cols', [])
+    meta_cols  = cfg.get('metadata_cols', [])
+    rows = []
+    for i, row in df.iterrows():
+        leaf_parts  = [sv(row.get(c, '')) for c in leaf_cols]
+        leaf_parts  = [p for p in leaf_parts if p]
+        label       = ' / '.join(leaf_parts) if leaf_parts else f'variable_{i+1}'
+        group_parts = [sv(row.get(c, '')) for c in group_cols]
+        group_parts = [p for p in group_parts if p and p.lower() not in ['nan', 'none']]
+        gpath       = ' > '.join(group_parts) if group_parts else 'Ungrouped'
+        parts = []
+        for c in list(dict.fromkeys(group_cols + leaf_cols + text_cols + meta_cols)):
+            v = sv(row.get(c, ''))
+            if v:
+                parts.append(f'{c}: {v}')
+        text = ' | '.join(parts) if parts else label
+        rows.append({
+            '_source_file': source,
+            '_row_index':   int(i),
+            '_leaf_label':  label,
+            '_leaf_id':     f'{gpath}.{label}' if gpath != 'Ungrouped' else label,
+            '_group_path':  gpath,
+            '_text':        text,
+        })
+    can = pd.DataFrame(rows)
+    if can['_leaf_id'].duplicated().any():
+        cnt: dict = defaultdict(int)
+        ids = []
+        for lid in can['_leaf_id']:
+            cnt[lid] += 1
+            ids.append(lid if cnt[lid] == 1 else f'{lid}__{cnt[lid]}')
+        can['_leaf_id'] = ids
+    return can
+
+# ─────────────────────────────────────────────────────────────────────────────
+# TAXONOMIZER CORE  [TAX + GON]
+#
+# Everything here is data-driven: TF-IDF over the variable text objects, cosine
+# distance, agglomerative clustering with the number of clusters chosen by
+# silhouette, and internal-node labels taken from each cluster's most
+# discriminative terms.  The ONLY lexical resource is the generic English
+# stop-word list (standard IR practice — not dataset-specific).
+# ─────────────────────────────────────────────────────────────────────────────
+def vectorize_texts(texts: list):
+    """TF-IDF text objects [GON].  Generic English stop-words only."""
+    vec = TfidfVectorizer(stop_words='english', ngram_range=(1, 2),
+                          max_features=2000, min_df=1, sublinear_tf=True)
+    X = vec.fit_transform(texts)
+    return X, vec
+
+def best_k(dist: np.ndarray, n: int, k_min: int = 2, k_max: int = 8) -> int:
+    """Pick the number of clusters that maximises the silhouette score.
+
+    Fully data-driven — no fixed cluster count.  Returns 1 if no split with
+    >=2 clusters is well separated.
+    """
+    k_hi = min(k_max, n - 1)
+    if k_hi < k_min:
+        return 1
+    best, best_s = 1, -1.0
+    for k in range(k_min, k_hi + 1):
+        labels = AgglomerativeClustering(n_clusters=k, metric='precomputed',
+                                         linkage='average').fit_predict(dist)
+        if len(set(labels)) < 2:
+            continue
+        try:
+            s = silhouette_score(dist, labels, metric='precomputed')
+        except Exception:
+            continue
+        if s > best_s:
+            best_s, best = s, k
+    return best
+
+def discriminative_label(inside: np.ndarray, outside, terms: np.ndarray,
+                         used: set, top_n: int = 2) -> str:
+    """Label a cluster by the terms that most separate it from its siblings.
+
+    inside  = mean TF-IDF vector of the cluster's members
+    outside = mean TF-IDF vector of the sibling pool (or 0 if none)
+    """
+    scores = inside - (outside if outside is not None else 0)
+    picks: list = []
+    for i in np.argsort(scores)[::-1]:
+        term = terms[i]
+        if len(term) <= 2 or scores[i] <= 0 or term in used:
+            continue
+        picks.append(term)
+        if len(picks) >= top_n:
+            break
+    if not picks:  # degenerate: fall back to highest raw mean term
+        for i in np.argsort(inside)[::-1]:
+            if len(terms[i]) > 2:
+                picks = [terms[i]]
+                break
+    return ' / '.join(p.title() for p in picks) if picks else 'Group'
+
+# ─────────────────────────────────────────────────────────────────────────────
+# HIERARCHY CONSTRUCTION  [TAX + GON]
+# ─────────────────────────────────────────────────────────────────────────────
+def _nmap(nodes: list) -> dict:
+    return {int(n['id']): n for n in nodes}
+
+def _next_id(nodes: list) -> int:
+    return max((int(n['id']) for n in nodes), default=0) + 1
+
+def _add_child(nodes: list, parent_id: int, child_id: int):
+    m = _nmap(nodes)
+    p = m.get(int(parent_id))
+    if p is None:
+        return
+    rel = list(p.get('related', []))
+    if int(child_id) not in rel:
+        rel.append(int(child_id))
+    p['related'] = rel
+
+def _make_agg(nid: int, name: str, desc: str = '') -> dict:
+    return {'id': int(nid), 'name': str(name), 'related': [],
+            'type': 'aggregation', 'isShown': True, 'desc': desc, 'dtype': 'determine'}
+
+def _leaf_ids(nodes: list, nid: int) -> list:
+    m = _nmap(nodes)
+    out: list = []
+    def rec(x):
+        n = m.get(int(x))
+        if not n:
+            return
+        if n.get('type') == 'attribute':
+            out.append(int(x))
+            return
+        for c in n.get('related', []):
+            rec(int(c))
+    rec(nid)
+    return list(dict.fromkeys(out))
+
+def build_hierarchy(can: pd.DataFrame, project: str = 'project',
+                    max_depth: int = 3, min_cluster_size: int = 6,
+                    branch_max: int = 8) -> list:
+    """Pure Taxonomizer construction [TAX].
+
+    Builds an abstract-to-concrete taxonomy by recursively clustering the
+    variables' TF-IDF text objects.  At each level the number of clusters is
+    chosen by silhouette; each resulting internal node is labelled with the
+    terms that most discriminate its members from their siblings.  No group
+    column, no hardcoded patterns are used in construction — so the recovered
+    structure can be fairly evaluated against the original group column.
+    """
+    # ── leaf attribute nodes (ids 1..N) ──────────────────────────────────────
+    nodes: list = [{'id': 0, 'name': project, 'type': 'root',
+                    'dtype': 'root', 'isShown': True, 'related': [], 'desc': 'Root node'}]
+    row_to_node: list = []
+    for i, (_, r) in enumerate(can.iterrows(), start=1):
+        nodes.append({'id': i, 'name': r['_leaf_label'], 'dtype': 'determine',
+                      'related': [], 'isShown': True, 'type': 'attribute',
+                      'desc': r['_text'],
+                      'metadata': {'leaf_id': r['_leaf_id'], 'group_path': r['_group_path']}})
+        row_to_node.append(i)
+    row_to_node = np.array(row_to_node)
+
+    # ── TF-IDF text objects + full cosine distance matrix [GON] ───────────────
+    texts = (can['_leaf_label'].astype(str) + ' . ' + can['_text'].astype(str)).tolist()
+    X, vec = vectorize_texts(texts)
+    Xd     = X.toarray()
+    terms  = vec.get_feature_names_out()
+    full_dist = cosine_distances(X).astype(float)
+    np.fill_diagonal(full_dist, 0.0)
+
+    # ── recursive clustering ─────────────────────────────────────────────────
+    def attach_leaves(parent_id: int, idx: np.ndarray):
+        for i in idx:
+            _add_child(nodes, parent_id, int(row_to_node[i]))
+
+    def recurse(parent_id: int, idx: np.ndarray, depth: int, used: set):
+        n = len(idx)
+        if n <= min_cluster_size or depth >= max_depth:
+            attach_leaves(parent_id, idx)
+            return
+
+        sub = full_dist[np.ix_(idx, idx)]
+        k_cap = min(branch_max, max(2, n // min_cluster_size))
+        k = best_k(sub, n, k_min=2, k_max=k_cap)
+        if k <= 1:
+            attach_leaves(parent_id, idx)
+            return
+
+        labels  = AgglomerativeClustering(n_clusters=k, metric='precomputed',
+                                          linkage='average').fit_predict(sub)
+        pool_Xd = Xd[idx]
+        for c in range(k):
+            mask    = labels == c
+            members = idx[mask]
+            if len(members) == 0:
+                continue
+            if len(members) == 1:           # don't create singleton internal nodes
+                _add_child(nodes, parent_id, int(row_to_node[members[0]]))
+                continue
+            inside  = pool_Xd[mask].mean(axis=0)
+            outside = pool_Xd[~mask].mean(axis=0) if (~mask).any() else None
+            label   = discriminative_label(inside, outside, terms, used)
+            nid = _next_id(nodes)
+            nodes.append(_make_agg(nid, label,
+                                   desc=f'Cluster of {len(members)} variables — '
+                                        f'discriminative terms: {label}'))
+            _add_child(nodes, parent_id, nid)
+            recurse(nid, members, depth + 1, used | {label.lower()})
+
+    recurse(0, np.arange(len(can)), 0, set())
+
+    for n in nodes:
+        n['related'] = list(dict.fromkeys(int(x) for x in n.get('related', [])))
+    return nodes
+
+# ─────────────────────────────────────────────────────────────────────────────
+# VISUALISATION
+# ─────────────────────────────────────────────────────────────────────────────
+def _parent_map(nodes: list) -> dict:
+    pm: dict = {}
+    for n in nodes:
+        for c in n.get('related', []):
+            if int(c) not in pm:
+                pm[int(c)] = int(n['id'])
+    return pm
+
+# ─────────────────────────────────────────────────────────────────────────────
+# EVALUATION HELPERS
+# ─────────────────────────────────────────────────────────────────────────────
+def _eval_cluster_assignments(nodes: list, can: pd.DataFrame) -> list[int]:
+    """Return predicted cluster id (depth-1 aggregation ancestor) for each row in can."""
+    pm = _parent_map(nodes)
+    def depth1(nid: int) -> int:
+        # Walk up until our parent is root (id==0) or we have no parent
+        while pm.get(nid, -1) not in (-1, 0):
+            nid = pm[nid]
+        return nid
+    lid_to_nid = {n['metadata']['leaf_id']: int(n['id'])
+                  for n in nodes if n.get('type') == 'attribute' and 'metadata' in n}
+    return [depth1(lid_to_nid[lid]) if lid in lid_to_nid else -1
+            for lid in can['_leaf_id']]
+
+def _purity(y_true, y_pred) -> float:
+    from collections import Counter
+    clusters: dict = {}
+    for t, p in zip(y_true, y_pred):
+        clusters.setdefault(p, []).append(t)
+    correct = sum(Counter(v).most_common(1)[0][1] for v in clusters.values())
+    return correct / max(len(y_true), 1)
+
+def _structural_stats(nodes: list) -> dict:
+    pm = _parent_map(nodes)
+    def depth_of(nid: int) -> int:
+        d = 0
+        while nid in pm:
+            nid = pm[nid]; d += 1
+        return d
+    agg   = [n for n in nodes if n.get('type') == 'aggregation']
+    leafs = [n for n in nodes if n.get('type') == 'attribute']
+    depths   = [depth_of(int(n['id'])) for n in leafs]
+    branches = [len(n.get('related', [])) for n in agg]
+    singletons = sum(1 for b in branches if b == 1)
+    return {
+        'n_aggregation_nodes':  len(agg),
+        'max_depth':            int(max(depths, default=0)),
+        'avg_leaf_depth':       round(float(np.mean(depths)), 2) if depths else 0.0,
+        'avg_branching_factor': round(float(np.mean(branches)), 2) if branches else 0.0,
+        'singleton_nodes_%':    round(100.0 * singletons / max(len(agg), 1), 1),
+    }
+
+def _wrap(text: str, width: int = 70) -> str:
+    """Wrap long hover text onto multiple <br> lines so it never runs off-screen."""
+    import textwrap
+    text = str(text).replace('<', '&lt;')
+    lines: list = []
+    for para in text.split('\n'):
+        wrapped = textwrap.wrap(para, width=width) or ['']
+        lines.extend(wrapped)
+    return '<br>'.join(lines)
+
+def plot_sunburst(nodes: list, max_depth: int = 4) -> go.Figure:
+    pm = _parent_map(nodes)
+    ids, labels, parents, values, hover = [], [], [], [], []
+    for n in nodes:
+        nid = int(n['id'])
+        lc  = len(_leaf_ids(nodes, nid))
+        ids.append(str(nid))
+        labels.append(str(n.get('name', ''))[:40])
+        parents.append('' if nid == 0 else str(pm.get(nid, 0)))
+        values.append(max(1, lc))
+        desc = _wrap(n.get('desc', ''))
+        hover.append(f'<b>{_wrap(n.get("name",""))}</b><br>Type: {n.get("type","")}'
+                     f'<br>Variables: {lc}<br><br>{desc}')
+    fig = go.Figure(go.Sunburst(
+        ids=ids, labels=labels, parents=parents, values=values,
+        branchvalues='total', hovertext=hover, hoverinfo='text',
+        maxdepth=max_depth, insidetextorientation='radial',
+        marker=dict(colorscale='Greens', line=dict(width=1, color='white')),
+    ))
+    fig.update_layout(height=700, margin=dict(l=10, r=10, t=40, b=10),
+                      title='Click a sector to drill down — click centre to go back')
+    return fig
+
+def plot_treemap(nodes: list) -> go.Figure:
+    pm = _parent_map(nodes)
+    ids, labels, parents, values, hover = [], [], [], [], []
+    for n in nodes:
+        nid = int(n['id'])
+        lc  = len(_leaf_ids(nodes, nid))
+        ids.append(str(nid))
+        labels.append(str(n.get('name', ''))[:40])
+        parents.append('' if nid == 0 else str(pm.get(nid, 0)))
+        values.append(max(1, lc))
+        desc = _wrap(n.get('desc', ''))
+        hover.append(f'<b>{_wrap(n.get("name",""))}</b><br>Variables: {lc}<br>{desc}')
+    fig = go.Figure(go.Treemap(
+        ids=ids, labels=labels, parents=parents, values=values,
+        branchvalues='total', hovertext=hover, hoverinfo='text',
+        textinfo='label+value',
+        marker=dict(colorscale='Greens', line=dict(width=1, color='white')),
+    ))
+    fig.update_layout(height=700, margin=dict(l=10, r=10, t=10, b=10))
+    return fig
+
+# ───────────────────────────────────────────────────────────────���─────────────
+# SIDEBAR
+# ─────────────────────────────────────────────────────────────────────────────
+with st.sidebar:
+    st.header('1. Upload')
+    uploaded = st.file_uploader(
+        'Upload a metadata file',
+        type=['csv', 'tsv', 'txt', 'xlsx', 'xls', 'json'],
+        accept_multiple_files=False,
+    )
+    st.header('2. Taxonomizer settings')
+    tx_max_depth = st.slider('Max taxonomy depth', 2, 5, 3, 1,
+                             help='How many abstract-to-concrete levels to build')
+    tx_min_size  = st.slider('Min cluster size', 3, 20, 6, 1,
+                             help='Clusters smaller than this stop splitting (leaves attach directly)')
+    tx_branch    = st.slider('Max branches per node', 3, 12, 8, 1,
+                             help='Upper bound on clusters per split; the actual number is chosen by silhouette')
+
+    st.header('3. Display')
+    max_items     = st.slider('Maximum variables', 25, 1200, 300, 25)
+    group_filter  = st.text_input('Group filter (optional)', value='',
+                                  help='Filter rows whose group path contains this text')
+    display_depth = st.slider('Sunburst depth', 2, 6, 4, 1)
+
+# ─────────────────────────────────────────────────────────────────────────────
+# MAIN
+# ─────────────────────────────────────────────────────────────────────────────
+if not uploaded:
+    st.info('Upload a metadata CSV / XLSX / JSON file to begin.')
+    st.markdown("""
+    ### Baseline algorithm — pure Taxonomizer
+
+    The simplest of the three approaches — no hardcoded domain patterns, no
+    external APIs, no neural embeddings. Works on any dataset.
+
+    | Step | Method | Paper |
+    |------|--------|-------|
+    | Text object | Concatenate all metadata fields per variable | Goncalves et al. |
+    | Representation | TF-IDF (generic English stop-words only) | Goncalves et al. |
+    | Hierarchy construction | Recursive agglomerative clustering (cosine), #clusters chosen by silhouette | Taxonomizer (Sultanum et al.) |
+    | Node labelling | Most discriminative terms of each cluster vs its siblings | Taxonomizer / HiExpan |
+
+    The group column is **not** used for construction, so the recovered taxonomy
+    can be fairly evaluated against it (NMI / ARI / Purity in the Evaluation tab).
+
+    **Approach 1** adds SBERT embeddings + Wikidata/BioPortal enrichment + HiExpan refinement.
+
+    **Approach 2** adds NMF/FASTopic aspect discovery + GMM clustering + optional LLM labels.
+    """)
+    st.stop()
+
+path = save_upload(uploaded)
+
+@st.cache_data(show_spinner=False)
+def _load_profile(path_str: str):
+    df = load_any(Path(path_str))
+    cfg, prof = detect_roles(df)
+    return df, cfg, prof
+
+with st.spinner('Loading file…'):
+    df, auto_cfg, prof = _load_profile(str(path))
+
+st.subheader('Step 1 — File preview')
+with st.expander(f'📄 {uploaded.name}  ({len(df):,} rows, {len(df.columns)} columns)',
+                 expanded=False):
+    st.dataframe(df.head(10), use_container_width=True)
+    score_cols = [c for c in ['column', 'leaf_score', 'group_score', 'text_score', 'metadata_score']
+                  if c in prof.columns]
+    st.dataframe(prof[score_cols].sort_values('leaf_score', ascending=False),
+                 use_container_width=True)
+
+st.subheader('Step 2 — Confirm column roles')
+cols = list(df.columns)
+with st.expander('Column configuration', expanded=True):
+    left, right = st.columns(2)
+    with left:
+        leaf_cols = st.multiselect('Leaf variable column(s)', cols,
+            default=[c for c in auto_cfg.get('leaf_cols', []) if c in cols], key='leaf')
+        group_cols = st.multiselect('Group/task column(s)', cols,
+            default=[c for c in auto_cfg.get('group_cols', []) if c in cols], key='group')
+    with right:
+        text_cols = st.multiselect('Text/description column(s)', cols,
+            default=[c for c in auto_cfg.get('text_cols', []) if c in cols], key='text')
+        meta_cols = st.multiselect('Metadata/type column(s)', cols,
+            default=[c for c in auto_cfg.get('metadata_cols', []) if c in cols], key='meta')
+
+if not leaf_cols:
+    st.error('Choose at least one leaf variable column.')
+    st.stop()
+
+cfg = {'leaf_cols': leaf_cols, 'group_cols': group_cols,
+       'text_cols': text_cols, 'metadata_cols': meta_cols}
+
+if st.button('Build baseline hierarchy', type='primary'):
+    with st.spinner('Building hierarchy…'):
+        _can = build_canonical(df, cfg, source=Path(uploaded.name).stem)
+
+        if group_filter.strip():
+            _can = _can[_can['_group_path'].str.contains(
+                group_filter.strip(), case=False, na=False)].copy()
+
+        if len(_can) > max_items:
+            _can = _can.head(max_items).copy()
+
+        _can = _can.reset_index(drop=True)
+
+        if len(_can) < 2:
+            st.error('Need at least 2 variables after filtering.')
+            st.stop()
+
+        _pname = Path(uploaded.name).stem
+        _nodes = build_hierarchy(_can, project=_pname,
+                                 max_depth=tx_max_depth,
+                                 min_cluster_size=tx_min_size,
+                                 branch_max=tx_branch)
+
+    st.session_state['_bl_nodes']   = _nodes
+    st.session_state['_bl_can']     = _can
+    st.session_state['_bl_project'] = _pname
+
+if '_bl_nodes' not in st.session_state:
+    st.info('Configure columns above then click **Build baseline hierarchy**.')
+    st.stop()
+
+nodes        = st.session_state['_bl_nodes']
+can          = st.session_state['_bl_can']
+project_name = st.session_state['_bl_project']
+
+_sm = _structural_stats(nodes)
+n_leaves   = len([n for n in nodes if n['type'] == 'attribute'])
+n_internal = len([n for n in nodes if n['type'] == 'aggregation'])
+
+st.divider()
+c1, c2, c3, c4 = st.columns(4)
+c1.metric('Variables', n_leaves)
+c2.metric('Aggregation nodes', n_internal)
+c3.metric('Max depth', _sm['max_depth'])
+c4.metric('Avg branching', _sm['avg_branching_factor'])
+
+tabs = st.tabs(['Sunburst', 'Treemap', 'Node detail', 'Canonical table', 'Export', '📊 Evaluation'])
+
+with tabs[0]:
+    st.plotly_chart(plot_sunburst(nodes, max_depth=display_depth), use_container_width=True)
+    st.caption('Green = Baseline. Click a sector to drill down; click the centre to go back.')
+
+with tabs[1]:
+    st.plotly_chart(plot_treemap(nodes), use_container_width=True)
+
+with tabs[2]:
+    nm = _nmap(nodes)
+    agg_nodes = [n for n in nodes if n['type'] in ('aggregation', 'root')]
+    options   = [f'{n["name"]}  [{len(_leaf_ids(nodes, int(n["id"])))} vars]'
+                 for n in agg_nodes]
+    if options:
+        sel      = st.selectbox('Select a node', options)
+        sel_name = sel.split('  [')[0]
+        sel_node = next((n for n in agg_nodes if n['name'] == sel_name), None)
+        if sel_node:
+            lids = _leaf_ids(nodes, int(sel_node['id']))
+            leaf_ids_set = {nm[i]['metadata']['leaf_id']
+                            for i in lids if i in nm and 'metadata' in nm[i]}
+            sub = can[can['_leaf_id'].isin(leaf_ids_set)]
+            st.write(f'**{len(lids)} variables** under "{sel_node["name"]}"')
+            st.dataframe(sub[['_leaf_label', '_group_path', '_text']].reset_index(drop=True),
+                         use_container_width=True)
+
+with tabs[3]:
+    st.dataframe(can, use_container_width=True)
+
+with tabs[4]:
+    _base = safe_name(project_name)
+    col1, col2 = st.columns(2)
+    with col1:
+        st.download_button(
+            'Hierarchy JSON',
+            data=json.dumps(nodes, indent=2, ensure_ascii=False).encode('utf-8'),
+            file_name=f'{_base}_baseline_hierarchy.json',
+            mime='application/json',
+            use_container_width=True,
+        )
+    with col2:
+        st.download_button(
+            'Canonical CSV',
+            data=can.to_csv(index=False).encode('utf-8'),
+            file_name=f'{_base}_baseline_canonical.csv',
+            mime='text/csv',
+            use_container_width=True,
+        )
+
+    st.divider()
+    # ── Save directly into the project's outputs/baseline/ folder ──────────────
+    _out_dir = Path(__file__).resolve().parent / 'outputs' / 'baseline'
+    st.markdown('### Save to project folder')
+    st.caption(
+        'The download buttons above go to your browser’s Downloads folder (a browser '
+        f'restriction). This button instead writes the files into `{_out_dir}` with the '
+        'dataset name — convenient for `evaluate_all.py`.'
+    )
+    if st.button('💾 Save all to outputs/baseline/', type='primary',
+                 use_container_width=True):
+        try:
+            _out_dir.mkdir(parents=True, exist_ok=True)
+            (_out_dir / f'{_base}_baseline_hierarchy.json').write_text(
+                json.dumps(nodes, indent=2, ensure_ascii=False), encoding='utf-8')
+            can.to_csv(_out_dir / f'{_base}_baseline_canonical.csv', index=False)
+            st.success(f'Saved to `{_out_dir}`:\n\n'
+                       f'- {_base}_baseline_hierarchy.json\n'
+                       f'- {_base}_baseline_canonical.csv')
+        except Exception as _e:
+            st.error(f'Could not save: {_e}')
+
+with tabs[5]:
+    import hierarchy_eval as he
+
+    st.subheader('Hierarchy Quality Evaluation')
+    st.caption(
+        'The group column is a *construction input* (Gonçalves text object), so it '
+        'cannot serve as ground truth. The primary metrics below are **reference-free** '
+        '— they assess the hierarchy itself, with no gold standard.'
+    )
+
+    with st.spinner('Computing reference-free metrics…'):
+        tm = he.traco_metrics(nodes)
+        npmi = he.npmi_coherence(nodes, can['_text'].tolist())
+
+    # ── PRIMARY: reference-free hierarchy quality ─────────────────────────────
+    st.markdown('#### Primary — reference-free hierarchy quality')
+    p1, p2, p3 = st.columns(3)
+    p1.metric('Parent–child coherence', tm['pc_coherence'],
+              help='TraCo (Wu et al., AAAI 2024). Mean similarity of each node to its parent. '
+                   'Higher = children correctly nest under their parent theme.')
+    p2.metric('Sibling diversity', tm['sibling_diversity'],
+              help='TraCo (Wu et al., AAAI 2024). Mean distance between sibling nodes. '
+                   'Higher = siblings are distinct (LOW = redundant/repeated siblings).')
+    p3.metric('NPMI label coherence', npmi,
+              help='Lau et al., EACL 2014. Whether node-label terms genuinely co-occur in the '
+                   'data. Higher = meaningful labels, not arbitrary term salads.')
+    st.caption(f'Embedding backend: **{tm["encoder"]}**.  '
+               'Coherence & diversity ∈ [−1, 1]; NPMI ∈ ≈[−1, 1].')
+
+    # ── Structural metrics ────────────────────────────────────────────────────
+    st.markdown('#### Structural statistics')
+    sm = he.structural_stats(nodes)
+    s1, s2, s3, s4, s5 = st.columns(5)
+    s1.metric('Aggregation nodes', sm['n_aggregation_nodes'])
+    s2.metric('Max leaf depth',    sm['max_depth'])
+    s3.metric('Avg leaf depth',    sm['avg_leaf_depth'])
+    s4.metric('Avg branching',     sm['avg_branching_factor'])
+    s5.metric('Singleton nodes',   f"{sm['singleton_nodes_%']}%",
+              help='Aggregation nodes with a single child (sparse-hierarchy indicator)')
+
+    # ── SECONDARY: group preservation (caveated) ──────────────────────────────
+    st.markdown('#### Secondary — group-structure preservation *(descriptive)*')
+    st.caption(
+        '⚠️ The group column was an **input** to construction, so these are NOT accuracy '
+        'metrics — they only describe how much the discovered hierarchy still reflects the '
+        'pre-existing group column. High values are expected and not evidence of quality.'
+    )
+    gp = he.group_preservation(nodes, can)
+    g1, g2, g3 = st.columns(3)
+    g1.metric('NMI', gp['NMI']);  g2.metric('ARI', gp['ARI']);  g3.metric('Purity', gp['Purity'])

data/ai-mind-variable-descriptions(in).csv

@@ -0,0 +1,109 @@
+Task,Variant,name,description,Decimal Places
+DMS,DMS Recommended Standard,DMSCC,"DMS Mean Choices to Correct: The mean number of choices that the subject made on each trial, including the correct choice. Calculated across all trials where the subject eventually made the correct choice (simultaneous and all delays).",2
+DMS,DMS Recommended Standard,DMSL0SD,"DMS Correct Latency Standard Deviation (SD) (0 second delay): The standard deviation of response latencies for trials containing a zero second delay between the presentation of target and response stimuli, where subjects selected the correct box on their first attempt. Calculated across all assessed trials containing a zero second delay.",4
+DMS,DMS Recommended Standard,DMSL12SD,"DMS Correct Latency Standard Deviation (SD) (12 second delay): The standard deviation of response latencies for trials containing a twelve second delay between the presentation of target and response stimuli, where subjects selected the correct box on their first attempt. Calculated across all assessed trials containing a twelve second delay.",4
+DMS,DMS Recommended Standard,DMSL4SD,"DMS Correct Latency Standard Deviation (SD) (4 second delay): The standard deviation of response latencies for trials containing a four second delay between the presentation of target and response stimuli, where subjects selected the correct box on their first attempt. Calculated across all assessed trials containing a four second delay.",4
+DMS,DMS Recommended Standard,DMSLADSD,"DMS Correct Latency Standard Deviation (SD) (all delays): The standard deviation of response latencies for trials containing a delay between the presentation of target stimulus and response stimuli, where subjects selected the correct box on their first attempt. Calculated across all assessed trials containing a delay.",4
+DMS,DMS Recommended Standard,DMSLSD,DMS Correct Latency Standard Deviation (SD): The standard deviation of response latencies for trials where subjects selected the correct box on their first attempt. Calculated across all correct assessed trials (simultaneous and all delays).,4
+DMS,DMS Recommended Standard,DMSLSSD,"DMS Correct Latency Standard Deviation (SD) (simultaneous): The standard deviation of response latencies for trials containing a simultaneous presentation of target and response stimuli, where subjects selected the correct box on their first attempt. Calculated across all assessed trials containing simultaneous presentations.",4
+DMS,DMS Recommended Standard,DMSMDL,DMS Median Correct Latency: The median latency between the presentation of the response stimuli options and the subject selecting the correct box on their first attempt. Calculated across all correct assessed trials (simultaneous and all delays).,4
+DMS,DMS Recommended Standard,DMSMDL0,DMS Median Correct Latency (0 seconds delay): The median latency between the presentation of the response stimuli options and the subject selecting the correct box on their first attempt for trials containing a zero second delay. Calculated across all assessed trials containing a zero second delay.,4
+DMS,DMS Recommended Standard,DMSMDL12,DMS Median Correct Latency (12 seconds delay): The median latency between the presentation of the response stimuli options and the subject selecting the correct box on their first attempt for trials containing a twelve second delay. Calculated across all assessed trials containing a twelve second delay.,4
+DMS,DMS Recommended Standard,DMSMDL4,DMS Median Correct Latency (4 seconds delay): The median latency between the presentation of the response stimuli options and the subject selecting the correct box on their first attempt for trials containing a four second delay. Calculated across all assessed trials containing a four second delay.,4
+DMS,DMS Recommended Standard,DMSMDLAD,DMS Median Correct Latency (all delays): The median latency between the presentation of the response stimuli options and the subject selecting the correct box on their first attempt for trials containing a delay between target and response stimuli presentation. Calculated across all assessed trials containing a delay.,4
+DMS,DMS Recommended Standard,DMSMDLS,DMS Median Correct Latency (simultaneous): The median latency between the presentation of the response stimuli options and the subject selecting the correct box on their first attempt for trials containing a simultaneous presentation of target and response stimuli. Calculated across all assessed trials containing simultaneous presentation.,4
+DMS,DMS Recommended Standard,DMSML,DMS Mean Correct Latency: The mean latency between the presentation of the response stimuli options and the subject selecting the correct box on their first attempt. Calculated across all correct assessed trials (simultaneous and all delays).,4
+DMS,DMS Recommended Standard,DMSML0,DMS Mean Correct Latency (0 seconds delay): The mean latency between the presentation of the response stimuli options and the subject selecting the correct box on their first attempt for trials containing a zero second delay. Calculated across all assessed trials containing a zero second delay.,4
+DMS,DMS Recommended Standard,DMSML12,DMS Mean Correct Latency (12 seconds delay): The mean latency between the presentation of the response stimuli options and the subject selecting the correct box on their first attempt for trials containing a twelve second delay. Calculated across all assessed trials containing a twelve second delay.,4
+DMS,DMS Recommended Standard,DMSML4,DMS Mean Correct Latency (4 seconds delay): The mean latency between the presentation of the response stimuli options and the subject selecting the correct box on their first attempt for trials containing a four second delay. Calculated across all assessed trials containing a four second delay.,4
+DMS,DMS Recommended Standard,DMSMLAD,DMS Mean Correct Latency (all delays): The mean latency between the presentation of the response stimuli options and the subject selecting the correct box on their first attempt for trials containing a delay between target and response stimuli presentation. Calculated across all assessed trials containing a delay.,4
+DMS,DMS Recommended Standard,DMSMLS,DMS Mean Correct Latency (simultaneous): The mean latency between the presentation of the response stimuli options and the subject selecting the correct box on their first attempt for trials containing a simultaneous presentation of target and response stimuli. Calculated across all assessed trials containing simultaneous presentation.,4
+DMS,DMS Recommended Standard,DMSPC,DMS Percent Correct: The percentage of assessment trials during which the subject chose the correct box on their first box choice. Calculated across all assessed trials (simultaneous presentation and all delays).,0
+DMS,DMS Recommended Standard,DMSPC0,KEY: DMS Percent Correct (0 seconds delay): The percentage of assessment trials containing a zero second delay during which the subject chose the correct box on their first box choice. Calculated across all assessed trials containing a zero second delay.,0
+DMS,DMS Recommended Standard,DMSPC12,KEY: DMS Percent Correct (12 second delay): The percentage of assessment trials containing a twelve second delay during which the subject chose the correct box on their first box choice. Calculated across all assessed trials containing a twelve second delay.,0
+DMS,DMS Recommended Standard,DMSPC4,KEY: DMS Percent Correct (4 second delay): The percentage of assessment trials containing a four second delay during which the subject chose the correct box on their first box choice. Calculated across all assessed trials containing a four second delay.,0
+DMS,DMS Recommended Standard,DMSPCAD,KEY: DMS Percent Correct (all delays): The percentage of assessment trials containing a delay during which the subject chose the correct box on their first box choice. Calculated across all assessed trials containing a delay.,0
+DMS,DMS Recommended Standard,DMSPCS,KEY: DMS Percent Correct (simultaneous): The percentage of assessment trials where the target and response stimuli were presented simultaneously during which the subject chose the correct box on their first box choice. Calculated across all assessed trials containing the simultaneous presentation of stimuli.,0
+DMS,DMS Recommended Standard,DMSPEGC,DMS Probability of Error Given Correct: This measure reports the probability of an error being made when the previous trial was responded to correctly by the subject. Calculated across all assessed trials (simultaneous and all delays).,4
+DMS,DMS Recommended Standard,DMSPEGE,KEY: DMS Probability of Error Given Error: This measure reports the probability of an error occurring when the previous trial was responded to incorrectly. Calculated across all assessed trials (simultaneous and all delays).,4
+DMS,DMS Recommended Standard,DMSTC,DMS Total Correct: The total number of times a subject chose the correct answer on their first box choice. Calculated across all assessed trials (simultaneous presentation and all delays).,0
+DMS,DMS Recommended Standard,DMSTC0,DMS Total Correct (0 second delay): The total number of times a subject chose the correct answer on their first box choice for trials where the response stimuli appeared on screen after a 0 second delay after the target stimulus was shown. Calculated across all assessed trials which contained a delay of zero seconds.,0
+DMS,DMS Recommended Standard,DMSTC12,DMS Total Correct (12 second delay): The total number of times a subject chose the correct answer on their first box choice for trials where the response stimuli appeared on screen after a 12 second delay after the target stimulus was shown. Calculated across all assessed trials which contained a delay of twelve seconds.,0
+DMS,DMS Recommended Standard,DMSTC4,DMS Total Correct (4 second delay): The total number of times a subject chose the correct answer on their first box choice for trials where the response stimuli appeared on screen after a 4 second delay after the target stimulus was shown. Calculated across all assessed trials which contained a delay of four seconds.,0
+DMS,DMS Recommended Standard,DMSTCAD,DMS Total Correct (all delays): The total number of times a subject chose the correct answer on their first box choice for all trials where the response stimuli were presented after a delay. Calculated across all assessed trials containing a delay.,0
+DMS,DMS Recommended Standard,DMSTCS,DMS Total Correct (simultaneous): The total number of times a subject chose the correct answer on their first box choice for trials where the target stimulus and response stimuli appeared on screen simultaneously. Calculated across all assessed trials that included a simultaneous presentation (no delay) of target and response stimuli.,0
+DMS,DMS Recommended Standard,DMSTE,"DMS Total Errors: The total number of times a subject failed to choose the correct box on their first selection, thus making an error. Calculated across all assessed trials (simultaneous and all delays) regardless of which incorrect box (out of the 3 possible incorrect boxes) was chosen.",0
+DMS,DMS Recommended Standard,DMSTEAD,DMS Total Errors (all delays): The total number of times a subject failed to choose the correct box on their first selection for any trial containing a delay between the presentation of the target stimulus and response stimuli. Calculated across all assessed trials containing a delay component.,0
+DMS,DMS Recommended Standard,DMSTEC,"DMS Error (incorrect colour): The number of times that the subject failed to select the correct box on their first selection, and instead chose the distractor stimulus that contained the same pattern/ physical attributes, but different colours. Calculated across all assessed trials (simultaneous and all delays).",0
+DMS,DMS Recommended Standard,DMSTECAD,"DMS Error (all delays, incorrect colour): The number of times that the subject failed to select the correct box on their first selection, and instead chose the distractor stimulus that contained the same colour elements, but different physical attributes. Calculated across all assessed trials which contained a delay component.",0
+DMS,DMS Recommended Standard,DMSTED,"DMS Error (distractor): The number of times that the subject failed to select the correct box on their first selection, and instead chose the distractor stimulus that contained no common elements to the original target stimulus. Calculated across all assessed trials (simultaneous and all delays).",0
+DMS,DMS Recommended Standard,DMSTEDAD,"DMS Error (all delays, distractor): The number of times that the subject failed to select the correct box on their first selection, and instead chose the distractor stimulus that contained no common elements to the original target stimulus. Calculated across all assessed trials which contained a delay component.",0
+DMS,DMS Recommended Standard,DMSTEP,"DMS Error (incorrect pattern): The number of times that the subject failed to select the correct box on their first selection, and instead chose the distractor stimulus that contained the same colour elements, but different pattern/ physical attributes. Calculated across all assessed trials (simultaneous and all delays).",0
+DMS,DMS Recommended Standard,DMSTEPAD,"DMS Error (all delays, incorrect pattern): The number of times that the subject failed to select the correct box on their first selection, and instead chose the distractor stimulus that contained the same pattern/ physical attributes, but different colour elements. Calculated across all assessed trials which contained a delay component.",0
+MOT,MOT Tone 2.0,MOTML,The mean latency from the display of a stimulus to a correct response to that stimulus during assessment trials.,1
+MOT,MOT Tone 2.0,MOTSDL,"This is the standard deviation of the latency, calculated from the display of a stimulus to a correct response to that stimulus during assessment trials.",2
+MOT,MOT Tone 2.0,MOTTC,The total number of assessment trials on which the subject made a correct response.,0
+MOT,MOT Tone 2.0,MOTTE,The total number of assessment trials on which the subject failed to make a correct response.,0
+PAL,PAL Recommended Standard Extended,PALFAMS28,"KEY: PAL First Attempt Memory Score: The number of times a subject chose the correct box on their first attempt when recalling the pattern locations. Calculated across assessed trials, omitting 12 box level to provide a direct comparison to Recommended Standard..",0
+PAL,PAL Recommended Standard Extended,PALMETS28,PAL Mean Errors to Success: The mean number of attempts made by a subject needed for them to successfully complete the stage.  Does not include 12 box level to provide a direct comparison to Recommended Standard.,0
+PAL,PAL Recommended Standard Extended,PALNPR28,PAL Number of Patterns Reached: The number of patterns presented to the subject on the last problem they reached.,0
+PAL,PAL Recommended Standard Extended,PALTA12,PAL Total Attempts 12 Patterns: The total number of attempts made (but not necessarily completed) by the subject during assessment problems containing a total of 12 shapes to recall.,0
+PAL,PAL Recommended Standard Extended,PALTA2,PAL Total Attempts 2 Patterns: The total number of attempts made (but not necessarily completed) by the subject during assessment problems containing a total of 2 shapes to recall.,0
+PAL,PAL Recommended Standard Extended,PALTA28,PAL Total Attempts: The total number of attempts made (but not necessarily completed) by the subject during assessment problems.  Does not include 12 box level to provide a direct comparison to Recommended Standard.,0
+PAL,PAL Recommended Standard Extended,PALTA4,PAL Total Attempts 4 patterns: The total number of attempts made (but not necessarily completed) by the subject during assessment problems containing a total of 4 shapes to recall.,0
+PAL,PAL Recommended Standard Extended,PALTA6,PAL Total Attempts 6 Patterns: The total number of attempts made (but not necessarily completed) by the subject during assessment problems containing a total of 6 shapes to recall.,0
+PAL,PAL Recommended Standard Extended,PALTA8,PAL Total Attempts 8 Patterns: The total number of attempts made (but not necessarily completed) by the subject during assessment problems containing a total of 8 shapes to recall.,0
+PAL,PAL Recommended Standard Extended,PALTE12,PAL Total Errors 12 Patterns: The total number of times a subject selected an incorrect box when attempting to recall a pattern location on trials containing a total of 12 patterns. Calculated across all 12-pattern assessed trials.,0
+PAL,PAL Recommended Standard Extended,PALTE2,PAL Total Errors 2 Patterns: The total number of times a subject selected an incorrect box when attempting to recall a pattern location on trials containing a total of 2 patterns. Calculated across all 2-pattern assessed trials.,0
+PAL,PAL Recommended Standard Extended,PALTE28,PAL Total Errors: The total number of times a subject selected an incorrect box when attempting to recall a pattern location. Calculated across all assessed trials.  Does not include 12 box level to provide a direct comparison to Recommended Standard.,0
+PAL,PAL Recommended Standard Extended,PALTE4,PAL Total Errors 4 Patterns: The total number of times a subject selected an incorrect box when attempting to recall a pattern location on trials containing a total of 4 patterns. Calculated across all 4-pattern assessed trials.,0
+PAL,PAL Recommended Standard Extended,PALTE6,PAL Total Errors 6 Patterns: The total number of times a subject selected an incorrect box when attempting to recall a pattern location on trials containing a total of 6 patterns. Calculated across all 6-pattern assessed trials.,0
+PAL,PAL Recommended Standard Extended,PALTE8,PAL Total Errors 8 Patterns: The total number of times a subject selected an incorrect box when attempting to recall a pattern location on trials containing a total of 8 patterns. Calculated across all 8-pattern assessed trials.,0
+PAL,PAL Recommended Standard Extended,PALTEA12,"PAL Total Errors 12 Shapes (Adjusted): The number of times the subject chose the incorrect box for a stimulus on assessment problems, where the number of shapes was equal to 12 (PALTE12), plus an adjustment for the estimated number of errors they would have made on any other 12 pattern problems, attempts and recalls they did not reach.",0
+PAL,PAL Recommended Standard Extended,PALTEA2,"PAL Total Errors 2 Shapes (Adjusted): The number of times the subject chose the incorrect box for a stimulus on assessment problems, where the number of shapes required to remember was equal to 2 (PALTE2), plus an adjustment for the estimated number of errors they would have made on any other 2 pattern problems, attempts and recalls they did not reach.",0
+PAL,PAL Recommended Standard Extended,PALTEA28,"KEY: PAL Total Errors (Adjusted): The number of times the subject chose the incorrect box for a stimulus on assessment problems (PALTE), plus an adjustment for the estimated number of errors they would have made on any problems, attempts and recalls they did not reach. This measure allows you to compare performance on errors made across all subjects regardless of those who terminated early versus those completing the final stage of the task.  In this task variant PALTEA does not include 12 box level to provide a direct comparison to Recommended Standard.",0
+PAL,PAL Recommended Standard Extended,PALTEA4,"PAL Total Errors 4 Shapes (Adjusted): The number of times the subject chose the incorrect box for a stimulus on assessment problems, where the number of shapes was equal to 4 (PALTE4), plus an adjustment for the estimated number of errors they would have made on any other 4 pattern problems, attempts and recalls they did not reach.",0
+PAL,PAL Recommended Standard Extended,PALTEA6,"PAL Total Errors 6 Shapes (Adjusted): The number of times the subject chose the incorrect box for a stimulus on assessment problems, where the number of shapes was equal to 6 (PALTE6), plus an adjustment for the estimated number of errors they would have made on any other 6 pattern problems, attempts and recalls they did not reach.",0
+PAL,PAL Recommended Standard Extended,PALTEA8,"PAL Total Errors 8 Shapes (Adjusted): The number of times the subject chose the incorrect box for a stimulus on assessment problems, where the number of shapes was equal to 8 (PALTE8), plus an adjustment for the estimated number of errors they would have made on any other 8 pattern problems, attempts and recalls they did not reach.",0
+PRM,PRM Recommended Standard 18 Extended,PRMCLSDD,"PRM Correct Latency (SD) Delayed: The standard deviation for the latency of a subject's response to correctly choose the appropriate pattern in the delayed forced-choice condition, measured in milliseconds.",2
+PRM,PRM Recommended Standard 18 Extended,PRMCLSDI,"PRM Correct Latency (SD) Immediate: The standard deviation for the latency of a subject's response to correctly select the appropriate pattern in the immediate forced-choice condition, measured in milliseconds.",2
+PRM,PRM Recommended Standard 18 Extended,PRMMCLD,"PRM Mean Correct Latency Delayed: The mean latency for a subject to correctly select the appropriate pattern during the delayed forced-choice condition, measured in milliseconds.",2
+PRM,PRM Recommended Standard 18 Extended,PRMMCLI,"PRM Mean Correct Latency Immediate: The mean latency for a subject to correctly select the appropriate pattern during the immediate forced-choice condition, measured in milliseconds.",2
+PRM,PRM Recommended Standard 18 Extended,PRMMDCLD,"PRM Median Correct Latency Delayed: The median latency for a subject to correctly select the appropriate pattern during the delayed forced-choice condition, measured in milliseconds.",2
+PRM,PRM Recommended Standard 18 Extended,PRMMDCLI,"PRM Median Correct Latency Immediate: The median latency for a subject to correctly select the appropriate pattern during the immediate forced-choice condition, measured in milliseconds.",2
+PRM,PRM Recommended Standard 18 Extended,PRMPCD,"KEY: PRM Percent Correct Delayed: The number of correct patterns selected by the subject in the delayed forced-choice condition, expressed as a percentage.",2
+PRM,PRM Recommended Standard 18 Extended,PRMPCI,"KEY: PRM Percent Correct Immediate: The number of correct patterns selected by the subject in the immediate forced-choice condition, expressed as a percentage.",2
+PRM,PRM Recommended Standard 18 Extended,PRMTSDSP,PRM Time Since Delayed Stimuli Presentation: The length of time between the end of the stimuli presentation for the delayed phase and the start of the delayed forced-choice condition.,2
+RVP,RVP 3 Targets,RVPA,"KEY: RVP A?: A? (A prime) is the signal detection measure of a subject's sensitivity to the target sequence (string of three numbers), regardless of response tendency (the expected range is 0.00 to 1.00; bad to good). In essence, this metric is a measure of how good the subject is at detecting target sequences.",4
+RVP,RVP 3 Targets,RVPLSD,RVP Response Latency (SD): The standard deviation of response latency on trials where the subject responded correctly. Calculated across all assessed trials.,4
+RVP,RVP 3 Targets,RVPMDL,KEY: RVP Median Response Latency: The median response latency on trials where the subject responded correctly. Calculated across all assessed trials.,4
+RVP,RVP 3 Targets,RVPML,RVP Mean Response Latency: The mean response latency on trials where the subject responded correctly. Calculated across all assessed trials.,4
+RVP,RVP 3 Targets,RVPPFA,KEY: RVP Probability of False Alarm: The number of sequence presentations that were false alarms divided by the number of sequence presentations that were false alarms plus the number of sequence presentations that were correct rejections: (False Alarms ÷ (False Alarms + Correct Rejections)),4
+RVP,RVP 3 Targets,RVPPH,"RVP Probability of Hit: The number of target sequences during assessment blocks that were correctly responded to within the time allowed, divided by the number of target sequences during assessment blocks (Correct hits ÷ total number of sequences)",4
+RVP,RVP 3 Targets,RVPTFA,RVP Total False Alarms: The total number of stimulus presentations during assessment blocks that were false alarms.,0
+RVP,RVP 3 Targets,RVPTH,RVP Total Hits: The total number of target sequences that were correctly responded to (Correct Hits) within the allowed time during assessment sequence blocks.,0
+RVP,RVP 3 Targets,RVPTM,RVP Total Misses: The total number of target sequences that were not responded to within the allowed time during assessment sequence blocks.,0
+SWM,SWM Recommended Standard 2.0 Extended,SWMBE12,KEY: SWM Between errors 12 boxes: The number of times the subject revisits a box in which a token has previously been found. Calculated across all trials with 12 tokens only.,0
+SWM,SWM Recommended Standard 2.0 Extended,SWMBE4,KEY: SWM Between errors 4 boxes: The number of times a subject revisits a box in which a token has previously been found. Calculated across all trials with 4 tokens only.,0
+SWM,SWM Recommended Standard 2.0 Extended,SWMBE468,"KEY: SWM Between Errors: The number of times the subject incorrectly revisits a box in which a token has previously been found. Calculated across all assessed four, six and eight token trials.",0
+SWM,SWM Recommended Standard 2.0 Extended,SWMBE6,KEY: SWM Between errors 6 boxes: The number of times the subject revisits a box in which a token has previously been found. Calculated across all trials with 6 tokens only.,0
+SWM,SWM Recommended Standard 2.0 Extended,SWMBE8,KEY: SWM Between errors 8 boxes: The number of times the subject revisits a box in which a token has previously been found. Calculated across all trials with 8 tokens only.,0
+SWM,SWM Recommended Standard 2.0 Extended,SWMDE12,SWM Double errors 12 boxes: The number of times a subject commits an error that is both a within error and a between error. Calculated across all trials with 12 tokens only.,0
+SWM,SWM Recommended Standard 2.0 Extended,SWMDE4,SWM Double errors 4 boxes: The number of times a subject commits an error that is both a within error and a between error. Calculated across all trials with 4 tokens only.,0
+SWM,SWM Recommended Standard 2.0 Extended,SWMDE468,"SWM Double Errors: The number of times a subject commits an error that is both a within error and a between error. Calculated across all assessed four, six and eight token trials.",0
+SWM,SWM Recommended Standard 2.0 Extended,SWMDE6,SWM Double errors 6 boxes: The number of times a subject commits an error that is both a within error and a between error. Calculated across all trials with 6 tokens only.,0
+SWM,SWM Recommended Standard 2.0 Extended,SWMDE8,SWM Double errors 8 boxes: The number of times a subject commits an error that is both a within error and a between error. Calculated across all trials with 8 tokens only.,0
+SWM,SWM Recommended Standard 2.0 Extended,SWMPR,"SWM Problem Reached: This measure reports the problem number that the subject reached, but did not necessarily complete.",0
+SWM,SWM Recommended Standard 2.0 Extended,SWMS,"KEY: SWM Strategy (6-8 boxes): The number of times a subject begins a new search pattern from the same box they started with previously. If they always begin a search from the same starting point we infer that the subject is employing a planned strategy for finding the tokens. Therefore a low score indicates high strategy use (1 = they always begin the search from the same box), a high score indicates that they are beginning their searches from many different boxes. Calculated across assessed trials with 6 tokens or 8 tokens.",0
+SWM,SWM Recommended Standard 2.0 Extended,SWMS6,"SWM Strategy (6 box only): This measure computes the strategy score for the 6 box stage of the task only. The strategy score is calculated based on the number of times a subject begins a new search pattern from the same box they started with previously. If they always begin a search from the same starting point we infer that the subject is employing a planned strategy for finding the tokens. Therefore a low score indicates high strategy use (1 = they always begin the search from the same box), a high score indicates that they are beginning their searches from many different boxes.",0
+SWM,SWM Recommended Standard 2.0 Extended,SWMSX,"SWM Strategy (6-12 boxes): The number of times a subject begins a new search pattern from the same box they started with previously. If they always begin a search from the same starting point we infer that the subject is employing a planned strategy for finding the tokens. Therefore a low score indicates high strategy use (1 = they always begin the search from the same box), a high score indicates that they are beginning their searches from many different boxes. Calculated across assessed trials with 6 tokens or more.",0
+SWM,SWM Recommended Standard 2.0 Extended,SWMTE12,"SWM Total errors 12 boxes: The number of times a box is selected that is certain not to contain a token and therefore should not have been visited by the subject, i.e. between errors + within errors - double errors. Calculated across all trials with 12 tokens only.",0
+SWM,SWM Recommended Standard 2.0 Extended,SWMTE4,"SWM Total errors 4 boxes: The number of times a box is selected that is certain not to contain a token and therefore should not have been visited by the subject, i.e. between errors + within errors - double errors. Calculated across all trials with 4 tokens only.",0
+SWM,SWM Recommended Standard 2.0 Extended,SWMTE468,"SWM Total Errors: The total number of times a box is selected that is certain not to contain a token and therefore should not have been visited by the subject, i.e. between errors + within errors - double errors. Calculated across all assessed four, six and eight token trials.",0
+SWM,SWM Recommended Standard 2.0 Extended,SWMTE6,"SWM Total errors 6 boxes: The number of times a box is selected that is certain not to contain a token and therefore should not have been visited by the subject, i.e. between errors + within errors - double errors. Calculated across all trials with 6 tokens only.",0
+SWM,SWM Recommended Standard 2.0 Extended,SWMTE8,"SWM Total errors 8 boxes: The number of times a box is selected that is certain not to contain a token and therefore should not have been visited by the subject, i.e. between errors + within errors - double errors. Calculated across all trials with 8 tokens only.",0
+SWM,SWM Recommended Standard 2.0 Extended,SWMWE12,SWM Within errors 12 boxes: The number of times a subject revisits a box already found to be empty during the same search. Calculated across all trials with 12 tokens only.,0
+SWM,SWM Recommended Standard 2.0 Extended,SWMWE4,SWM Within errors 4 boxes: The number of times a subject revisits a box already found to be empty during the same search. Calculated across all trials with 4 tokens only.,0
+SWM,SWM Recommended Standard 2.0 Extended,SWMWE468,"SWM Within Errors: The number of times a subject revisits a box already shown to be empty during the same search. Calculated across all assessed four, six and eight token trials.",0
+SWM,SWM Recommended Standard 2.0 Extended,SWMWE6,SWM Within errors 6 boxes: The number of times a subject revisits a box already found to be empty during the same search. Calculated across all trials with 6 tokens only.,0
+SWM,SWM Recommended Standard 2.0 Extended,SWMWE8,SWM Within errors 8 boxes: The number of times a subject revisits a box already found to be empty during the same search. Calculated across all trials with 8 tokens only.,0

data/dictionary_harmonized_categories.csv

@@ -0,0 +1,571 @@
+"","variable_codename_use","variable_description_use","harmonized_categories","harmonized_categories_description","in_dataset","corrected_codename_2day_separate"
+"1","DMDBORN4","In what country {were you/was SP} born?","1","Born in 50 US states or Washington, DC","Demographics",NA
+"2","DMDBORN4","In what country {were you/was SP} born?","2","Others","Demographics",NA
+"3","DMDBORN4","In what country {were you/was SP} born?","77","Refused","Demographics",NA
+"4","DMDBORN4","In what country {were you/was SP} born?","99","Don't Know","Demographics",NA
+"5","DMDEDUC2","Education level - Adults 20+","1","Less Than 9th Grade","Demographics",NA
+"6","DMDEDUC2","Education level - Adults 20+","2","9-11th Grade (Includes 12th grade with no diploma)","Demographics",NA
+"7","DMDEDUC2","Education level - Adults 20+","3","High School Grad/GED or Equivalent","Demographics",NA
+"8","DMDEDUC2","Education level - Adults 20+","4","Some College or AA degree","Demographics",NA
+"9","DMDEDUC2","Education level - Adults 20+","5","College Graduate or above","Demographics",NA
+"10","DMDEDUC2","Education level - Adults 20+","7","Refused","Demographics",NA
+"11","DMDEDUC2","Education level - Adults 20+","9","Don't know","Demographics",NA
+"12","DMDEDUC3","Education level - Children/Youth 6-19","0","Never Attended / Kindergarten Only","Demographics",NA
+"13","DMDEDUC3","Education level - Children/Youth 6-19","1","1st Grade","Demographics",NA
+"14","DMDEDUC3","Education level - Children/Youth 6-19","2","2nd Grade","Demographics",NA
+"15","DMDEDUC3","Education level - Children/Youth 6-19","3","3rd Grade","Demographics",NA
+"16","DMDEDUC3","Education level - Children/Youth 6-19","4","4th Grade","Demographics",NA
+"17","DMDEDUC3","Education level - Children/Youth 6-19","5","5th Grade","Demographics",NA
+"18","DMDEDUC3","Education level - Children/Youth 6-19","6","6th Grade","Demographics",NA
+"19","DMDEDUC3","Education level - Children/Youth 6-19","7","7th Grade","Demographics",NA
+"20","DMDEDUC3","Education level - Children/Youth 6-19","8","8th Grade","Demographics",NA
+"21","DMDEDUC3","Education level - Children/Youth 6-19","9","9th Grade","Demographics",NA
+"22","DMDEDUC3","Education level - Children/Youth 6-19","10","10th Grade","Demographics",NA
+"23","DMDEDUC3","Education level - Children/Youth 6-19","11","11th Grade","Demographics",NA
+"24","DMDEDUC3","Education level - Children/Youth 6-19","12","12th Grade, No Diploma","Demographics",NA
+"25","DMDEDUC3","Education level - Children/Youth 6-19","13","High School Graduate","Demographics",NA
+"26","DMDEDUC3","Education level - Children/Youth 6-19","14","GED or Equivalent","Demographics",NA
+"27","DMDEDUC3","Education level - Children/Youth 6-19","15","More than high school","Demographics",NA
+"28","DMDEDUC3","Education level - Children/Youth 6-19","55","Less Than 5th Grade","Demographics",NA
+"29","DMDEDUC3","Education level - Children/Youth 6-19","66","Less Than 9th Grade","Demographics",NA
+"30","DMDEDUC3","Education level - Children/Youth 6-19","77","Refused","Demographics",NA
+"31","DMDEDUC3","Education level - Children/Youth 6-19","99","Don't know","Demographics",NA
+"32","DMDFMSIZ","Total number of people in the Family","1","1","Demographics",NA
+"33","DMDFMSIZ","Total number of people in the Family","2","2","Demographics",NA
+"34","DMDFMSIZ","Total number of people in the Family","3","3","Demographics",NA
+"35","DMDFMSIZ","Total number of people in the Family","4","4","Demographics",NA
+"36","DMDFMSIZ","Total number of people in the Family","5","5","Demographics",NA
+"37","DMDFMSIZ","Total number of people in the Family","6","6","Demographics",NA
+"38","DMDFMSIZ","Total number of people in the Family","7","7 or more people in the Family","Demographics",NA
+"39","DMDHHSIZ","Total number of people in the Household","1","1","Demographics",NA
+"40","DMDHHSIZ","Total number of people in the Household","2","2","Demographics",NA
+"41","DMDHHSIZ","Total number of people in the Household","3","3","Demographics",NA
+"42","DMDHHSIZ","Total number of people in the Household","4","4","Demographics",NA
+"43","DMDHHSIZ","Total number of people in the Household","5","5","Demographics",NA
+"44","DMDHHSIZ","Total number of people in the Household","6","6","Demographics",NA
+"45","DMDHHSIZ","Total number of people in the Household","7","7 or more people in the Household","Demographics",NA
+"46","DMDHRAGE","Age in years of the household reference person at the time of HH screening.","1","<20 years","Demographics",NA
+"47","DMDHRAGE","Age in years of the household reference person at the time of HH screening.","2","20-39 years","Demographics",NA
+"48","DMDHRAGE","Age in years of the household reference person at the time of HH screening.","3","40-59 years","Demographics",NA
+"49","DMDHRAGE","Age in years of the household reference person at the time of HH screening.","4","60+ years","Demographics",NA
+"50","DMDYRSUS","Length of time the participant has been in the US.","1","Less than 1 year","Demographics",NA
+"51","DMDYRSUS","Length of time the participant has been in the US.","2","1 yr., less than 5 yrs.","Demographics",NA
+"52","DMDYRSUS","Length of time the participant has been in the US.","3","5 yrs., less than 10 yrs.","Demographics",NA
+"53","DMDYRSUS","Length of time the participant has been in the US.","4","10 yrs., less than 15 yrs.","Demographics",NA
+"54","DMDHRBR4","HH reference person's country of birth","1","Born in 50 US states or Washington, DC","Demographics",NA
+"55","DMDHRBR4","HH reference person's country of birth","2","Others","Demographics",NA
+"56","DMDHRBR4","HH reference person's country of birth","77","Refused","Demographics",NA
+"57","DMDHRBR4","HH reference person's country of birth","99","Don't Know","Demographics",NA
+"58","DMDHREDU","HH reference person's education level","1","Less than high school degree","Demographics",NA
+"59","DMDHREDU","HH reference person's education level","2","High school grad/GED or some college/AA degree","Demographics",NA
+"60","DMDHREDU","HH reference person's education level","3","College graduate or above","Demographics",NA
+"61","DMDHREDU","HH reference person's education level","7","Refused","Demographics",NA
+"62","DMDHREDU","HH reference person's education level","9","Don't know","Demographics",NA
+"63","DMDHREDU","HH reference person's education level","9","Don't Know","Demographics",NA
+"64","DMDHREDU","HH reference person's education level","3","High school grad/GED or some college/AA degree","Demographics",NA
+"65","DMDHRGND","Gender of the household reference person","1","Male","Demographics",NA
+"66","DMDHRGND","Gender of the household reference person","2","Female","Demographics",NA
+"67","DMDHRMAR","Marital Status of household reference person","1","Married/Living with partner","Demographics",NA
+"68","DMDHRMAR","Marital Status of household reference person","2","Widowed/Divorced/Separated","Demographics",NA
+"69","DMDHRMAR","Marital Status of household reference person","3","Never Married","Demographics",NA
+"70","DMDHRMAR","Marital Status of household reference person","77","Refused","Demographics",NA
+"71","DMDHRMAR","Marital Status of household reference person","99","Don't Know","Demographics",NA
+"72","DMDHSEDU","HH reference person's spouse's education level","1","Less than high school degree","Demographics",NA
+"73","DMDHSEDU","HH reference person's spouse's education level","2","High school grad/GED or some college/AA degree","Demographics",NA
+"74","DMDHSEDU","HH reference person's spouse's education level","3","College graduate or above","Demographics",NA
+"75","DMDHSEDU","HH reference person's spouse's education level","7","Refused","Demographics",NA
+"76","DMDHSEDU","HH reference person's spouse's education level","9","Don't Know","Demographics",NA
+"77","DMDMARTL","Marital status","1","Married","Demographics",NA
+"78","DMDMARTL","Marital status","2","Widowed","Demographics",NA
+"79","DMDMARTL","Marital status","3","Divorced","Demographics",NA
+"80","DMDMARTL","Marital status","4","Separated","Demographics",NA
+"81","DMDMARTL","Marital status","5","Never married","Demographics",NA
+"82","DMDMARTL","Marital status","6","Living with partner","Demographics",NA
+"83","DMDMARTL","Marital status","77","Refused","Demographics",NA
+"84","DMDMARTL","Marital status","99","Don't know","Demographics",NA
+"85","DMDYRSUS","Length of time the participant has been in the US.","5","15 yrs., less than 20 yrs.","Demographics",NA
+"86","DMDYRSUS","Length of time the participant has been in the US.","6","20 yrs., less than 30 yrs.","Demographics",NA
+"87","DMDYRSUS","Length of time the participant has been in the US.","7","30 yrs., less than 40 yrs.","Demographics",NA
+"88","DMDYRSUS","Length of time the participant has been in the US.","8","40 yrs., less than 50 yrs.","Demographics",NA
+"89","DMDYRSUS","Length of time the participant has been in the US.","9","50 years or more","Demographics",NA
+"90","DMDYRSUS","Length of time the participant has been in the US.","77","Refused","Demographics",NA
+"91","DMDYRSUS","Length of time the participant has been in the US.","99","Don't know","Demographics",NA
+"92","FIALANG","Language of the Family Interview Instrument","1","English","Demographics",NA
+"93","FIALANG","Language of the Family Interview Instrument","2","Spanish","Demographics",NA
+"94","FIALANG","Language of the Family Interview Instrument","3","Other","Demographics",NA
+"95","INDFMIN2","Total family income (reported as a range value in dollars)","1","$ 0 to $ 4,999","Demographics",NA
+"96","INDFMIN2","Total family income (reported as a range value in dollars)","2","$ 5,000 to $ 9,999","Demographics",NA
+"97","INDFMIN2","Total family income (reported as a range value in dollars)","3","$10,000 to $14,999","Demographics",NA
+"98","INDFMIN2","Total family income (reported as a range value in dollars)","4","$15,000 to $19,999","Demographics",NA
+"99","INDFMIN2","Total family income (reported as a range value in dollars)","5","$20,000 to $24,999","Demographics",NA
+"100","INDFMIN2","Total family income (reported as a range value in dollars)","6","$25,000 to $34,999","Demographics",NA
+"101","INDFMIN2","Total family income (reported as a range value in dollars)","7","$35,000 to $44,999","Demographics",NA
+"102","INDFMIN2","Total family income (reported as a range value in dollars)","8","$45,000 to $54,999","Demographics",NA
+"103","INDFMIN2","Total family income (reported as a range value in dollars)","16","$50,000 and over","Demographics",NA
+"104","INDFMIN2","Total family income (reported as a range value in dollars)","99","Don't know","Demographics",NA
+"105","INDFMIN2","Total family income (reported as a range value in dollars)","9","$55,000 to $64,999","Demographics",NA
+"106","INDFMIN2","Total family income (reported as a range value in dollars)","10","$65,000 to $74,999","Demographics",NA
+"107","INDFMIN2","Total family income (reported as a range value in dollars)","12","$20,000 and Over","Demographics",NA
+"108","INDFMIN2","Total family income (reported as a range value in dollars)","13","Under $20,000","Demographics",NA
+"109","INDFMIN2","Total family income (reported as a range value in dollars)","14","$75,000 to $99,999","Demographics",NA
+"110","INDFMIN2","Total family income (reported as a range value in dollars)","15","$100,000 and Over","Demographics",NA
+"111","INDFMIN2","Total family income (reported as a range value in dollars)","77","Refused","Demographics",NA
+"112","INDFMIN2","Total family income (reported as a range value in dollars)","11","$75,000 and Over","Demographics",NA
+"113","RIDRETH1","Recode of reported race and Hispanic origin information","1","Mexican American","Demographics",NA
+"114","RIDRETH1","Recode of reported race and Hispanic origin information","3","Non-Hispanic White","Demographics",NA
+"115","RIDRETH1","Recode of reported race and Hispanic origin information","4","Non-Hispanic Black","Demographics",NA
+"116","RIDRETH1","Recode of reported race and Hispanic origin information","5","Other Race - Including Multi-Racial","Demographics",NA
+"117","RIDRETH1","Recode of reported race and Hispanic origin information","2","Other Hispanic","Demographics",NA
+"118","RIDSTATR","Interview and Examination Status of the Sample Person.","1","Interviewed Only","Demographics",NA
+"119","RIDSTATR","Interview and Examination Status of the Sample Person.","2","Both Interviewed and MEC examined","Demographics",NA
+"120","MCD180B","Age when told you had congestive heart failure","16","16 years or younger","Questionnaire",NA
+"121","MCD180B","Age when told you had congestive heart failure","17-79","17-79 years old","Questionnaire",NA
+"122","MCD180B","Age when told you had congestive heart failure","17-84","17-84 years old","Questionnaire",NA
+"123","MCD180B","Age when told you had congestive heart failure","17-89","17-89 years old","Questionnaire",NA
+"124","MCD180B","Age when told you had congestive heart failure","18-79","18-79 years old","Questionnaire",NA
+"125","MCD180B","Age when told you had congestive heart failure","80","80 years or older","Questionnaire",NA
+"126","MCD180B","Age when told you had congestive heart failure","85","85 years or older","Questionnaire",NA
+"127","MCD180B","Age when told you had congestive heart failure","90","90 + years","Questionnaire",NA
+"128","MCD180B","Age when told you had congestive heart failure","99999","Don't know","Questionnaire",NA
+"129","MCD180B","Age when told you had congestive heart failure","77777","Refused","Questionnaire",NA
+"130","MCD180C","Age when told had coronary heart disease","16","16 years or younger","Questionnaire",NA
+"131","MCD180C","Age when told had coronary heart disease","17-79","17-79 years old","Questionnaire",NA
+"132","MCD180C","Age when told had coronary heart disease","17-84","17-84 years old","Questionnaire",NA
+"133","MCD180C","Age when told had coronary heart disease","20-79","20-79 years old","Questionnaire",NA
+"134","MCD180C","Age when told had coronary heart disease","80","80 years or older","Questionnaire",NA
+"135","MCD180C","Age when told had coronary heart disease","85","85 years or older","Questionnaire",NA
+"136","MCD180C","Age when told had coronary heart disease","99999","Don't know","Questionnaire",NA
+"137","MCD180C","Age when told had coronary heart disease","77777","Refused","Questionnaire",NA
+"138","MCD180D","Age when told you had angina pectoris","16","16 years or younger","Questionnaire",NA
+"139","MCD180D","Age when told you had angina pectoris","17-84","17-84 years old","Questionnaire",NA
+"140","MCD180D","Age when told you had angina pectoris","85","85 years or older","Questionnaire",NA
+"141","MCD180D","Age when told you had angina pectoris","99999","Don't know","Questionnaire",NA
+"142","MCD180D","Age when told you had angina pectoris","77777","Refused","Questionnaire",NA
+"143","MCD180D","Age when told you had angina pectoris","17-79","17-79 years old","Questionnaire",NA
+"144","MCD180D","Age when told you had angina pectoris","20-79","20-79 years old","Questionnaire",NA
+"145","MCD180D","Age when told you had angina pectoris","80","80 years or older","Questionnaire",NA
+"146","MCD180E","Age when told you had heart attack","16","16 years or younger","Questionnaire",NA
+"147","MCD180E","Age when told you had heart attack","17-79","17-79 years old","Questionnaire",NA
+"148","MCD180E","Age when told you had heart attack","17-84","17-84 years old","Questionnaire",NA
+"149","MCD180E","Age when told you had heart attack","17-88","17-88 years old","Questionnaire",NA
+"150","MCD180E","Age when told you had heart attack","19-79","19-79 years old","Questionnaire",NA
+"151","MCD180E","Age when told you had heart attack","80","80 years or older","Questionnaire",NA
+"152","MCD180E","Age when told you had heart attack","85","85 years or older","Questionnaire",NA
+"153","MCD180E","Age when told you had heart attack","90","90 + years","Questionnaire",NA
+"154","MCD180E","Age when told you had heart attack","99999","Don't know","Questionnaire",NA
+"155","MCD180E","Age when told you had heart attack","77777","Refused","Questionnaire",NA
+"156","MCD180F","Age when told you had a stroke","16","16 years or younger","Questionnaire",NA
+"157","MCD180F","Age when told you had a stroke","17-79","17-79 years old","Questionnaire",NA
+"158","MCD180F","Age when told you had a stroke","17-84","17-84 years old","Questionnaire",NA
+"159","MCD180F","Age when told you had a stroke","17-89","17-89 years old","Questionnaire",NA
+"160","MCD180F","Age when told you had a stroke","80","80 years or older","Questionnaire",NA
+"161","MCD180F","Age when told you had a stroke","85","85 years or older","Questionnaire",NA
+"162","MCD180F","Age when told you had a stroke","90","90 + years","Questionnaire",NA
+"163","MCD180F","Age when told you had a stroke","99999","Don't know","Questionnaire",NA
+"164","MCD180F","Age when told you had a stroke","77777","Refused","Questionnaire",NA
+"165","MCD180G","Age when told you had emphysema","16","16 years or younger","Questionnaire",NA
+"166","MCD180G","Age when told you had emphysema","17-79","17-79 years old","Questionnaire",NA
+"167","MCD180G","Age when told you had emphysema","17-84","17-84 years old","Questionnaire",NA
+"168","MCD180G","Age when told you had emphysema","17-89","17-89 years old","Questionnaire",NA
+"169","MCD180G","Age when told you had emphysema","80","80 years or older","Questionnaire",NA
+"170","MCD180G","Age when told you had emphysema","85","85 years or older","Questionnaire",NA
+"171","MCD180G","Age when told you had emphysema","90","90 + years","Questionnaire",NA
+"172","MCD180G","Age when told you had emphysema","99999","Don't know","Questionnaire",NA
+"173","MCD180G","Age when told you had emphysema","77777","Refused","Questionnaire",NA
+"174","MCD180K","Age when told you had chronic bronchitis","16","16 years or younger","Questionnaire",NA
+"175","MCD180K","Age when told you had chronic bronchitis","17-79","17-79 years old","Questionnaire",NA
+"176","MCD180K","Age when told you had chronic bronchitis","17-83","17-83 years old","Questionnaire",NA
+"177","MCD180K","Age when told you had chronic bronchitis","17-89","17-89 years old","Questionnaire",NA
+"178","MCD180K","Age when told you had chronic bronchitis","80","80 years or older","Questionnaire",NA
+"179","MCD180K","Age when told you had chronic bronchitis","85","85 years or older","Questionnaire",NA
+"180","MCD180K","Age when told you had chronic bronchitis","90","90 + years","Questionnaire",NA
+"181","MCD180K","Age when told you had chronic bronchitis","99999","Don't know","Questionnaire",NA
+"182","MCD180K","Age when told you had chronic bronchitis","77777","Refused","Questionnaire",NA
+"183","MCD180L","Age when told you had a liver condition","16","16 years or younger","Questionnaire",NA
+"184","MCD180L","Age when told you had a liver condition","17-78","17-78 years old","Questionnaire",NA
+"185","MCD180L","Age when told you had a liver condition","17-79","17-79 years old","Questionnaire",NA
+"186","MCD180L","Age when told you had a liver condition","17-83","17-83 years old","Questionnaire",NA
+"187","MCD180L","Age when told you had a liver condition","80","80 years or older","Questionnaire",NA
+"188","MCD180L","Age when told you had a liver condition","85","85 years or older","Questionnaire",NA
+"189","MCD180L","Age when told you had a liver condition","99999","Don't know","Questionnaire",NA
+"190","MCD180L","Age when told you had a liver condition","77777","Refused","Questionnaire",NA
+"191","MCQ180H","Age when told you had a goiter","16","16 years or younger","Questionnaire",NA
+"192","MCQ180H","Age when told you had a goiter","17-84","17-84 years old","Questionnaire",NA
+"193","MCQ180H","Age when told you had a goiter","90","90 + years","Questionnaire",NA
+"194","MCQ180H","Age when told you had a goiter","99999","Don't know","Questionnaire",NA
+"195","MCD180M","Age when told you had thyroid problem","17-89","17-89 years old","Questionnaire",NA
+"196","MCD180M","Age when told you had thyroid problem","16","16 years or younger","Questionnaire",NA
+"197","MCD180M","Age when told you had thyroid problem","99999","Don't know","Questionnaire",NA
+"198","MCD180M","Age when told you had thyroid problem","17-84","17-84 years old","Questionnaire",NA
+"199","MCD180M","Age when told you had thyroid problem","80","80 years or older","Questionnaire",NA
+"200","MCD180M","Age when told you had thyroid problem","85","85 years or older","Questionnaire",NA
+"201","MCD180M","Age when told you had thyroid problem","77777","Refused","Questionnaire",NA
+"202","MCD180M","Age when told you had thyroid problem","17-79","17-79 years old","Questionnaire",NA
+"203","MCD180N","Age when told you had gout","16","16 years or younger","Questionnaire",NA
+"204","MCD180N","Age when told you had gout","17-79","17-79 years old","Questionnaire",NA
+"205","MCD180N","Age when told you had gout","17-86","17-86 years old","Questionnaire",NA
+"206","MCD180N","Age when told you had gout","80","80 years or older","Questionnaire",NA
+"207","MCD180N","Age when told you had gout","99999","Don't know","Questionnaire",NA
+"208","MCD180N","Age when told you had gout","77777","Refused","Questionnaire",NA
+"209","MCQ025","Age when first had asthma","1-19","1-19 years old","Questionnaire",NA
+"210","MCQ025","Age when first had asthma","1-79","1-79 years old","Questionnaire",NA
+"211","MCQ025","Age when first had asthma","1-84","1-84 years old","Questionnaire",NA
+"212","MCQ025","Age when first had asthma","1-88","1-88 years old","Questionnaire",NA
+"213","MCQ025","Age when first had asthma","80","80 years or older","Questionnaire",NA
+"214","MCQ025","Age when first had asthma","85","85 years or older","Questionnaire",NA
+"215","MCQ025","Age when first had asthma","99999","Don't know","Questionnaire",NA
+"216","MCQ025","Age when first had asthma","1","Less than 1 year","Questionnaire",NA
+"217","MCQ025","Age when first had asthma","77777","Refused","Questionnaire",NA
+"218","MCD180A","Age when told you had arthritis","16","16 years or younger","Questionnaire",NA
+"219","MCD180A","Age when told you had arthritis","17-89","17-89 years old","Questionnaire",NA
+"220","MCD180A","Age when told you had arthritis","90","90 + years","Questionnaire",NA
+"221","MCD180A","Age when told you had arthritis","99999","Don't know","Questionnaire",NA
+"222","MCD180A","Age when told you had arthritis","17-79","17-79 years old","Questionnaire",NA
+"223","MCD180A","Age when told you had arthritis","80","80 years or older","Questionnaire",NA
+"224","MCD180A","Age when told you had arthritis","77777","Refused","Questionnaire",NA
+"225","MCD180A","Age when told you had arthritis","17-84","17-84 years old","Questionnaire",NA
+"226","MCD180A","Age when told you had arthritis","85","85 years or older","Questionnaire",NA
+"227","MCQ180H","Age when told you had a goiter","17-72","17-72 years old","Questionnaire",NA
+"228","MCQ180H","Age when told you had a goiter","85","85 years or older","Questionnaire",NA
+"229","MCQ180H","Age when told you had a goiter","77777","Refused","Questionnaire",NA
+"230","MCQ195","Which type of arthritis was it","9","Don't know","Questionnaire",NA
+"231","MCQ195","Which type of arthritis was it","2","Osteoarthritis or degenerative arthritis","Questionnaire",NA
+"232","MCQ195","Which type of arthritis was it","4","Other","Questionnaire",NA
+"233","MCQ195","Which type of arthritis was it","3","Psoriatic arthritis","Questionnaire",NA
+"234","MCQ195","Which type of arthritis was it","7","Refused","Questionnaire",NA
+"235","MCQ195","Which type of arthritis was it","1","Rheumatoid arthritis","Questionnaire",NA
+"236","MCQ240A","Age when bladder cancer first diagnosed","17-78","17-78 years old","Questionnaire",NA
+"237","MCQ240A","Age when bladder cancer first diagnosed","17-83","17-83 years old","Questionnaire",NA
+"238","MCQ240A","Age when bladder cancer first diagnosed","16","16 years or younger","Questionnaire",NA
+"239","MCQ240A","Age when bladder cancer first diagnosed","80","80 years or older","Questionnaire",NA
+"240","MCQ240A","Age when bladder cancer first diagnosed","85","85 years or older","Questionnaire",NA
+"241","MCQ240A","Age when bladder cancer first diagnosed","99999","Don't know","Questionnaire",NA
+"242","MCQ240A","Age when bladder cancer first diagnosed","77777","Refused","Questionnaire",NA
+"243","MCQ240B","Age when blood cancer was first diagnosed","16","16 years or younger","Questionnaire",NA
+"244","MCQ240B","Age when blood cancer was first diagnosed","17-66","17-66 years old","Questionnaire",NA
+"245","MCQ240B","Age when blood cancer was first diagnosed","17-70","17-70 years old","Questionnaire",NA
+"246","MCQ240B","Age when blood cancer was first diagnosed","80","80 years or older","Questionnaire",NA
+"247","MCQ240B","Age when blood cancer was first diagnosed","85","85 years or older","Questionnaire",NA
+"248","MCQ240B","Age when blood cancer was first diagnosed","99999","Don't know","Questionnaire",NA
+"249","MCQ240B","Age when blood cancer was first diagnosed","77777","Refused","Questionnaire",NA
+"250","MCQ240C","Age when bone cancer was first diagnosed","17-77","17-77 years old","Questionnaire",NA
+"251","MCQ240C","Age when bone cancer was first diagnosed","17-84","17-84 years old","Questionnaire",NA
+"252","MCQ240C","Age when bone cancer was first diagnosed","16","16 years or younger","Questionnaire",NA
+"253","MCQ240C","Age when bone cancer was first diagnosed","55-76","55-76 years old","Questionnaire",NA
+"254","MCQ240C","Age when bone cancer was first diagnosed","80","80 years or older","Questionnaire",NA
+"255","MCQ240C","Age when bone cancer was first diagnosed","85","85 years or older","Questionnaire",NA
+"256","MCQ240C","Age when bone cancer was first diagnosed","99999","Don't know","Questionnaire",NA
+"257","MCQ240C","Age when bone cancer was first diagnosed","77777","Refused","Questionnaire",NA
+"258","MCQ240CC","Age when uterine cancer was first diagnosed","16","16 years or younger","Questionnaire",NA
+"259","MCQ240CC","Age when uterine cancer was first diagnosed","17-77","17-77 years old","Questionnaire",NA
+"260","MCQ240CC","Age when uterine cancer was first diagnosed","17-84","17-84 years old","Questionnaire",NA
+"261","MCQ240CC","Age when uterine cancer was first diagnosed","20-72","20-72 years old","Questionnaire",NA
+"262","MCQ240CC","Age when uterine cancer was first diagnosed","80","80 years or older","Questionnaire",NA
+"263","MCQ240CC","Age when uterine cancer was first diagnosed","85","85 years or older","Questionnaire",NA
+"264","MCQ240CC","Age when uterine cancer was first diagnosed","99999","Don't know","Questionnaire",NA
+"265","MCQ240CC","Age when uterine cancer was first diagnosed","77777","Refused","Questionnaire",NA
+"266","MCQ240D","Age when brain cancer was first diagnosed","16","16 years or younger","Questionnaire",NA
+"267","MCQ240D","Age when brain cancer was first diagnosed","17-73","17-73 years old","Questionnaire",NA
+"268","MCQ240D","Age when brain cancer was first diagnosed","17-75","17-75 years old","Questionnaire",NA
+"269","MCQ240D","Age when brain cancer was first diagnosed","80","80 years or older","Questionnaire",NA
+"270","MCQ240D","Age when brain cancer was first diagnosed","85","85 years or older","Questionnaire",NA
+"271","MCQ240D","Age when brain cancer was first diagnosed","99999","Don't know","Questionnaire",NA
+"272","MCQ240D","Age when brain cancer was first diagnosed","77777","Refused","Questionnaire",NA
+"273","MCQ240DD","Age when some other type of cancer was first diagnosed","16","16 years or younger","Questionnaire",NA
+"274","MCQ240DD","Age when some other type of cancer was first diagnosed","17-77","17-77 years old","Questionnaire",NA
+"275","MCQ240DD","Age when some other type of cancer was first diagnosed","17-78","17-78 years old","Questionnaire",NA
+"276","MCQ240DD","Age when some other type of cancer was first diagnosed","17-83","17-83 years old","Questionnaire",NA
+"277","MCQ240DD","Age when some other type of cancer was first diagnosed","80","80 years or older","Questionnaire",NA
+"278","MCQ240DD","Age when some other type of cancer was first diagnosed","85","85 years or older","Questionnaire",NA
+"279","MCQ240DD","Age when some other type of cancer was first diagnosed","99999","Don't know","Questionnaire",NA
+"280","MCQ240DD","Age when some other type of cancer was first diagnosed","77777","Refused","Questionnaire",NA
+"281","MCQ240DK","Age when cancer was first diagnosed","20-80","20-80 years old","Questionnaire",NA
+"282","MCQ240DK","Age when cancer was first diagnosed","23-47","23-47 years old","Questionnaire",NA
+"283","MCQ240DK","Age when cancer was first diagnosed","80","80 years or older","Questionnaire",NA
+"284","MCQ240DK","Age when cancer was first diagnosed","85","85 years or older","Questionnaire",NA
+"285","MCQ240DK","Age when cancer was first diagnosed","99999","Don't know","Questionnaire",NA
+"286","MCQ240DK","Age when cancer was first diagnosed","77777","Refused","Questionnaire",NA
+"287","MCQ240E","Age when breast cancer was first diagnosed","16","16 years or younger","Questionnaire",NA
+"288","MCQ240E","Age when breast cancer was first diagnosed","17-78","17-78 years old","Questionnaire",NA
+"289","MCQ240E","Age when breast cancer was first diagnosed","17-79","17-79 years old","Questionnaire",NA
+"290","MCQ240E","Age when breast cancer was first diagnosed","17-84","17-84 years old","Questionnaire",NA
+"291","MCQ240E","Age when breast cancer was first diagnosed","80","80 years or older","Questionnaire",NA
+"292","MCQ240E","Age when breast cancer was first diagnosed","85","85 years or older","Questionnaire",NA
+"293","MCQ240E","Age when breast cancer was first diagnosed","99999","Don't know","Questionnaire",NA
+"294","MCQ240E","Age when breast cancer was first diagnosed","77777","Refused","Questionnaire",NA
+"295","MCQ240F","Age when cervical cancer was first diagnosed","16","16 years or younger","Questionnaire",NA
+"296","MCQ240F","Age when cervical cancer was first diagnosed","17-65","17-65 years old","Questionnaire",NA
+"297","MCQ240F","Age when cervical cancer was first diagnosed","17-73","17-73 years old","Questionnaire",NA
+"298","MCQ240F","Age when cervical cancer was first diagnosed","80","80 years or older","Questionnaire",NA
+"299","MCQ240F","Age when cervical cancer was first diagnosed","85","85 years or older","Questionnaire",NA
+"300","MCQ240F","Age when cervical cancer was first diagnosed","99999","Don't know","Questionnaire",NA
+"301","MCQ240F","Age when cervical cancer was first diagnosed","77777","Refused","Questionnaire",NA
+"302","MCQ240G","Age when colon cancer was first diagnosed","16","16 years or younger","Questionnaire",NA
+"303","MCQ240G","Age when colon cancer was first diagnosed","17-79","17-79 years old","Questionnaire",NA
+"304","MCQ240G","Age when colon cancer was first diagnosed","17-84","17-84 years old","Questionnaire",NA
+"305","MCQ240G","Age when colon cancer was first diagnosed","21-79","21-79 years old","Questionnaire",NA
+"306","MCQ240G","Age when colon cancer was first diagnosed","80","80 years or older","Questionnaire",NA
+"307","MCQ240G","Age when colon cancer was first diagnosed","85","85 years or older","Questionnaire",NA
+"308","MCQ240G","Age when colon cancer was first diagnosed","99999","Don't know","Questionnaire",NA
+"309","MCQ240G","Age when colon cancer was first diagnosed","77777","Refused","Questionnaire",NA
+"310","MCQ240L","Age when leukemia was first diagnosed","17-70","17-70 years old","Questionnaire",NA
+"311","MCQ240L","Age when leukemia was first diagnosed","17-75","17-75 years old","Questionnaire",NA
+"312","MCQ240L","Age when leukemia was first diagnosed","28-84","28-84 years old","Questionnaire",NA
+"313","MCQ240L","Age when leukemia was first diagnosed","16","16 years or younger","Questionnaire",NA
+"314","MCQ240L","Age when leukemia was first diagnosed","80","80 years or older","Questionnaire",NA
+"315","MCQ240L","Age when leukemia was first diagnosed","85","85 years or older","Questionnaire",NA
+"316","MCQ240L","Age when leukemia was first diagnosed","99999","Don't know","Questionnaire",NA
+"317","MCQ240L","Age when leukemia was first diagnosed","77777","Refused","Questionnaire",NA
+"318","MCQ240N","Age when lung cancer was first diagnosed","16","16 years or younger","Questionnaire",NA
+"319","MCQ240N","Age when lung cancer was first diagnosed","17-76","17-76 years old","Questionnaire",NA
+"320","MCQ240N","Age when lung cancer was first diagnosed","17-84","17-84 years old","Questionnaire",NA
+"321","MCQ240N","Age when lung cancer was first diagnosed","29-79","29-79 years old","Questionnaire",NA
+"322","MCQ240N","Age when lung cancer was first diagnosed","80","80 years or older","Questionnaire",NA
+"323","MCQ240N","Age when lung cancer was first diagnosed","85","85 years or older","Questionnaire",NA
+"324","MCQ240N","Age when lung cancer was first diagnosed","99999","Don't know","Questionnaire",NA
+"325","MCQ240N","Age when lung cancer was first diagnosed","77777","Refused","Questionnaire",NA
+"326","MCQ240O","Age when lymphoma or Hodgkin's Disease was first diagnosed","16","16 years or younger","Questionnaire",NA
+"327","MCQ240O","Age when lymphoma or Hodgkin's Disease was first diagnosed","17-76","17-76 years old","Questionnaire",NA
+"328","MCQ240O","Age when lymphoma or Hodgkin's Disease was first diagnosed","17-80","17-80 years old","Questionnaire",NA
+"329","MCQ240O","Age when lymphoma or Hodgkin's Disease was first diagnosed","19-79","19-79 years old","Questionnaire",NA
+"330","MCQ240O","Age when lymphoma or Hodgkin's Disease was first diagnosed","80","80 years or older","Questionnaire",NA
+"331","MCQ240O","Age when lymphoma or Hodgkin's Disease was first diagnosed","85","85 years or older","Questionnaire",NA
+"332","MCQ240O","Age when lymphoma or Hodgkin's Disease was first diagnosed","99999","Don't know","Questionnaire",NA
+"333","MCQ240O","Age when lymphoma or Hodgkin's Disease was first diagnosed","77777","Refused","Questionnaire",NA
+"334","MCQ240P","Age when melanoma was first diagnosed","16","16 years or younger","Questionnaire",NA
+"335","MCQ240P","Age when melanoma was first diagnosed","17-78","17-78 years old","Questionnaire",NA
+"336","MCQ240P","Age when melanoma was first diagnosed","17-79","17-79 years old","Questionnaire",NA
+"337","MCQ240P","Age when melanoma was first diagnosed","17-83","17-83 years old","Questionnaire",NA
+"338","MCQ240P","Age when melanoma was first diagnosed","80","80 years or older","Questionnaire",NA
+"339","MCQ240P","Age when melanoma was first diagnosed","85","85 years or older","Questionnaire",NA
+"340","MCQ240P","Age when melanoma was first diagnosed","99999","Don't know","Questionnaire",NA
+"341","MCQ240P","Age when melanoma was first diagnosed","77777","Refused","Questionnaire",NA
+"342","MCQ240Q","Age when mouth, tongue, or lip cancer was first diagnosed","16","16 years or younger","Questionnaire",NA
+"343","MCQ240Q","Age when mouth, tongue, or lip cancer was first diagnosed","17-79","17-79 years old","Questionnaire",NA
+"344","MCQ240Q","Age when mouth, tongue, or lip cancer was first diagnosed","27-70","27-70 years old","Questionnaire",NA
+"345","MCQ240Q","Age when mouth, tongue, or lip cancer was first diagnosed","30-70","30-70 years old","Questionnaire",NA
+"346","MCQ240Q","Age when mouth, tongue, or lip cancer was first diagnosed","80","80 years or older","Questionnaire",NA
+"347","MCQ240Q","Age when mouth, tongue, or lip cancer was first diagnosed","85","85 years or older","Questionnaire",NA
+"348","MCQ240Q","Age when mouth, tongue, or lip cancer was first diagnosed","99999","Don't know","Questionnaire",NA
+"349","MCQ240Q","Age when mouth, tongue, or lip cancer was first diagnosed","77777","Refused","Questionnaire",NA
+"350","MCQ240U","Age when prostate cancer was first diagnosed","17-79","17-79 years old","Questionnaire",NA
+"351","MCQ240U","Age when prostate cancer was first diagnosed","17-84","17-84 years old","Questionnaire",NA
+"352","MCQ240U","Age when prostate cancer was first diagnosed","16","16 years or younger","Questionnaire",NA
+"353","MCQ240U","Age when prostate cancer was first diagnosed","32-79","32-79 years old","Questionnaire",NA
+"354","MCQ240U","Age when prostate cancer was first diagnosed","80","80 years or older","Questionnaire",NA
+"355","MCQ240U","Age when prostate cancer was first diagnosed","85","85 years or older","Questionnaire",NA
+"356","MCQ240U","Age when prostate cancer was first diagnosed","99999","Don't know","Questionnaire",NA
+"357","MCQ240U","Age when prostate cancer was first diagnosed","77777","Refused","Questionnaire",NA
+"358","MCQ240W","Age when non-melanoma skin cancer was first diagnosed","16","16 years or younger","Questionnaire",NA
+"359","MCQ240W","Age when non-melanoma skin cancer was first diagnosed","17-78","17-78 years old","Questionnaire",NA
+"360","MCQ240W","Age when non-melanoma skin cancer was first diagnosed","17-79","17-79 years old","Questionnaire",NA
+"361","MCQ240W","Age when non-melanoma skin cancer was first diagnosed","17-84","17-84 years old","Questionnaire",NA
+"362","MCQ240W","Age when non-melanoma skin cancer was first diagnosed","80","80 years or older","Questionnaire",NA
+"363","MCQ240W","Age when non-melanoma skin cancer was first diagnosed","85","85 years or older","Questionnaire",NA
+"364","MCQ240W","Age when non-melanoma skin cancer was first diagnosed","99999","Don't know","Questionnaire",NA
+"365","MCQ240W","Age when non-melanoma skin cancer was first diagnosed","77777","Refused","Questionnaire",NA
+"366","MCQ240X","Age when the unknown kind of skin cancer was first diagnosed","16","16 years or younger","Questionnaire",NA
+"367","MCQ240X","Age when the unknown kind of skin cancer was first diagnosed","17-79","17-79 years old","Questionnaire",NA
+"368","MCQ240X","Age when the unknown kind of skin cancer was first diagnosed","17-84","17-84 years old","Questionnaire",NA
+"369","MCQ240X","Age when the unknown kind of skin cancer was first diagnosed","18-79","18-79 years old","Questionnaire",NA
+"370","MCQ240X","Age when the unknown kind of skin cancer was first diagnosed","80","80 years or older","Questionnaire",NA
+"371","MCQ240X","Age when the unknown kind of skin cancer was first diagnosed","85","85 years or older","Questionnaire",NA
+"372","MCQ240X","Age when the unknown kind of skin cancer was first diagnosed","99999","Don't know","Questionnaire",NA
+"373","MCQ240X","Age when the unknown kind of skin cancer was first diagnosed","77777","Refused","Questionnaire",NA
+"374","MCQ240Z","Age when stomach cancer was first diagnosed","16","16 years or younger","Questionnaire",NA
+"375","MCQ240Z","Age when stomach cancer was first diagnosed","17-79","17-79 years old","Questionnaire",NA
+"376","MCQ240Z","Age when stomach cancer was first diagnosed","22-82","22-82 years old","Questionnaire",NA
+"377","MCQ240Z","Age when stomach cancer was first diagnosed","32-76","32-76 years old","Questionnaire",NA
+"378","MCQ240Z","Age when stomach cancer was first diagnosed","80","80 years or older","Questionnaire",NA
+"379","MCQ240Z","Age when stomach cancer was first diagnosed","85","85 years or older","Questionnaire",NA
+"380","MCQ240Z","Age when stomach cancer was first diagnosed","99999","Don't know","Questionnaire",NA
+"381","MCQ240Z","Age when stomach cancer was first diagnosed","77777","Refused","Questionnaire",NA
+"382","MCQ280","About how old was she when she fractured her hip (the first time)?","1-101","1-101 years old","Questionnaire",NA
+"383","MCQ280","About how old was she when she fractured her hip (the first time)?","555","50 +","Questionnaire",NA
+"384","MCQ280","About how old was she when she fractured her hip (the first time)?","9-107","9-107 years old","Questionnaire",NA
+"385","MCQ280","About how old was she when she fractured her hip (the first time)?","99999","Don't know","Questionnaire",NA
+"386","MCQ280","About how old was she when she fractured her hip (the first time)?","77777","Refused","Questionnaire",NA
+"387","MCQ280","About how old was she when she fractured her hip (the first time)?","444","Under 50","Questionnaire",NA
+"388","MCQ320","How old {were you/was SP} when {you/he} first had {your/his} PSA test?","16","16 years or younger","Questionnaire",NA
+"389","MCQ320","How old {were you/was SP} when {you/he} first had {your/his} PSA test?","17-79","17-79 years old","Questionnaire",NA
+"390","MCQ320","How old {were you/was SP} when {you/he} first had {your/his} PSA test?","17-85","17-85 years old","Questionnaire",NA
+"391","MCQ320","How old {were you/was SP} when {you/he} first had {your/his} PSA test?","80","80 years or older","Questionnaire",NA
+"392","MCQ320","How old {were you/was SP} when {you/he} first had {your/his} PSA test?","999","Don't know","Questionnaire",NA
+"393","MCQ320","How old {were you/was SP} when {you/he} first had {your/his} PSA test?","777","Refused","Questionnaire",NA
+"394","MCQ570","How old {were you/was SP} when {you /s/he} first had gallbladder surgery?","11-99","11-99 years old","Questionnaire",NA
+"395","MCQ570","How old {were you/was SP} when {you /s/he} first had gallbladder surgery?","15-79","15-79 years old","Questionnaire",NA
+"396","MCQ570","How old {were you/was SP} when {you /s/he} first had gallbladder surgery?","20-87","20-87 years old","Questionnaire",NA
+"397","MCQ570","How old {were you/was SP} when {you /s/he} first had gallbladder surgery?","80","80 years or older","Questionnaire",NA
+"398","MCQ570","How old {were you/was SP} when {you /s/he} first had gallbladder surgery?","90","90 + years","Questionnaire",NA
+"399","MCQ570","How old {were you/was SP} when {you /s/he} first had gallbladder surgery?","99999","Don't know","Questionnaire",NA
+"400","MCQ570","How old {were you/was SP} when {you /s/he} first had gallbladder surgery?","77777","Refused","Questionnaire",NA
+"401","PAD120","[Over the past 30 days], how often did {you/SP} do these tasks in or around {your/his/her} home or yard, that is tasks requiring at least moderate effort? [Such as raking leaves, mowing the lawn or heavy cleaning.]","1-95","1-95 times","Questionnaire",NA
+"402","PAD120","[Over the past 30 days], how often did {you/SP} do these tasks in or around {your/his/her} home or yard, that is tasks requiring at least moderate effort? [Such as raking leaves, mowing the lawn or heavy cleaning.]","1-99","1-99 times","Questionnaire",NA
+"403","PAD120","[Over the past 30 days], how often did {you/SP} do these tasks in or around {your/his/her} home or yard, that is tasks requiring at least moderate effort? [Such as raking leaves, mowing the lawn or heavy cleaning.]","100","100 +","Questionnaire",NA
+"404","PAD120","[Over the past 30 days], how often did {you/SP} do these tasks in or around {your/his/her} home or yard, that is tasks requiring at least moderate effort? [Such as raking leaves, mowing the lawn or heavy cleaning.]","99999","Don't know","Questionnaire",NA
+"405","PAD120","[Over the past 30 days], how often did {you/SP} do these tasks in or around {your/his/her} home or yard, that is tasks requiring at least moderate effort? [Such as raking leaves, mowing the lawn or heavy cleaning.]","77777","Refused","Questionnaire",NA
+"406","PAD460","[Over the past 30 days], how often did {you/SP} do these physical activities? [Activities designed to strengthen {your/his/her} muscles such as lifting weights, push-ups or sit-ups.]","1-91","1-91 times","Questionnaire",NA
+"407","PAD460","[Over the past 30 days], how often did {you/SP} do these physical activities? [Activities designed to strengthen {your/his/her} muscles such as lifting weights, push-ups or sit-ups.]","1-99","1-99 times","Questionnaire",NA
+"408","PAD460","[Over the past 30 days], how often did {you/SP} do these physical activities? [Activities designed to strengthen {your/his/her} muscles such as lifting weights, push-ups or sit-ups.]","100","100 +","Questionnaire",NA
+"409","PAD460","[Over the past 30 days], how often did {you/SP} do these physical activities? [Activities designed to strengthen {your/his/her} muscles such as lifting weights, push-ups or sit-ups.]","999","Don't know","Questionnaire",NA
+"410","PAD460","[Over the past 30 days], how often did {you/SP} do these physical activities? [Activities designed to strengthen {your/his/her} muscles such as lifting weights, push-ups or sit-ups.]","777","Refused","Questionnaire",NA
+"411","PAQ050Q","[Over the past 30 days], how often did {you/SP} do this? [Walk or bicycle as part of getting to and from work, or school, or to do errands.] PROBE: How many times per day, per week, or per month did {you/s/he} do these activities?","1-91","1-91 times","Questionnaire",NA
+"412","PAQ050Q","[Over the past 30 days], how often did {you/SP} do this? [Walk or bicycle as part of getting to and from work, or school, or to do errands.] PROBE: How many times per day, per week, or per month did {you/s/he} do these activities?","1-99","1-99 times","Questionnaire",NA
+"413","PAQ050Q","[Over the past 30 days], how often did {you/SP} do this? [Walk or bicycle as part of getting to and from work, or school, or to do errands.] PROBE: How many times per day, per week, or per month did {you/s/he} do these activities?","100","100 +","Questionnaire",NA
+"414","PAQ050Q","[Over the past 30 days], how often did {you/SP} do this? [Walk or bicycle as part of getting to and from work, or school, or to do errands.] PROBE: How many times per day, per week, or per month did {you/s/he} do these activities?","99999","Don't know","Questionnaire",NA
+"415","PAQ050Q","[Over the past 30 days], how often did {you/SP} do this? [Walk or bicycle as part of getting to and from work, or school, or to do errands.] PROBE: How many times per day, per week, or per month did {you/s/he} do these activities?","77777","Refused","Questionnaire",NA
+"416","BMIWAIST","Waist Circumference Comment","1","Could not obtain","Response",NA
+"417",NA,NA,"1","Breakfast","Dietary","DR1.030Z"
+"418",NA,NA,"2","Brunch","Dietary","DR1.030Z"
+"419",NA,NA,"3","Lunch","Dietary","DR1.030Z"
+"420",NA,NA,"4","Snack/beverage","Dietary","DR1.030Z"
+"421",NA,NA,"5","Dinner/supper","Dietary","DR1.030Z"
+"422",NA,NA,"6","Infant feeding","Dietary","DR1.030Z"
+"423",NA,NA,"7","Extended consumption","Dietary","DR1.030Z"
+"424",NA,NA,"8","Other","Dietary","DR1.030Z"
+"425",NA,NA,"9","Desayuno (Spanish)","Dietary","DR1.030Z"
+"426",NA,NA,"10","Almuerzo (Spanish)","Dietary","DR1.030Z"
+"427",NA,NA,"11","Comida (Spanish)","Dietary","DR1.030Z"
+"428",NA,NA,"12","Merienda (Spanish)","Dietary","DR1.030Z"
+"429",NA,NA,"13","Cena (Spanish)","Dietary","DR1.030Z"
+"430",NA,NA,"14","Entre comida/bebida (Spanish)","Dietary","DR1.030Z"
+"431",NA,NA,"15","Bocadillo (Spanish)","Dietary","DR1.030Z"
+"432",NA,NA,"16","Botana (Spanish)","Dietary","DR1.030Z"
+"433",NA,NA,"99","Don't know","Dietary","DR1.030Z"
+"434",NA,NA,"2","Lunch","Dietary","DR1.030Z"
+"435",NA,NA,"3","Dinner/supper","Dietary","DR1.030Z"
+"436",NA,NA,"5","Brunch","Dietary","DR1.030Z"
+"437",NA,NA,"6","Snack/beverage","Dietary","DR1.030Z"
+"438",NA,NA,"8","Infant feeding","Dietary","DR1.030Z"
+"439",NA,NA,"9","Extended consumption","Dietary","DR1.030Z"
+"440",NA,NA,"10","Desayano (Spanish)","Dietary","DR1.030Z"
+"441",NA,NA,"11","Almuerzo (Spanish)","Dietary","DR1.030Z"
+"442",NA,NA,"12","Comida (Spanish)","Dietary","DR1.030Z"
+"443",NA,NA,"13","Merienda (Spanish)","Dietary","DR1.030Z"
+"444",NA,NA,"14","Cena (Spanish)","Dietary","DR1.030Z"
+"445",NA,NA,"15","Entre comida/bebida/tentempie (Spanish)","Dietary","DR1.030Z"
+"446",NA,NA,"17","Bocadillo (Spanish)","Dietary","DR1.030Z"
+"447",NA,NA,"91","Other","Dietary","DR1.030Z"
+"448",NA,NA,"3","Dinner","Dietary","DR1.030Z"
+"449",NA,NA,"4","Supper","Dietary","DR1.030Z"
+"450",NA,NA,"6","Snack","Dietary","DR1.030Z"
+"451",NA,NA,"7","Drink","Dietary","DR1.030Z"
+"452",NA,NA,"10","Desayano (breakfast)","Dietary","DR1.030Z"
+"453",NA,NA,"11","Almuerzo (breakfast)","Dietary","DR1.030Z"
+"454",NA,NA,"12","Comida (lunch)","Dietary","DR1.030Z"
+"455",NA,NA,"13","Merienda (snack)","Dietary","DR1.030Z"
+"456",NA,NA,"14","Cena (dinner)","Dietary","DR1.030Z"
+"457",NA,NA,"15","Entre comida (snack)","Dietary","DR1.030Z"
+"458",NA,NA,"16","Botana (snack)","Dietary","DR1.030Z"
+"459",NA,NA,"17","Bocadillo (snack)","Dietary","DR1.030Z"
+"460",NA,NA,"18","Tentempie (snack)","Dietary","DR1.030Z"
+"461",NA,NA,"19","Bebida (drink)","Dietary","DR1.030Z"
+"462",NA,NA,"0","Non-combination food","Dietary","DR1CCMTX"
+"463",NA,NA,"90","Other mixtures","Dietary","DR1CCMTX"
+"464",NA,NA,"9","Dried beans and vegetable w/ additions","Dietary","DR1CCMTX"
+"465",NA,NA,"1","Beverage w/ additions","Dietary","DR1CCMTX"
+"466",NA,NA,"3","Bread/baked products w/ additions","Dietary","DR1CCMTX"
+"467",NA,NA,"2","Cereal w/ additions","Dietary","DR1CCMTX"
+"468",NA,NA,"14","Chips w/ additions","Dietary","DR1CCMTX"
+"469",NA,NA,"12","Meat, poultry, fish","Dietary","DR1CCMTX"
+"470",NA,NA,"7","Frozen meals","Dietary","DR1CCMTX"
+"471",NA,NA,"10","Fruit w/ additions","Dietary","DR1CCMTX"
+"472",NA,NA,"4","Salad","Dietary","DR1CCMTX"
+"473",NA,NA,"5","Sandwiches","Dietary","DR1CCMTX"
+"474",NA,NA,"6","Soup","Dietary","DR1CCMTX"
+"475",NA,NA,"11","Tortilla products","Dietary","DR1CCMTX"
+"476",NA,NA,"1","Beverage w/ adds","Dietary","DR1CCMTX"
+"477",NA,NA,"2","Cereal w/ adds","Dietary","DR1CCMTX"
+"478",NA,NA,"3","Bread/baked products w/ adds","Dietary","DR1CCMTX"
+"479",NA,NA,"8","Ice cream/frozen yogurt w/ additions","Dietary","DR1CCMTX"
+"480",NA,NA,"9","Dried beans and vegetable w/ adds","Dietary","DR1CCMTX"
+"481",NA,NA,"10","Fruit w/ adds","Dietary","DR1CCMTX"
+"482",NA,NA,"11","Tortilla Products","Dietary","DR1CCMTX"
+"483",NA,NA,"13","Lunchables","Dietary","DR1CCMTX"
+"484","DRXDRSTZ","Dietary Recall Status","1","Reliable and met the minimum criteria","Dietary","DR1DRSTZ"
+"485","DRXDRSTZ","Dietary Recall Status","2","Not reliable or not met the minimum criteria","Dietary","DR1DRSTZ"
+"486","DRXDRSTZ","Dietary Recall Status","9","Interview lost due to computer malfunction or file transfer problem","Dietary","DR1DRSTZ"
+"487","DRXDRSTZ","Dietary Recall Status","4","Reported consuming breast-milk","Dietary","DR1DRSTZ"
+"488","DRXDRSTZ","Dietary Recall Status","88","Blank but applicable","Dietary","DR1DRSTZ"
+"489","DRXDRSTZ","Dietary Recall Status","5","Not done","Dietary","DR1DRSTZ"
+"490",NA,NA,"2","No","Dietary","DR1.040Z"
+"491",NA,NA,"1","Yes (home)","Dietary","DR1.040Z"
+"492",NA,NA,"7","Refused","Dietary","DR1.040Z"
+"493",NA,NA,"9","Don't know","Dietary","DR1.040Z"
+"494","DRXTWSZ","Tap Water Source","1","Community supply","Dietary","DR1TWSZ"
+"495","DRXTWSZ","Tap Water Source","91","Other","Dietary","DR1TWSZ"
+"496","DRXTWSZ","Tap Water Source","4","Don't drink tap water","Dietary","DR1TWSZ"
+"497","DRXTWSZ","Tap Water Source","99","Don't know","Dietary","DR1TWSZ"
+"498","DBQ095Z","Type of salt you usually add at table","4","Doesn't use or add salt products at the table","Dietary","DBQ095Z"
+"499","DBQ095Z","Type of salt you usually add at table","1","Ordinary salt [includes regular iodized salt, sea salt and seasoning salts made with regular salt]","Dietary","DBQ095Z"
+"500","DBQ095Z","Type of salt you usually add at table","2","Lite salt","Dietary","DBQ095Z"
+"501","DBQ095Z","Type of salt you usually add at table","3","Salt substitute","Dietary","DBQ095Z"
+"502","DBQ095Z","Type of salt you usually add at table","88","Blank but applicable","Dietary","DBQ095Z"
+"503","DBQ095Z","Type of salt you usually add at table","99","Don't know","Dietary","DBQ095Z"
+"504","DBQ095Z","Type of salt you usually add at table","7","Refused","Dietary","DBQ095Z"
+"505","DBQ095Z","Type of salt you usually add at table","91","Other","Dietary","DBQ095Z"
+"506","DRXHELP","Who helped in responding for this interview","1","SP","Dietary","DR1HELP"
+"507","DRXHELP","Who helped in responding for this interview","4","Parent of SP","Dietary","DR1HELP"
+"508","DRXHELP","Who helped in responding for this interview","5","Spouse of SP","Dietary","DR1HELP"
+"509","DRXHELP","Who helped in responding for this interview","6","Child of SP","Dietary","DR1HELP"
+"510","DRXHELP","Who helped in responding for this interview","7","Grandparent of SP","Dietary","DR1HELP"
+"511","DRXHELP","Who helped in responding for this interview","8","Friend, Partner, Non Relative","Dietary","DR1HELP"
+"512","DRXHELP","Who helped in responding for this interview","9","Translator, not a HH member","Dietary","DR1HELP"
+"513","DRXHELP","Who helped in responding for this interview","10","Child care provider, Caretaker","Dietary","DR1HELP"
+"514","DRXHELP","Who helped in responding for this interview","11","Other Relative","Dietary","DR1HELP"
+"515","DRXHELP","Who helped in responding for this interview","12","No One","Dietary","DR1HELP"
+"516","DRXHELP","Who helped in responding for this interview","14","Other specify","Dietary","DR1HELP"
+"517","DRXHELP","Who helped in responding for this interview","77","Refused","Dietary","DR1HELP"
+"518","DRXHELP","Who helped in responding for this interview","99","Don't know","Dietary","DR1HELP"
+"519","DRXMRESP","Who was the main respondent for this interview?","1","SP","Dietary","DR1MRESP"
+"520","DRXMRESP","Who was the main respondent for this interview?","97","Proxy","Dietary","DR1MRESP"
+"521","DRXMRESP","Who was the main respondent for this interview?","98","SP and proxy","Dietary","DR1MRESP"
+"522","DRXMRESP","Who was the main respondent for this interview?","88","Blank but applicable","Dietary","DR1MRESP"
+"523","DRXMRESP","Who was the main respondent for this interview?","2","Mother of SP","Dietary","DR1MRESP"
+"524","DRXMRESP","Who was the main respondent for this interview?","3","Father of SP","Dietary","DR1MRESP"
+"525","DRXMRESP","Who was the main respondent for this interview?","5","Spouse of SP","Dietary","DR1MRESP"
+"526","DRXMRESP","Who was the main respondent for this interview?","6","Child of SP","Dietary","DR1MRESP"
+"527","DRXMRESP","Who was the main respondent for this interview?","7","Grandparent of SP","Dietary","DR1MRESP"
+"528","DRXMRESP","Who was the main respondent for this interview?","8","Friend, Partner, Non Relative","Dietary","DR1MRESP"
+"529","DRXMRESP","Who was the main respondent for this interview?","9","Translator, not a HH member","Dietary","DR1MRESP"
+"530","DRXMRESP","Who was the main respondent for this interview?","10","Child care provider, Caretaker","Dietary","DR1MRESP"
+"531","DRXMRESP","Who was the main respondent for this interview?","11","Other Relative","Dietary","DR1MRESP"
+"532","DRXMRESP","Who was the main respondent for this interview?","14","Other specify","Dietary","DR1MRESP"
+"533","DRXMRESP","Who was the main respondent for this interview?","77","Refused","Dietary","DR1MRESP"
+"534","DRXMRESP","Who was the main respondent for this interview?","99","Don't know","Dietary","DR1MRESP"
+"535","DRXTWSZ","Tap Water Source","1","Community supply","Dietary","DR2TWSZ"
+"536","DRXTWSZ","Tap Water Source","91","Other","Dietary","DR2TWSZ"
+"537","DRXTWSZ","Tap Water Source","4","Don't drink tap water","Dietary","DR2TWSZ"
+"538","DRXTWSZ","Tap Water Source","99","Don't know","Dietary","DR2TWSZ"
+"539","DRXHELP","Who helped in responding for this interview","1","SP","Dietary","DR2HELP"
+"540","DRXHELP","Who helped in responding for this interview","4","Parent of SP","Dietary","DR2HELP"
+"541","DRXHELP","Who helped in responding for this interview","5","Spouse of SP","Dietary","DR2HELP"
+"542","DRXHELP","Who helped in responding for this interview","6","Child of SP","Dietary","DR2HELP"
+"543","DRXHELP","Who helped in responding for this interview","7","Grandparent of SP","Dietary","DR2HELP"
+"544","DRXHELP","Who helped in responding for this interview","8","Friend, Partner, Non Relative","Dietary","DR2HELP"
+"545","DRXHELP","Who helped in responding for this interview","9","Translator, not a HH member","Dietary","DR2HELP"
+"546","DRXHELP","Who helped in responding for this interview","10","Child care provider, Caretaker","Dietary","DR2HELP"
+"547","DRXHELP","Who helped in responding for this interview","11","Other Relative","Dietary","DR2HELP"
+"548","DRXHELP","Who helped in responding for this interview","12","No One","Dietary","DR2HELP"
+"549","DRXHELP","Who helped in responding for this interview","14","Other specify","Dietary","DR2HELP"
+"550","DRXHELP","Who helped in responding for this interview","77","Refused","Dietary","DR2HELP"
+"551","DRXHELP","Who helped in responding for this interview","99","Don't know","Dietary","DR2HELP"
+"552","DRXMRESP","Who was the main respondent for this interview?","1","SP","Dietary","DR2MRESP"
+"553","DRXMRESP","Who was the main respondent for this interview?","2","Mother of SP","Dietary","DR2MRESP"
+"554","DRXMRESP","Who was the main respondent for this interview?","3","Father of SP","Dietary","DR2MRESP"
+"555","DRXMRESP","Who was the main respondent for this interview?","5","Spouse of SP","Dietary","DR2MRESP"
+"556","DRXMRESP","Who was the main respondent for this interview?","6","Child of SP","Dietary","DR2MRESP"
+"557","DRXMRESP","Who was the main respondent for this interview?","7","Grandparent of SP","Dietary","DR2MRESP"
+"558","DRXMRESP","Who was the main respondent for this interview?","8","Friend, Partner, Non Relative","Dietary","DR2MRESP"
+"559","DRXMRESP","Who was the main respondent for this interview?","9","Translator, not a HH member","Dietary","DR2MRESP"
+"560","DRXMRESP","Who was the main respondent for this interview?","10","Child care provider, Caretaker","Dietary","DR2MRESP"
+"561","DRXMRESP","Who was the main respondent for this interview?","11","Other Relative","Dietary","DR2MRESP"
+"562","DRXMRESP","Who was the main respondent for this interview?","14","Other specify","Dietary","DR2MRESP"
+"563","DRXMRESP","Who was the main respondent for this interview?","77","Refused","Dietary","DR2MRESP"
+"564","DRXMRESP","Who was the main respondent for this interview?","99","Don't know","Dietary","DR2MRESP"
+"565","DBD100","How often {do you/does SP} add ordinary salt to {your/his/her/SP's} food at the table? Would you say . . .","1","Rarely","Dietary","DBD100"
+"566","DBD100","How often {do you/does SP} add ordinary salt to {your/his/her/SP's} food at the table? Would you say . . .","2","Occasionally","Dietary","DBD100"
+"567","DBD100","How often {do you/does SP} add ordinary salt to {your/his/her/SP's} food at the table? Would you say . . .","3","Very often","Dietary","DBD100"
+"568","DBD100","How often {do you/does SP} add ordinary salt to {your/his/her/SP's} food at the table? Would you say . . .","88","Blank but applicable","Dietary","DBD100"
+"569","DBD100","How often {do you/does SP} add ordinary salt to {your/his/her/SP's} food at the table? Would you say . . .","9","Don't know","Dietary","DBD100"
+"570","DBD100","How often {do you/does SP} add ordinary salt to {your/his/her/SP's} food at the table? Would you say . . .","7","Refused","Dietary","DBD100"

data/tidytuesday_json_val.json

@@ -0,0 +1,1911 @@
+[
+    {
+        "date_posted": "2023-02-28",
+        "project_name": "African Language Sentiment",
+        "project_source": [
+            "https://r4ds.io/join",
+            "https://arxiv.org/pdf/2302.08956.pdf",
+            "https://github.com/shmuhammad2004",
+            "https://github.com/afrisenti-semeval/afrisent-semeval-2023"
+        ],
+        "description": "The data this week comes fromAfriSenti: Sentiment Analysis dataset for 14 African languagesvia@shmuhammad2004(the corresponding author on theassociated paper, and an active member of theR4DS Online Learning Community Slack). This repository contains data for the SemEval 2023 Shared Task 12: Sentiment Analysis in African Languages (AfriSenti-SemEval). The source repository also includes sentiment lexicons for several languages.",
+        "data_source_url": "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-02-28",
+        "data_dictionary": [
+            {
+                "variable": [
+                    "language_iso_code",
+                    "tweet",
+                    "label",
+                    "intended_use"
+                ],
+                "class": [
+                    "character",
+                    "character",
+                    "character",
+                    "character"
+                ],
+                "description": [
+                    "The unique code used to identify the language",
+                    "The text content of a tweet",
+                    "A sentiment label of positive, negative, or neutral assigned by a native speaker of that language",
+                    "Whether the data came from the dev, test, or train set for that language"
+                ]
+            },
+            {
+                "variable": [
+                    "language_iso_code",
+                    "language"
+                ],
+                "class": [
+                    "character",
+                    "character"
+                ],
+                "description": [
+                    "The unique code used to identify the language",
+                    "The name of the language"
+                ]
+            },
+            {
+                "variable": [
+                    "language_iso_code",
+                    "script"
+                ],
+                "class": [
+                    "character",
+                    "character"
+                ],
+                "description": [
+                    "The unique code used to identify the language",
+                    "The script used to write the language"
+                ]
+            },
+            {
+                "variable": [
+                    "language_iso_code",
+                    "country"
+                ],
+                "class": [
+                    "character",
+                    "character"
+                ],
+                "description": [
+                    "The unique code used to identify the language",
+                    "A country in which the language is spoken"
+                ]
+            },
+            {
+                "variable": [
+                    "country",
+                    "region"
+                ],
+                "class": [
+                    "character",
+                    "character"
+                ],
+                "description": [
+                    "A country in which the language is spoken",
+                    "The region of Africa in which that country is categorized. Note that Mozambique is categorized as \\\"East Africa\\\", \\\"Southern Africa\\\", and \\\"Southeastern Africa\\\""
+                ]
+            }
+        ],
+        "data": {
+            "file_name": [
+                "afrisenti.csv",
+                "country_regions.csv",
+                "language_countries.csv",
+                "language_scripts.csv",
+                "languages.csv"
+            ],
+            "file_url": [
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-02-28/afrisenti.csv",
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-02-28/country_regions.csv",
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-02-28/language_countries.csv",
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-02-28/language_scripts.csv",
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-02-28/languages.csv"
+            ]
+        },
+        "data_load": {
+            "file_name": [
+                "afrisenti.csv",
+                "country_regions.csv",
+                "language_countries.csv",
+                "language_scripts.csv",
+                "languages.csv"
+            ],
+            "file_url": [
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-02-28/afrisenti.csv",
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-02-28/country_regions.csv",
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-02-28/language_countries.csv",
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-02-28/language_scripts.csv",
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-02-28/languages.csv"
+            ]
+        }
+    },
+    {
+        "date_posted": "2023-05-02",
+        "project_name": "The Portal Project",
+        "project_source": [
+            "https://www.weecology.org/",
+            "https://weecology.github.io/portalr/",
+            "https://portal.weecology.org/",
+            "https://datacarpentry.org/ecology-workshop/",
+            "https://www.data-retriever.org/"
+        ],
+        "description": "The data this week comes from thePortal Project. This is a long-term ecological research site studying the dynamics of desert rodents, plants, ants and weather in Arizona. The Portal Project is a long-term ecological study being conducted near Portal, AZ. Since 1977, the site has been used to study the interactions among rodents, ants and plants and their respective responses to climate. To study the interactions among organisms, they experimentally manipulate access to 24 study plots. This study has produced over 100 scientific papers and is one of the longest running ecological studies in the U.S. TheWeecology research groupmonitors rodents, plants, ants, and weather. All data from the Portal Project are made openly available in near real-time so that they can provide the maximum benefit to scientific research and outreach. The core dataset is managed using an automated living data workflow run using GitHub and Continuous Analysis. This dataset focuses on the rodent data. Full data is available through these resources: The Portal Project data can also be accessed through the Data Retriever, a package manager for data. Data Retriever A teaching focused version of the dataset is also maintained with some of the complexities of the data removed to make it easy to use for computational training purposes. This dataset serves as the core dataset for theData Carpentry Ecologymaterial and has been downloaded almost 50,000 times. Thanks to @ethanwhite for the data cleaning script. This script downloads the data using the{portalr}package. It filters for the species and plot data, and years greater than 1977.",
+        "data_source_url": "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-05-02",
+        "data_dictionary": [
+            {
+                "variable": [
+                    "plot",
+                    "treatment"
+                ],
+                "class": [
+                    "double",
+                    "character"
+                ],
+                "description": [
+                    "Plot number",
+                    "Treatment type"
+                ]
+            },
+            {
+                "variable": [
+                    "species",
+                    "scientificname",
+                    "taxa",
+                    "commonname",
+                    "censustarget",
+                    "unidentified",
+                    "rodent",
+                    "granivore",
+                    "minhfl",
+                    "meanhfl",
+                    "maxhfl",
+                    "minwgt",
+                    "meanwgt",
+                    "maxwgt",
+                    "juvwgt"
+                ],
+                "class": [
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double"
+                ],
+                "description": [
+                    "Species",
+                    "Scientific Name",
+                    "Taxa",
+                    "Common Name",
+                    "Target species (0 or 1)",
+                    "Unidentified (0 or 1)",
+                    "Rodent (0 or 1)",
+                    "Granivore (0 or 1)",
+                    "Minimum hindfoot length",
+                    "Mean hindfoot length",
+                    "Maximum hindfoot length",
+                    "Minimum weight",
+                    "Mean weight",
+                    "Maximum weight",
+                    "Juvenile weight"
+                ]
+            },
+            {
+                "variable": [
+                    "censusdate",
+                    "month",
+                    "day",
+                    "year",
+                    "treatment",
+                    "plot",
+                    "stake",
+                    "species",
+                    "sex",
+                    "reprod",
+                    "age",
+                    "testes",
+                    "vagina",
+                    "pregnant",
+                    "nipples",
+                    "lactation",
+                    "hfl",
+                    "wgt",
+                    "tag",
+                    "note2",
+                    "ltag",
+                    "note3"
+                ],
+                "class": [
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "character",
+                    "double",
+                    "double",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "double",
+                    "double",
+                    "character",
+                    "character",
+                    "character",
+                    "character"
+                ],
+                "description": [
+                    "Census date",
+                    "Month",
+                    "Day",
+                    "Year",
+                    "Treatment type",
+                    "Plot number",
+                    "Stake number",
+                    "Species code",
+                    "Sex",
+                    "Reproductive condition",
+                    "Age",
+                    "Testes (Scrotal, Recent, or Minor)",
+                    "Vagina (Swollen, Plugged, or Both)",
+                    "Pregnant",
+                    "Nipples (Enlarged, Swollen, or Both)",
+                    "Lactating",
+                    "Hindfoot length",
+                    "Weight",
+                    "Primary individual identifier",
+                    "Newly tagged individual for 'tag'",
+                    "Secondary tag information when ear tags were used in both ears",
+                    "Newly tagged individual for 'ltag'"
+                ]
+            }
+        ],
+        "data": {
+            "file_name": [
+                "plots.csv",
+                "species.csv",
+                "surveys.csv"
+            ],
+            "file_url": [
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-05-02/plots.csv",
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-05-02/species.csv",
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-05-02/surveys.csv"
+            ]
+        },
+        "data_load": {
+            "file_name": [
+                "plots.csv",
+                "species.csv",
+                "surveys.csv"
+            ],
+            "file_url": [
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-05-02/plots.csv",
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-05-02/species.csv",
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-05-02/surveys.csv"
+            ]
+        }
+    },
+    {
+        "date_posted": "2023-04-04",
+        "project_name": "Premier League Match Data 2021-2022",
+        "project_source": [
+            "https://www.kaggle.com/datasets/evangower/premier-league-match-data",
+            "https://theathletic.com/3459766/2022/07/29/liverpool-manchester-city-premier-league-fouls-yellow-card/",
+            "https://github.com/evangower",
+            "https://www.kaggle.com/code/evangower/who-wins-the-epl-if-games-end-at-half-time/"
+        ],
+        "description": "The data this week comes from thePremier League Match Data 2021-2022viaEvan Goweron Kaggle. You can explore match day statistics of every game and every team during the 2021-22 season of the English Premier League Data. Data includes teams playing, date, referee, and stats for home and away side such as fouls, shots, cards, and more! Also included is a dataset of the weekly rankings for the season. The data was collected from the official website of the Premier League. Evan then cleaned the data using google sheets. Evan did an analysis ofWho wins the EPL if games end at half time?and there'san article from the Athleticabout fouls conceded per yellow card article. No data cleaning",
+        "data_source_url": "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-04-04",
+        "data_dictionary": [
+            {
+                "variable": [
+                    "Date",
+                    "HomeTeam",
+                    "AwayTeam",
+                    "FTHG",
+                    "FTAG",
+                    "FTR",
+                    "HTHG",
+                    "HTAG",
+                    "HTR",
+                    "Referee",
+                    "HS",
+                    "AS",
+                    "HST",
+                    "AST",
+                    "HF",
+                    "AF",
+                    "HC",
+                    "AC",
+                    "HY",
+                    "AY",
+                    "HR",
+                    "AR"
+                ],
+                "class": [
+                    "character",
+                    "character",
+                    "character",
+                    "double",
+                    "double",
+                    "character",
+                    "double",
+                    "double",
+                    "character",
+                    "character",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double"
+                ],
+                "description": [
+                    "The date when the match was played",
+                    "The home team",
+                    "The away team",
+                    "Full time home goals",
+                    "Full time away goals",
+                    "Full time result",
+                    "Halftime home goals",
+                    "Halftime away goals",
+                    "Halftime results",
+                    "Referee of the match",
+                    "Number of shots taken by the home team",
+                    "Number of shots taken by the away team",
+                    "Number of shots on target by the home team",
+                    "Number of shots on target by the away team",
+                    "Number of fouls by the home team",
+                    "Number of fouls by the away team",
+                    "Number of corners taken by the home team",
+                    "Number of corners taken by the away team",
+                    "Number of yellow cards received by the home team",
+                    "Number of yellow cards received by the away team",
+                    "Number of red cards received by the home team",
+                    "Number of red cards received by the away team"
+                ]
+            }
+        ],
+        "data": {
+            "file_name": [
+                "soccer21-22.csv"
+            ],
+            "file_url": [
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-04-04/soccer21-22.csv"
+            ]
+        },
+        "data_load": {
+            "file_name": [
+                "soccer21-22.csv"
+            ],
+            "file_url": [
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-04-04/soccer21-22.csv"
+            ]
+        }
+    },
+    {
+        "date_posted": "2023-02-07",
+        "project_name": "Big Tech Stock Prices",
+        "project_source": [
+            "https://github.com/rfordatascience/tidytuesday/issues/509",
+            "https://www.morningstar.com/articles/1129535/5-charts-on-big-tech-stocks-collapse",
+            "https://www.kaggle.com/datasets/evangower/big-tech-stock-prices"
+        ],
+        "description": "The data this week comes from Yahoo Finance viaKaggle(byEvan Gower). This dataset consists of the daily stock prices and volume of 14 different tech companies, including Apple (AAPL), Amazon (AMZN), Alphabet (GOOGL), and Meta Platforms (META) and more! A number of articles have examined the collapse of \"Big Tech\" stock prices, includingthis article from morningstar.com. Note: Allstock_symbols have 3271 prices, except META (2688) and TSLA (3148) because they were not publicly traded for part of the period examined.",
+        "data_source_url": "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-02-07",
+        "data_dictionary": [
+            {
+                "variable": [
+                    "stock_symbol",
+                    "date",
+                    "open",
+                    "high",
+                    "low",
+                    "close",
+                    "adj_close",
+                    "volume"
+                ],
+                "class": [
+                    "character",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double"
+                ],
+                "description": [
+                    "stock_symbol",
+                    "date",
+                    "The price at market open.",
+                    "The highest price for that day.",
+                    "The lowest price for that day.",
+                    "The price at market close, adjusted for splits.",
+                    "The closing price after adjustments for all applicable splits and dividend distributions. Data is adjusted using appropriate split and dividend multipliers, adhering to Center for Research in Security Prices (CRSP) standards.",
+                    "The number of shares traded on that day."
+                ]
+            },
+            {
+                "variable": [
+                    "stock_symbol",
+                    "company"
+                ],
+                "class": [
+                    "character",
+                    "character"
+                ],
+                "description": [
+                    "stock_symbol",
+                    "Full name of the company."
+                ]
+            }
+        ],
+        "data": {
+            "file_name": [
+                "big_tech_companies.csv",
+                "big_tech_stock_prices.csv"
+            ],
+            "file_url": [
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-02-07/big_tech_companies.csv",
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-02-07/big_tech_stock_prices.csv"
+            ]
+        },
+        "data_load": {
+            "file_name": [
+                "big_tech_companies.csv",
+                "big_tech_stock_prices.csv"
+            ],
+            "file_url": [
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-02-07/big_tech_companies.csv",
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-02-07/big_tech_stock_prices.csv"
+            ]
+        }
+    },
+    {
+        "date_posted": "2023-03-21",
+        "project_name": "Programming Languages",
+        "project_source": [
+            "https://github.com/rfordatascience/tidytuesday/issues/530",
+            "https://pldb.com/posts/does-every-programming-language-support-line-comments.html",
+            "https://pldb.com/csv.html",
+            "https://pldb.com/index.html",
+            "https://pldb.com/posts/index.html"
+        ],
+        "description": "The data this week comes from theProgramming Language DataBase. Thanks toJesus M. Castagnettofor the suggestion! The PLDB has ablogwith numerous articles exploring the data, such asDoes every programming language have line comments?. The data is user-submitted, so you might want to confirm the accuracy of anything particularly surprising that you find before stating it with certainty! Thefull data dictionaryis available from PLDB.com.",
+        "data_source_url": "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-03-21",
+        "data_dictionary": [
+            {
+                "variable": [
+                    "pldb_id",
+                    "title",
+                    "description",
+                    "type",
+                    "appeared",
+                    "creators",
+                    "website",
+                    "domain_name",
+                    "domain_name_registered",
+                    "reference",
+                    "isbndb",
+                    "book_count",
+                    "semantic_scholar",
+                    "language_rank",
+                    "github_repo",
+                    "github_repo_stars",
+                    "github_repo_forks",
+                    "github_repo_updated",
+                    "github_repo_subscribers",
+                    "github_repo_created",
+                    "github_repo_description",
+                    "github_repo_issues",
+                    "github_repo_first_commit",
+                    "github_language",
+                    "github_language_tm_scope",
+                    "github_language_type",
+                    "github_language_ace_mode",
+                    "github_language_file_extensions",
+                    "github_language_repos",
+                    "wikipedia",
+                    "wikipedia_daily_page_views",
+                    "wikipedia_backlinks_count",
+                    "wikipedia_summary",
+                    "wikipedia_page_id",
+                    "wikipedia_appeared",
+                    "wikipedia_created",
+                    "wikipedia_revision_count",
+                    "wikipedia_related",
+                    "features_has_comments",
+                    "features_has_semantic_indentation",
+                    "features_has_line_comments",
+                    "line_comment_token",
+                    "last_activity",
+                    "number_of_users",
+                    "number_of_jobs",
+                    "origin_community",
+                    "central_package_repository_count",
+                    "file_type",
+                    "is_open_source"
+                ],
+                "class": [
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "double",
+                    "character",
+                    "character",
+                    "character",
+                    "double",
+                    "character",
+                    "double",
+                    "double",
+                    "integer",
+                    "double",
+                    "character",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "character",
+                    "double",
+                    "double",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "double",
+                    "character",
+                    "double",
+                    "double",
+                    "character",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "character",
+                    "logical",
+                    "logical",
+                    "logical",
+                    "character",
+                    "double",
+                    "double",
+                    "double",
+                    "character",
+                    "double",
+                    "character",
+                    "logical"
+                ],
+                "description": [
+                    "A standardized, uniquified version of the language name, used as an ID on the PLDB site.",
+                    "The official title of the language.",
+                    "Description of the repo on GitHub.",
+                    "Which category in PLDB's subjective ontology does this entity fit into.",
+                    "What year was the language publicly released and/or announced?",
+                    "Name(s) of the original creators of the language delimited by \\\" and \\\"",
+                    "URL of the official homepage for the language project.",
+                    "If the project website is on its own domain.",
+                    "When was this domain first registered?",
+                    "A link to more info about this entity.",
+                    "Books about this language from ISBNdb.",
+                    "Computed; the number of books found for this language at isbndb.com",
+                    "Papers about this language from Semantic Scholar.",
+                    "Computed; A rank for the language, taking into account various online rankings. The computation for this column is not currently clear.",
+                    "URL of the official GitHub repo for the project if it hosted there.",
+                    "How many stars of the repo?",
+                    "How many forks of the repo?",
+                    "What year was the last commit made?",
+                    "How many subscribers to the repo?",
+                    "When was the Github repo for this entity created?",
+                    "Description of the repo on GitHub.",
+                    "How many isses on the repo?",
+                    "What year the first commit made in this git repo?",
+                    "GitHub has a set of supported languages as defined here",
+                    "The TextMate scope that represents this programming language.",
+                    "Either data, programming, markup, prose, or nil.",
+                    "A String name of the Ace Mode used for highlighting whenever a file is edited. This must match one of the filenames in http://git.io/3XO_Cg. Use \\\"text\\\" if a mode does not exist.",
+                    "An Array of associated extensions (the first one is considered the primary extension, the others should be listed alphabetically).",
+                    "How many repos for this language does GitHub report?",
+                    "URL of the entity on Wikipedia, if and only if it has a page dedicated to it.",
+                    "How many page views per day does this Wikipedia page get? Useful as a signal for rankings. Available via WP api.",
+                    "How many pages on WP link to this page?",
+                    "What is the text summary of the language from the Wikipedia page?",
+                    "Waht is the internal ID for this entity on WP?",
+                    "When does Wikipedia claim this entity first appeared?",
+                    "When was the Wikipedia page for this entity created?",
+                    "How many revisions does this page have?",
+                    "What languages does Wikipedia have as related?",
+                    "Does this language have a comment character?",
+                    "Does indentation have semantic meaning in this language?",
+                    "Does this language support inline comments (as opposed to comments that must span an entire line)?",
+                    "Defined as a token that can be placed anywhere on a line and starts a comment that cannot be stopped except by a line break character or end of file.",
+                    "Computed; The most recent of any year field in the PLDB for this language.",
+                    "Computed; \\\"Crude user estimate from a linear model.",
+                    "Computed; The estimated number of job openings for programmers in this language.",
+                    "In what community(ies) did the language first originate?",
+                    "Number of packages in a central repository. If this value is not known, it is set to 0 (so \\\"0\\\" can mean \\\"no repository exists\\\", \\\"the repository exists but is empty\\\" (unlikely), or \\\"we do not know if a repository exists\\\". This value is definitely incorrect for R.",
+                    "What is the file encoding for programs in this language?",
+                    "Is it an open source project?"
+                ]
+            }
+        ],
+        "data": {
+            "file_name": [
+                "languages.csv"
+            ],
+            "file_url": [
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-03-21/languages.csv"
+            ]
+        },
+        "data_load": {
+            "file_name": [
+                "languages.csv"
+            ],
+            "file_url": [
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-03-21/languages.csv"
+            ]
+        }
+    },
+    {
+        "date_posted": "2023-05-23",
+        "project_name": "Central Park Squirrel Census",
+        "project_source": [
+            "https://data.cityofnewyork.us/Environment/2018-Central-Park-Squirrel-Census-Squirrel-Data/vfnx-vebw",
+            "https://www.thesquirrelcensus.com/"
+        ],
+        "description": "Squirrel data! The data this week comes from the2018 Central Park Squirrel Census. The Squirrel Censusis a multimedia science, design, and storytelling project focusing on the Eastern gray (Sciurus carolinensis). They count squirrels and present their findings to the public. The dataset contains squirrel data for each of the 3,023 sightings, including location coordinates, age, primary and secondary fur color, elevation, activities, communications, and interactions between squirrels and with humans. No data cleaning",
+        "data_source_url": "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-05-23",
+        "data_dictionary": [
+            {
+                "variable": [
+                    "X",
+                    "Y",
+                    "Unique Squirrel ID",
+                    "Hectare",
+                    "Shift",
+                    "Date",
+                    "Hectare Squirrel Number",
+                    "Age",
+                    "Primary Fur Color",
+                    "Highlight Fur Color",
+                    "Combination of Primary and Highlight Color",
+                    "Color notes",
+                    "Location",
+                    "Above Ground Sighter Measurement",
+                    "Specific Location",
+                    "Running",
+                    "Chasing",
+                    "Climbing",
+                    "Eating",
+                    "Foraging",
+                    "Other Activities",
+                    "Kuks",
+                    "Quaas",
+                    "Moans",
+                    "Tail flags",
+                    "Tail twitches",
+                    "Approaches",
+                    "Indifferent",
+                    "Runs from",
+                    "Other Interactions",
+                    "Lat/Long"
+                ],
+                "class": [
+                    "double",
+                    "double",
+                    "character",
+                    "character",
+                    "character",
+                    "double",
+                    "double",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "logical",
+                    "logical",
+                    "logical",
+                    "logical",
+                    "logical",
+                    "character",
+                    "logical",
+                    "logical",
+                    "logical",
+                    "logical",
+                    "logical",
+                    "logical",
+                    "logical",
+                    "logical",
+                    "character",
+                    "character"
+                ],
+                "description": [
+                    "Longitude coordinate for squirrel sighting point",
+                    "Latitude coordinate for squirrel sighting point",
+                    "Identification tag for each squirrel sightings. The tag is comprised of \\\"Hectare ID\\\" + \\\"Shift\\\" + \\\"Date\\\" + \\\"Hectare Squirrel Number.\\\"",
+                    "ID tag, which is derived from the hectare grid used to divide and count the park area. One axis that runs predominantly north-to-south is numerical (1-42), and the axis that runs predominantly east-to-west is roman characters (A-I).",
+                    "Value is either \\\"AM\\\" or \\\"PM,\\\" to communicate whether or not the sighting session occurred in the morning or late afternoon.",
+                    "Concatenation of the sighting session day and month.",
+                    "Number within the chronological sequence of squirrel sightings for a discrete sighting session.",
+                    "Value is either \\\"Adult\\\" or \\\"Juvenile.\\\"",
+                    "Primary Fur Color - value is either \\\"Gray,\\\" \\\"Cinnamon\\\" or \\\"Black.\\\"",
+                    "Discrete value or string values comprised of \\\"Gray,\\\" \\\"Cinnamon\\\" or \\\"Black.\\\"",
+                    "A combination of the previous two columns; this column gives the total permutations of primary and highlight colors observed.",
+                    "Sighters occasionally added commentary on the squirrel fur conditions. These notes are provided here.",
+                    "Value is either \\\"Ground Plane\\\" or \\\"Above Ground.\\\" Sighters were instructed to indicate the location of where the squirrel was when first sighted.",
+                    "For squirrel sightings on the ground plane, fields were populated with a value of \\\"FALSE.\\\"",
+                    "Sighters occasionally added commentary on the squirrel location. These notes are provided here.",
+                    "Squirrel was seen running.",
+                    "Squirrel was seen chasing another squirrel.",
+                    "Squirrel was seen climbing a tree or other environmental landmark.",
+                    "Squirrel was seen eating.",
+                    "Squirrel was seen foraging for food.",
+                    "Other activities squirrels were observed doing.",
+                    "Squirrel was heard kukking, a chirpy vocal communication used for a variety of reasons.",
+                    "Squirrel was heard quaaing, an elongated vocal communication which can indicate the presence of a ground predator such as a dog.",
+                    "Squirrel was heard moaning, a high-pitched vocal communication which can indicate the presence of an air predator such as a hawk.",
+                    "Squirrel was seen flagging its tail. Flagging is a whipping motion used to exaggerate squirrel's size and confuse rivals or predators. Looks as if the squirrel is scribbling with tail into the air.",
+                    "Squirrel was seen twitching its tail. Looks like a wave running through the tail, like a breakdancer doing the arm wave. Often used to communicate interest, curiosity.",
+                    "Squirrel was seen approaching human, seeking food.",
+                    "Squirrel was indifferent to human presence.",
+                    "Squirrel was seen running from humans, seeing them as a threat.",
+                    "Sighter notes on other types of interactions between squirrels and humans.",
+                    "Latitude and longitude"
+                ]
+            }
+        ],
+        "data": {
+            "file_name": [
+                "squirrel_data.csv"
+            ],
+            "file_url": [
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-05-23/squirrel_data.csv"
+            ]
+        },
+        "data_load": {
+            "file_name": [
+                "squirrel_data.csv"
+            ],
+            "file_url": [
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-05-23/squirrel_data.csv"
+            ]
+        }
+    },
+    {
+        "date_posted": "2023-01-17",
+        "project_name": "Art History",
+        "project_source": [
+            "https://research.repository.duke.edu/concern/datasets/q811kk70n?locale=en",
+            "https://github.com/hollandstam1/thesis",
+            "https://saralemus7.github.io/arthistory/",
+            "https://github.com/saralemus7/arthistory"
+        ],
+        "description": "The data this week comes from thearthistory data package This dataset contains data that was used for Holland Stam's thesis work, titledQuantifying art historical narratives. The data was collected to assess the demographic representation of artists through editions of Janson's History of Art and Gardner's Art Through the Ages, two of the most popular art history textbooks used in the American education system. In this package specifically, both artist-level and work-level data was collected along with variables regarding the artists' demographics and numeric metrics for describing how much space they or their work took up in each edition of each textbook. This package contains three datasets: Acknowledging arthistory Citation Lemus S, Stam H (2022). arthistory: Art History Textbook Data.https://github.com/saralemus7/arthistory,https://saralemus7.github.io/arthistory/. Examples of analyses are included inHolland Stam's thesisin Quarto files. No data cleaning",
+        "data_source_url": "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-01-17",
+        "data_dictionary": [
+            {
+                "variable": [
+                    "artist_name",
+                    "edition_number",
+                    "year",
+                    "artist_nationality",
+                    "artist_nationality_other",
+                    "artist_gender",
+                    "artist_race",
+                    "artist_ethnicity",
+                    "book",
+                    "space_ratio_per_page_total",
+                    "artist_unique_id",
+                    "moma_count_to_year",
+                    "whitney_count_to_year",
+                    "artist_race_nwi"
+                ],
+                "class": [
+                    "character",
+                    "double",
+                    "double",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "character"
+                ],
+                "description": [
+                    "The name of each artist",
+                    "The edition number of the textbook from either Janson's History or Art or Gardner's Art Through the Ages.",
+                    "The year of publication for a given edition of Janson or Gardner.",
+                    "The nationality of a given artist.",
+                    "The nationality of the artist. Of the total count of artists through all editions of Janson's History of Art and Gardner's Art Through the Ages, 77.32% account for French, Spanish, British, American and German. Therefore, the categorical strings of this variable are French, Spanish, British, American, German and Other",
+                    "The gender of the artist",
+                    "The race of the artist",
+                    "The ethnicity of the artist",
+                    "Which book, either Janson or Gardner the particular artist at that particular time was included.",
+                    "The area in centimeters squared of both the text and the figure of a particular artist in a given edition of Janson's History of Art divided by the area in centimeters squared of a single page of the respective edition. This variable is continuous.",
+                    "The unique identifying number assigned to artists across books is denoted in alphabetical order. This variable is discrete.",
+                    "The total count of exhibitions ever held by the Museum of Modern Art (MoMA) of a particular artist at a given year of publication. This variable is discrete.",
+                    "The count of exhibitions held by The Whitney of a particular artist at a particular moment of time, as highlighted by year. This variable in discrete.",
+                    "The non-white indicator for artist race, meaning if an artist's race is denoted as either white or non-white."
+                ]
+            }
+        ],
+        "data": {
+            "file_name": [
+                "artists.csv"
+            ],
+            "file_url": [
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-01-17/artists.csv"
+            ]
+        },
+        "data_load": {
+            "file_name": [
+                "artists.csv"
+            ],
+            "file_url": [
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-01-17/artists.csv"
+            ]
+        }
+    },
+    {
+        "date_posted": "2023-07-04",
+        "project_name": "Historical Markers",
+        "project_source": [
+            "http://www.geonames.org/",
+            "https://www.hmdb.org/geolists.asp?c=United%20States%20of%20America",
+            "https://www.hmdb.org/stats.asp",
+            "https://www.hmdb.org/",
+            "https://github.com/rfordatascience/tidytuesday/issues/574#issuecomment-1601050053"
+        ],
+        "description": "The data this week comes from theHistorical Marker Database USA Index. Learn more about the markers on theHMDb.org site, which includes a number of articles, includingDatabase Counts and Statistics. We included a dataset of places that donothave entries in the Historical Markers Database. You might try to combine that with information fromgeonames.org(code: HSTS) to find markers that need to be submitted. Thanks toJesus M. Castagnettofor the geonames tip!",
+        "data_source_url": "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-07-04",
+        "data_dictionary": [
+            {
+                "variable": [
+                    "marker_id",
+                    "marker_no",
+                    "title",
+                    "subtitle",
+                    "addl_subtitle",
+                    "year_erected",
+                    "erected_by",
+                    "latitude_minus_s",
+                    "longitude_minus_w",
+                    "street_address",
+                    "city_or_town",
+                    "section_or_quarter",
+                    "county_or_parish",
+                    "state_or_prov",
+                    "location",
+                    "missing",
+                    "link"
+                ],
+                "class": [
+                    "double",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "integer",
+                    "character",
+                    "double",
+                    "double",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "character"
+                ],
+                "description": [
+                    "Unique ID for this marker in the HMdb.",
+                    "Number of this marker in the state numbering scheme.",
+                    "Main title of the marker.",
+                    "Subtitle of the marker, if any.",
+                    "Additional subtitle text.",
+                    "The year in which the marker was erected.",
+                    "The organization which erected the marker.",
+                    "The latitude of the marker.",
+                    "The longitude of the marker.",
+                    "The street address of the marker, if available.",
+                    "The city, town, etc in which the marker is located.",
+                    "The section of the city, town, etc, when available.",
+                    "The county, parish, or similar designation in which the marker appears.",
+                    "The state, province, territory, etc in which the marker appears.",
+                    "A description of the marker's location.",
+                    "Whether the marker is \\\"Reported missing\\\" or \\\"Confirmed missing\\\". NA values indicate that the marker has neither been reported missing nor confirmed as missing.",
+                    "The HMDb link to the marker. Links include additional details, such as photos and topic lists to which this marker belongs."
+                ]
+            },
+            {
+                "variable": [
+                    "county",
+                    "state"
+                ],
+                "class": [
+                    "character",
+                    "character"
+                ],
+                "description": [
+                    "County or equivalent.",
+                    "State or territory."
+                ]
+            }
+        ],
+        "data": {
+            "file_name": [
+                "historical_markers.csv",
+                "no_markers.csv"
+            ],
+            "file_url": [
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-07-04/historical_markers.csv",
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-07-04/no_markers.csv"
+            ]
+        },
+        "data_load": {
+            "file_name": [
+                "historical_markers.csv",
+                "no_markers.csv"
+            ],
+            "file_url": [
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-07-04/historical_markers.csv",
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-07-04/no_markers.csv"
+            ]
+        }
+    },
+    {
+        "date_posted": "2023-02-14",
+        "project_name": "Hollywood Age Gaps",
+        "project_source": [
+            "https://www.data-is-plural.com/archive/2018-02-07-edition/",
+            "https://tidytues.day/2021/2021-03-09",
+            "https://hollywoodagegap.com/"
+        ],
+        "description": "The data this week comes fromHollywood Age GapviaData Is Plural. An informational site showing the age gap between movie love interests. The data follows certain rules: The two (or more) actors play actual love interests (not just friends, coworkers, or some other non-romantic type of relationship) The youngest of the two actors is at least 17 years old Not animated characters We previously provided a dataset about theBechdel Test. It might be interesting to see whether there is any correlation between these datasets! The Bechdel Test dataset also included additional information about the films that were used in that dataset. Note: The age gaps dataset includes \"gender\" columns, which always contain the values \"man\" or \"woman\". These values appear to indicate how thecharactersin each film identify. Some of these values do not match how theactoridentifies. We apologize if any characters are misgendered in the data!",
+        "data_source_url": "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-02-14",
+        "data_dictionary": [
+            {
+                "variable": [
+                    "movie_name",
+                    "release_year",
+                    "director",
+                    "age_difference",
+                    "couple_number",
+                    "actor_1_name",
+                    "actor_2_name",
+                    "character_1_gender",
+                    "character_2_gender",
+                    "actor_1_birthdate",
+                    "actor_2_birthdate",
+                    "actor_1_age",
+                    "actor_2_age"
+                ],
+                "class": [
+                    "character",
+                    "integer",
+                    "character",
+                    "integer",
+                    "integer",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "date",
+                    "date",
+                    "integer",
+                    "integer"
+                ],
+                "description": [
+                    "Name of the film",
+                    "Release year",
+                    "Director of the film",
+                    "Age difference between the characters in whole years",
+                    "An identifier for the couple in case multiple couples are listed for this film",
+                    "The name of the older actor in this couple",
+                    "The name of the younger actor in this couple",
+                    "The gender of the older character, as identified by the person who submitted the data for this couple",
+                    "The gender of the younger character, as identified by the person who submitted the data for this couple",
+                    "The birthdate of the older member of the couple",
+                    "The birthdate of the younger member of the couple",
+                    "The age of the older actor when the film was released",
+                    "The age of the younger actor when the film was released"
+                ]
+            }
+        ],
+        "data": {
+            "file_name": [
+                "age_gaps.csv"
+            ],
+            "file_url": [
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-02-14/age_gaps.csv"
+            ]
+        },
+        "data_load": {
+            "file_name": [
+                "age_gaps.csv"
+            ],
+            "file_url": [
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-02-14/age_gaps.csv"
+            ]
+        }
+    },
+    {
+        "date_posted": "2023-08-15",
+        "project_name": "Spam E-mail",
+        "project_source": [
+            "https://vincentarelbundock.github.io/Rdatasets/index.html",
+            "https://archive.ics.uci.edu/dataset/94/spambase",
+            "https://search.r-project.org/CRAN/refmans/kernlab/html/spam.html",
+            "https://vincentarelbundock.github.io/Rdatasets/doc/DAAG/spam7.html"
+        ],
+        "description": "The data this week comes from Vincent Arel-Bundock's Rdatasets package(https://vincentarelbundock.github.io/Rdatasets/index.html). Rdatasets is a collection of 2246 datasets which were originally distributed alongside the statistical software environment R and some of its add-on packages. The goal is to make these data more broadly accessible for teaching and statistical software development. We're working with thespam emaildataset. This is a subset of thespam e-mail database. This is a dataset collected at Hewlett-Packard Labs by Mark Hopkins, Erik Reeber, George Forman, and Jaap Suermondt and shared with theUCI Machine Learning Repository. The dataset classifies 4601 e-mails as spam or non-spam, with additional variables indicating the frequency of certain words and characters in the e-mail. First column was removed.",
+        "data_source_url": "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-08-15",
+        "data_dictionary": [
+            {
+                "variable": [
+                    "crl.tot",
+                    "dollar",
+                    "bang",
+                    "money",
+                    "n000",
+                    "make",
+                    "yesno"
+                ],
+                "class": [
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "character"
+                ],
+                "description": [
+                    "Total length of uninterrupted sequences of capitals",
+                    "Occurrences of the dollar sign, as percent of total number of characters",
+                    "Occurrences of ‘!’, as percent of total number of characters",
+                    "Occurrences of ‘money’, as percent of total number of characters",
+                    "Occurrences of the string ‘000’, as percent of total number of words",
+                    "Occurrences of ‘make’, as a percent of total number of words",
+                    "Outcome variable, a factor with levels 'n' not spam, 'y' spam"
+                ]
+            }
+        ],
+        "data": {
+            "file_name": [
+                "spam.csv"
+            ],
+            "file_url": [
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-08-15/spam.csv"
+            ]
+        },
+        "data_load": {
+            "file_name": [
+                "spam.csv"
+            ],
+            "file_url": [
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-08-15/spam.csv"
+            ]
+        }
+    },
+    {
+        "date_posted": "2023-03-07",
+        "project_name": "Numbats in Australia",
+        "project_source": [
+            "/rfordatascience/tidytuesday/blob/master/data/2023/2023-03-07/data/numbats.csv",
+            "https://www.ala.org.au",
+            "https://github.com/numbats/numbats-tidytuesday",
+            "https://bie.ala.org.au/species/https://biodiversity.org.au/afd/taxa/6c72d199-f0f1-44d3-8197-224a2f7cff5f"
+        ],
+        "description": "The data this week comes from theAtlas of Living Australia. Thanks to Di Cook forpreparing this week's dataset! ThisNumbat page at the Atlas of Living Australiatalks about these endangered species in greater detail. Acsvfile of numbat sightings is provided. The code to refresh the data is below. Questions that would be interesting to answer are:",
+        "data_source_url": "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-03-07",
+        "data_dictionary": [
+            {
+                "variable": [
+                    "decimalLatitude",
+                    "decimalLongitude",
+                    "eventDate",
+                    "scientificName",
+                    "taxonConceptID",
+                    "recordID",
+                    "dataResourceName",
+                    "year",
+                    "month",
+                    "wday",
+                    "hour",
+                    "day",
+                    "dryandra",
+                    "prcp",
+                    "tmax",
+                    "tmin"
+                ],
+                "class": [
+                    "double",
+                    "double",
+                    "datetime",
+                    "factor",
+                    "factor",
+                    "character",
+                    "factor",
+                    "integer",
+                    "factor",
+                    "factor",
+                    "integer",
+                    "date",
+                    "logical",
+                    "double",
+                    "double",
+                    "double"
+                ],
+                "description": [
+                    "decimalLatitude",
+                    "decimalLongitude",
+                    "eventDate",
+                    "Either \\\"Myrmecobius fasciatus\\\" or \\\"Myrmecobius fasciatus rufus\\\"",
+                    "The URL for this (sub)species",
+                    "recordID",
+                    "dataResourceName",
+                    "The 4-digit year of the event (when available)",
+                    "The 3-letter month abbreviation of the event (when available)",
+                    "The 3-letter weekday abbreviation of the event (when available)",
+                    "The hour of the event (when available)",
+                    "The date of the event (when available)",
+                    "whether the observation was in Dryandra Woodland",
+                    "Precipitation on that day in Dryandra Woodland (when relevant), in millimeters",
+                    "Maximum temperature on that day in Dryandra Woodland (when relevant), in degrees Celsius",
+                    "Minimum temperature on that day in Dryandra Woodland (when relevant), in degrees Celsius"
+                ]
+            }
+        ],
+        "data": {
+            "file_name": [
+                "numbats.csv"
+            ],
+            "file_url": [
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-03-07/numbats.csv"
+            ]
+        },
+        "data_load": {
+            "file_name": [
+                "numbats.csv"
+            ],
+            "file_url": [
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-03-07/numbats.csv"
+            ]
+        }
+    },
+    {
+        "date_posted": "2023-11-28",
+        "project_name": "Doctor Who Episodes",
+        "project_source": [
+            "https://en.wikipedia.org/wiki/List_of_Doctor_Who_episodes_(2005%E2%80%93present)",
+            "https://github.com/KittJonathan/datardis/tree/main/misc",
+            "https://cran.r-project.org/package=datardis",
+            "https://github.com/KittJonathan/datardis"
+        ],
+        "description": "Doctor Who is an extremely long-running British television program. The show was revived in 2005, and has proven very popular since then. To celebrate this year's 60th anniversary of Doctor Who, we have three datasets. The data this week comes from Wikipedia's [List of Doctor Who episodes](https://en.wikipedia.org/wiki/List_of_Doctor_Who_episodes_(2005%E2%80%93present)via the{datardis} packagebyJonathan Kitt. Thank you to Jonathan for compiling and sharing this data! As of 2023-11-24, the data only includes episodes from the \"revived\" era. For an added challenge, consider submitting a pull request to the {datardis} package to update thedata-extraction scriptsto also fetch the \"classic\" era data! Clean data from the{datardis} package.",
+        "data_source_url": "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-11-28",
+        "data_dictionary": [
+            {
+                "variable": [
+                    "era",
+                    "season_number",
+                    "serial_title",
+                    "story_number",
+                    "episode_number",
+                    "episode_title",
+                    "type",
+                    "first_aired",
+                    "production_code",
+                    "uk_viewers",
+                    "rating",
+                    "duration"
+                ],
+                "class": [
+                    "character",
+                    "double",
+                    "character",
+                    "character",
+                    "double",
+                    "character",
+                    "character",
+                    "double",
+                    "character",
+                    "double",
+                    "double",
+                    "double"
+                ],
+                "description": [
+                    "Whether the episode is in the \\\"classic\\\" or \\\"revived\\\" era. All data in this dataset is within the \\\"revived\\\" era.",
+                    "The season number within the era. Note that some episodes are outside of a season.",
+                    "Serial title if available",
+                    "Story number",
+                    "Episode number in season",
+                    "Episode title",
+                    "\\\"episode\\\" or \\\"special\\\"",
+                    "Date the episode first aired in the U.K.",
+                    "Episode's production code if available",
+                    "Number of U.K. viewers (millions)",
+                    "Episode's rating",
+                    "Episode's duration in minutes"
+                ]
+            },
+            {
+                "variable": [
+                    "story_number",
+                    "director"
+                ],
+                "class": [
+                    "character",
+                    "character"
+                ],
+                "description": [
+                    "Story number",
+                    "Episode's director"
+                ]
+            },
+            {
+                "variable": [
+                    "story_number",
+                    "writer"
+                ],
+                "class": [
+                    "character",
+                    "character"
+                ],
+                "description": [
+                    "Story number",
+                    "Episode's writer"
+                ]
+            }
+        ],
+        "data": {
+            "file_name": [
+                "drwho_directors.csv",
+                "drwho_episodes.csv",
+                "drwho_writers.csv"
+            ],
+            "file_url": [
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-11-28/drwho_directors.csv",
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-11-28/drwho_episodes.csv",
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-11-28/drwho_writers.csv"
+            ]
+        },
+        "data_load": {
+            "file_name": [
+                "drwho_directors.csv",
+                "drwho_episodes.csv",
+                "drwho_writers.csv"
+            ],
+            "file_url": [
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-11-28/drwho_directors.csv",
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-11-28/drwho_episodes.csv",
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-11-28/drwho_writers.csv"
+            ]
+        }
+    },
+    {
+        "date_posted": "2023-11-14",
+        "project_name": "Diwali Sales Data",
+        "project_source": [
+            "https://www.kaggle.com/code/bhushanshelke69/diwali-data-exploration",
+            "https://github.com/vikasvachheta08/Diwali_Sales_Analysis_Using_Python",
+            "https://www.kaggle.com/datasets/saadharoon27/diwali-sales-dataset"
+        ],
+        "description": "This week is Diwali, the festival of lights! The data this week comes fromsales datafor a retail store during the Diwali festival period in India. The data is shared on Kaggle by Saad Haroon. This week we're sharing Python data analysis examples! There's a few out there, but these ones fromBrushan ShelkeorVikas Vachheta(see the Diwali_Sales_Analysis.ipynb file for the code) are some data exploration analyses. Data was downloaded fromKaggle, and theStatusandunnamed1columns removed.",
+        "data_source_url": "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-11-14",
+        "data_dictionary": [
+            {
+                "variable": [
+                    "User_ID",
+                    "Cust_name",
+                    "Product_ID",
+                    "Gender",
+                    "Age Group",
+                    "Age",
+                    "Marital_Status",
+                    "State",
+                    "Zone",
+                    "Occupation",
+                    "Product_Category",
+                    "Orders",
+                    "Amount"
+                ],
+                "class": [
+                    "double",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "double",
+                    "double",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "double",
+                    "double"
+                ],
+                "description": [
+                    "User identification number",
+                    "Customer name",
+                    "Product identification number",
+                    "Gender of the customer (e.g. Male, Female)",
+                    "Age group of the customer",
+                    "Age of the customer",
+                    "Marital status of the customer (e.g. Married, Single)",
+                    "State of the customer",
+                    "Geographic zone of the customer",
+                    "Occupation of the customer",
+                    "Category of the product",
+                    "Number of orders made by the customer",
+                    "Amount in Indian rupees spent by the customer"
+                ]
+            }
+        ],
+        "data": {
+            "file_name": [
+                "diwali_sales_data.csv"
+            ],
+            "file_url": [
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-11-14/diwali_sales_data.csv"
+            ]
+        },
+        "data_load": {
+            "file_name": [
+                "diwali_sales_data.csv"
+            ],
+            "file_url": [
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-11-14/diwali_sales_data.csv"
+            ]
+        }
+    },
+    {
+        "date_posted": "2023-12-12",
+        "project_name": "Holiday Movies",
+        "project_source": [
+            "https://networkdatascience.ceu.edu/article/2019-12-16/christmas-movies",
+            "https://developer.imdb.com/non-commercial-datasets/"
+        ],
+        "description": "Happy holidays! This week we're exploring \"holiday\" movies: movies with \"holiday\", \"Christmas\", \"Hanukkah\", or \"Kwanzaa\" (or variants thereof) in their title! The data this week comes from theInternet Movie Database. We don't have an article using exactly this dataset, but you might get inspiration from thisChristmas Moviesblog post by Milán Janosov at Central European University.",
+        "data_source_url": "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-12-12",
+        "data_dictionary": [
+            {
+                "variable": [
+                    "tconst",
+                    "title_type",
+                    "primary_title",
+                    "original_title",
+                    "year",
+                    "runtime_minutes",
+                    "genres",
+                    "simple_title",
+                    "average_rating",
+                    "num_votes",
+                    "christmas",
+                    "hanukkah",
+                    "kwanzaa",
+                    "holiday"
+                ],
+                "class": [
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "double",
+                    "double",
+                    "character",
+                    "character",
+                    "double",
+                    "double",
+                    "logical",
+                    "logical",
+                    "logical",
+                    "logical"
+                ],
+                "description": [
+                    "alphanumeric unique identifier of the title",
+                    "the type/format of the title (movie, video, or tvMovie)",
+                    "the more popular title / the title used by the filmmakers on promotional materials at the point of release",
+                    "original title, in the original language",
+                    "the release year of a title",
+                    "primary runtime of the title, in minutes",
+                    "includes up to three genres associated with the title (comma-delimited)",
+                    "the title in lowercase, with punctuation removed, for easier filtering and grouping",
+                    "weighted average of all the individual user ratings on IMDb",
+                    "number of votes the title has received on IMDb (titles with fewer than 10 votes were not included in this dataset)",
+                    "whether the title includes \\\"christmas\\\", \\\"xmas\\\", \\\"x mas\\\", etc",
+                    "whether the title includes \\\"hanukkah\\\", \\\"chanukah\\\", etc",
+                    "whether the title includes \\\"kwanzaa\\\"",
+                    "whether the title includes the word \\\"holiday\\\""
+                ]
+            },
+            {
+                "variable": [
+                    "tconst",
+                    "genres"
+                ],
+                "class": [
+                    "character",
+                    "character"
+                ],
+                "description": [
+                    "alphanumeric unique identifier of the title",
+                    "genres associated with the title, one row per genre"
+                ]
+            }
+        ],
+        "data": {
+            "file_name": [
+                "holiday_movie_genres.csv",
+                "holiday_movies.csv"
+            ],
+            "file_url": [
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-12-12/holiday_movie_genres.csv",
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-12-12/holiday_movies.csv"
+            ]
+        },
+        "data_load": {
+            "file_name": [
+                "holiday_movie_genres.csv",
+                "holiday_movies.csv"
+            ],
+            "file_url": [
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-12-12/holiday_movie_genres.csv",
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-12-12/holiday_movies.csv"
+            ]
+        }
+    },
+    {
+        "date_posted": "2024-02-13",
+        "project_name": "Valentine's Day Consumer Data",
+        "project_source": [
+            "https://github.com/rfordatascience/tidytuesday/tree/master/data/2022/2022-01-25",
+            "https://nrf.com/research-insights/holiday-data-and-trends/valentines-day/valentines-day-data-center",
+            "https://www.kaggle.com/datasets/infinator/happy-valentines-day-2022",
+            "https://github.com/rfordatascience/tidytuesday/tree/master/data/2022/2022-01-18"
+        ],
+        "description": "Happy Valentine's Day! This week we're exploringValentine's Day survey data. The National Retail Federation in the United States conducts surveys and has created aValentine's Day Data Centerso you can explore the data on how consumers celebrate. The NRF has surveyed consumers about how they plan to celebrate Valentine’s Day annually for over a decade. Take a deeper dive into the data from the last 10 years, and use the interactive charts to explore a demographic breakdown of total spending, average spending, types of gifts planned and spending per type of gift. The NRF has continued to collect data. The data for this week is from 2010 to 2022, as organized by Suraj Das for a Kaggle dataset. In the historical surveys gender was collected as only 'Men' and 'Women', which does not accurately include all genders. If you're looking for other Valentine's Day type datasets, check out previous datasets onchocolateorboard games(a good Valentine's Day activity!). Data was downloaded fromSunja aa Kaggle dataset. Data from historical_gift_trends_per_person_spending.csv, historical_spending_average_expected_spending.csv and historical_spending_percent_celebrating.csv were combined into historical_spending.csv. Data from planned_gifts_age.csv and spending_or_celebrating_age_1.csv were combined into gifts_age.csv. Data from planned_gifts_gender.csv and spending_or_celebrating_gender_1.csv were combined into gifts_gender.csv. Percentage signs and dollar signs were removed from all numerical values.",
+        "data_source_url": "https://github.com/rfordatascience/tidytuesday/blob/master/data/2024/2024-02-13",
+        "data_dictionary": [
+            {
+                "variable": [
+                    "Year",
+                    "PercentCelebrating",
+                    "PerPerson",
+                    "Candy",
+                    "Flowers",
+                    "Jewelry",
+                    "GreetingCards",
+                    "EveningOut",
+                    "Clothing",
+                    "GiftCards"
+                ],
+                "class": [
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double"
+                ],
+                "description": [
+                    "Year",
+                    "Percent of people celebrating Valentines Day",
+                    "Average amount each person is spending",
+                    "Average amount spending on candy",
+                    "Average amount spending on flowers",
+                    "Average amount spending on jewelry",
+                    "Average amount spending on greeting cards",
+                    "Average amount spending on an evening out",
+                    "Average amount spending on clothing",
+                    "Average amount spending on gift cards"
+                ]
+            },
+            {
+                "variable": [
+                    "Age",
+                    "SpendingCelebrating",
+                    "Candy",
+                    "Flowers",
+                    "Jewelry",
+                    "GreetingCards",
+                    "EveningOut",
+                    "Clothing",
+                    "GiftCards"
+                ],
+                "class": [
+                    "character",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double"
+                ],
+                "description": [
+                    "Age",
+                    "Percent spending money on or celebrating Valentines Day",
+                    "Average percent spending on candy",
+                    "Average percent spending on flowers",
+                    "Average percent spending on jewelry",
+                    "Average percent spending on greeting cards",
+                    "Average percent spending on an evening out",
+                    "Average percent spending on clothing",
+                    "Average percent spending on gift cards"
+                ]
+            },
+            {
+                "variable": [
+                    "Gender",
+                    "SpendingCelebrating",
+                    "Candy",
+                    "Flowers",
+                    "Jewelry",
+                    "GreetingCards",
+                    "EveningOut",
+                    "Clothing",
+                    "GiftCards"
+                ],
+                "class": [
+                    "character",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double",
+                    "double"
+                ],
+                "description": [
+                    "Gender only including Men or Women",
+                    "Percent spending money on or celebrating Valentines Day",
+                    "Average percent spending on candy",
+                    "Average percent spending on flowers",
+                    "Average percent spending on jewelry",
+                    "Average percent spending on greeting cards",
+                    "Average percent spending on an evening out",
+                    "Average percent spending on clothing",
+                    "Average percent spending on gift cards"
+                ]
+            }
+        ],
+        "data": {
+            "file_name": [
+                "gifts_age.csv",
+                "gifts_gender.csv",
+                "historical_spending.csv"
+            ],
+            "file_url": [
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2024/2024-02-13/gifts_age.csv",
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2024/2024-02-13/gifts_gender.csv",
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2024/2024-02-13/historical_spending.csv"
+            ]
+        },
+        "data_load": {
+            "file_name": [
+                "gifts_age.csv",
+                "gifts_gender.csv",
+                "historical_spending.csv"
+            ],
+            "file_url": [
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2024/2024-02-13/gifts_age.csv",
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2024/2024-02-13/gifts_gender.csv",
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2024/2024-02-13/historical_spending.csv"
+            ]
+        }
+    },
+    {
+        "date_posted": "2023-08-08",
+        "project_name": "Hot Ones Episodes",
+        "project_source": [
+            "https://en.wikipedia.org/wiki/List_of_Hot_Ones_episodes",
+            "https://github.com/borstell",
+            "https://github.com/rfordatascience/tidytuesday/issues/591",
+            "https://en.wikipedia.org/wiki/Hot_Ones"
+        ],
+        "description": "The data this week comes from Wikipedia articles:Hot OnesandList of Hot Ones episodes. Thank you toCarl Börstellfor thesuggestion and cleaning script! Hot Ones is an American YouTube talk show, created by Chris Schonberger, hosted by Sean Evans and produced by First We Feast and Complex Media. Its basic premise involves celebrities being interviewed by Evans over a platter of increasingly spicy chicken wings.",
+        "data_source_url": "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-08-08",
+        "data_dictionary": [
+            {
+                "variable": [
+                    "season",
+                    "episode_overall",
+                    "episode_season",
+                    "title",
+                    "original_release",
+                    "guest",
+                    "guest_appearance_number",
+                    "finished"
+                ],
+                "class": [
+                    "integer",
+                    "integer",
+                    "integer",
+                    "character",
+                    "date",
+                    "character",
+                    "integer",
+                    "logical"
+                ],
+                "description": [
+                    "The season number.",
+                    "The overall count of this episode, from 1-300.",
+                    "The count of this episode within this season.",
+                    "The title of the episode.",
+                    "The date on which the episode was originally available on YouTube.",
+                    "The name of the guest.",
+                    "The number of appearances by this guest so far as of this date.",
+                    "Whether the guest finished trying all of the sauces."
+                ]
+            },
+            {
+                "variable": [
+                    "season",
+                    "sauce_number",
+                    "sauce_name",
+                    "scoville"
+                ],
+                "class": [
+                    "integer",
+                    "integer",
+                    "character",
+                    "integer"
+                ],
+                "description": [
+                    "The season number.",
+                    "The number of this sauce, from 1 (least hot) to 10 (hottest).",
+                    "The name of the sauce.",
+                    "The rating of the sauce in Scoville heat units."
+                ]
+            },
+            {
+                "variable": [
+                    "season",
+                    "episodes",
+                    "note",
+                    "original_release",
+                    "last_release"
+                ],
+                "class": [
+                    "integer",
+                    "integer",
+                    "character",
+                    "date",
+                    "date"
+                ],
+                "description": [
+                    "The season number.",
+                    "The count of episodes in this season.",
+                    "Notes about this season.",
+                    "The date of the first episode in this season.",
+                    "The date of the last episode of this season (if that episode has aired at the time of scraping)."
+                ]
+            }
+        ],
+        "data": {
+            "file_name": [
+                "episodes.csv",
+                "sauces.csv",
+                "seasons.csv"
+            ],
+            "file_url": [
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-08-08/episodes.csv",
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-08-08/sauces.csv",
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-08-08/seasons.csv"
+            ]
+        },
+        "data_load": {
+            "file_name": [
+                "episodes.csv",
+                "sauces.csv",
+                "seasons.csv"
+            ],
+            "file_url": [
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-08-08/episodes.csv",
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-08-08/sauces.csv",
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-08-08/seasons.csv"
+            ]
+        }
+    },
+    {
+        "date_posted": "2023-07-25",
+        "project_name": "Scurvy",
+        "project_source": [
+            "https://github.com/higgi13425/medicaldata/tree/master/data-raw",
+            "https://htmlpreview.github.io/?https://github.com/higgi13425/medicaldata/blob/master/man/description_docs/scurvy_desc.html",
+            "https://higgi13425.github.io/medicaldata/"
+        ],
+        "description": "The data this week comes from themedicaldata R package. This is a data package from Peter Higgins, with 19 medical datasets for teaching Reproducible Medical Research with R. We're using thescurvy dataset. Source: This data set is from a study published in 1757 in A Treatise on the Scurvy in Three Parts, by James Lind. This data set contains 12 participants with scurvy. In 1757, it was not known that scurvy is a manifestation of vitamin C deficiency. A variety of remedies had been anecdotally reported, but Lind was the first to test different regimens of acidic substances (including citrus fruits) against each other in a randomized, controlled trial. 6 distinct therapies were tested in 12 seamen with symptomatic scurvy, who were selected for similar severity. Six days of therapy were provided, and endpoints were reported in the text at the end of 6 days. These include rotting of the gums, skin sores, weakness of the knees, and lassitude, which are described in terms of severity. These have been translated into Likert scales from 0(none) to 3(severe). A dichotomous endpoint, fitness for duty, was also reported. Scurvy was a common affliction of seamen on long voyages, leading to mouth sores, skin lesions, weakness of the knees, and lassitude. Scurvy could be fatal on long voyages. James Lind reported the treatment of 12 seamen with scurvy in 1757, in _A Treatise on the Scurvy in Three Parts). This 476 page bloviation can be found scanned to the Google Books website A Treatise on the Scurvy. Pages 149-153 are a rare gem among what can be generously described as 400+ pages of evidence-free blathering, and these 4 pages may represent the first report of a controlled clinical trial. Lind was the ship’s surgeon on board the HMS Salisbury, and had a number of scurvy-affected seamen at his disposal. Many remedies had been described and advocated for, with no more than anecdotal evidence. On May 20, 1747, Lind decided to try the 6 therapies on the Salisbury in a comparative study in 12 affected seamen. He selected 12 with roughly similar severity, with notable skin and mouth sores, weakness of the knees, and significant lassitude, making them unfit for duty. They each received the standard shipboard diet of gruel and mutton broth, supplemented with occasional biscuits and puddings. Each treatment was a dietary supplement (including citrus fruits) or a medicinal. This data frame was reconstructed from Lind’s account as recorded on these 4 pages, with his estimates of severity translated to a 4 point Likert scale (0-3) for each of the symptoms he described at his chosen endpoint on day 6. A somewhat fanciful study_id variable was added, along with detailed descriptions of the dosing schedule of each treatment. Of note, there is some dispute about whether this was truly the first clinical trial, or whether it actually happened, as there are no contemporaneous corroborating accounts. See link about the historical debate. Lind reported that the seamen treated with 2 lemons and an orange daily did best, followed by those treated with cider. Those treated with elixir of vitriol only had improvement in mouth sores. One imagines that acidic substances (like dilute sulfuric acid, vinegar, cider, and citrus fruits) might have been rather painful on these mouth sores. Unfortunately, the burial of the 4 valuable pages of data in 476 pages of noise, a publication delay of 10 years, and Lind’s half-hearted conclusions (he was focused on acidity), meant that it took until 1795 before the British Navy mandated daily limes for seamen. The first column was removed from the scurvy.csv file available athttps://github.com/higgi13425/medicaldata/tree/master/data-raw.",
+        "data_source_url": "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-07-25",
+        "data_dictionary": [
+            {
+                "variable": [
+                    "study_id",
+                    "treatment",
+                    "dosing_regimen_for_scurvy",
+                    "gum_rot_d6",
+                    "skin_sores_d6",
+                    "weakness_of_the_knees_d6",
+                    "lassitude_d6",
+                    "fit_for_duty_d6"
+                ],
+                "class": [
+                    "double",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "character",
+                    "character"
+                ],
+                "description": [
+                    "Participant ID",
+                    "Treatment; cider, dilute_sulfuric_acid, vinegar, sea_water, citrus, purgative_mixture",
+                    "Dosing Regimen; 1 quart per day; 25 drops of elixir of vitriol, three times a day; two spoonfuls, three times daily; half pint daily; two lemons and an orange daily; a nutmeg-sized paste of garlic, mustard seed, horseradish, balsam of Peru, and gum myrrh three times a day",
+                    "Gum Rot on Day 6; 0_none, 1_mild, 2_moderate, 3_severe",
+                    "Skin Sores on Day 6; 0_none, 1_mild, 2_moderate, 3_severe",
+                    "Weakness of the Knees on Day 6; 0_none, 1_mild, 2_moderate, 3_severe",
+                    "Lassitude on Day 6; 0_none, 1_mild, 2_moderate, 3_severe",
+                    "Fit for Duty on Day 6; 0_no, 1_yes"
+                ]
+            }
+        ],
+        "data": {
+            "file_name": [
+                "scurvy.csv"
+            ],
+            "file_url": [
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-07-25/scurvy.csv"
+            ]
+        },
+        "data_load": {
+            "file_name": [
+                "scurvy.csv"
+            ],
+            "file_url": [
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-07-25/scurvy.csv"
+            ]
+        }
+    },
+    {
+        "date_posted": "2023-11-07",
+        "project_name": "US House Election Results",
+        "project_source": [
+            "https://electionlab.mit.edu/",
+            "https://electionlab.mit.edu/articles/new-report-how-we-voted-2022",
+            "https://docs.posit.co/ide/user/ide/guide/tools/copilot.html",
+            "https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/IG0UN2"
+        ],
+        "description": "It's election day in the United States! To celebrate, the data this week comes from theMIT Election Data and Science Lab(MEDSL). Hat tip this week to theRStudio GitHub Copilot integration, which suggested the MEDSL. From the MEDSL's reportNew Report: How We Voted in 2022: The Survey of the Performance of American Elections (SPAE) provides information about how Americans experienced voting in the most recent federal election. The survey has been conducted after federal elections since 2008, and is the only public opinion project in the country that is dedicated explicitly to understanding how voters themselves experience the election process. We're specifically providing data on House elections from 1976-2022. Check out theMEDSL websitefor additional datasets and tools. Be sure to cite the MEDSL in your work: Clean data and dictionary downloaded from theHarvard Dataverse",
+        "data_source_url": "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-11-07",
+        "data_dictionary": [
+            {
+                "variable": [
+                    "year",
+                    "state",
+                    "state_po",
+                    "state_fips",
+                    "state_cen",
+                    "state_ic",
+                    "office",
+                    "district",
+                    "stage",
+                    "runoff",
+                    "special",
+                    "candidate",
+                    "party",
+                    "writein",
+                    "mode",
+                    "candidatevotes",
+                    "totalvotes",
+                    "unofficial",
+                    "version",
+                    "fusion_ticket"
+                ],
+                "class": [
+                    "double",
+                    "character",
+                    "character",
+                    "double",
+                    "double",
+                    "double",
+                    "character",
+                    "character",
+                    "character",
+                    "logical",
+                    "logical",
+                    "character",
+                    "character",
+                    "logical",
+                    "character",
+                    "double",
+                    "double",
+                    "logical",
+                    "double",
+                    "logical"
+                ],
+                "description": [
+                    "year in which election was held",
+                    "state name",
+                    "U.S. postal code state abbreviation",
+                    "State FIPS code",
+                    "U.S. Census state code",
+                    "ICPSR state code",
+                    "U.S. House (constant)",
+                    "district number. At-large districts are coded as 0 (zero)",
+                    "electoral stage (gen = general elections, pri = primary elections)",
+                    "runoff election",
+                    "special election",
+                    "name of the candidate as it appears in the House Clerk report",
+                    "party of the candidate (always entirely lowercase) (Parties are as they appear in the House Clerk report. In states that allow candidates to appear on multiple party lines, separate vote totals are indicated for each party. Therefore, for analysis that involves candidate totals, it will be necessary to aggregate across all party lines within a district. For analysis that focuses on two-party vote totals, it will be necessary to account for major party candidates who receive votes under multiple party labels. Minnesota party labels are given as they appear on the Minnesota ballots. Future versions of this file will include codes for candidates who are endorsed by major parties, regardless of the party label under which they receive votes.)",
+                    "vote totals associated with write-in candidates",
+                    "mode of voting; states with data that doesn't break down returns by mode are marked as \\\"total\\\"",
+                    "votes received by this candidate for this particular party",
+                    "total number of votes cast for this election",
+                    "TRUE/FALSE indicator for unofficial result (to be updated later); this appears only for 2018 data in some cases",
+                    "date when this dataset was finalized",
+                    "A TRUE/FALSE indicator as to whether the given candidate is running on a fusion party ticket, which will in turn mean that a candidate will appear multiple times, but by different parties, for a given election. States with fusion tickets include Connecticut, New Jersey, New York, and South Carolina."
+                ]
+            }
+        ],
+        "data": {
+            "file_name": [
+                "house.csv"
+            ],
+            "file_url": [
+                "https://github.com/rfordatascience/tidytuesday/blob/master/data/2023/2023-11-07/house.csv"
+            ]
+        },
+        "data_load": {
+            "file_name": [
+                "house.csv"
+            ],
+            "file_url": [
+                "https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2023/2023-11-07/house.csv"
+            ]
+        }
+    }
+]

data/variables.csv

@@ -0,0 +1,411 @@
+dataset_id,dataset,variable_id,variable,type,description
+1,tournaments,1,key_id,integer,The unique ID number for the observation.
+1,tournaments,2,tournament_id,text,"The unique ID number for the tournament. Has the format {WC-####}, where the number is the year of the tournament."
+1,tournaments,3,tournament_name,text,The name of the tournament.
+1,tournaments,4,year,integer,The year of the tournament.
+1,tournaments,5,start_date,date,The start date of the tournament in the format {YYYY-MM-DD}.
+1,tournaments,6,end_date,date,The end date of the tournament in the format {YYYY-MM-DD}.
+1,tournaments,7,host_country,text,The unique ID number for the country that hosted the tournament. References {team_id} in the {teams} dataset.
+1,tournaments,8,winner,text,The name of the team that won the tournament.
+1,tournaments,9,host_won,boolean,Whether one of the host countries won the tournament. Coded {1} if one of the host countries won and {0} otherwise.
+1,tournaments,10,count_teams,integer,The number of teams that participated in the tournament.
+1,tournaments,11,group_stage,boolean,Whether the match is a group stage match. Coded {1} if the match is a group stage match and {0} otherwise.
+1,tournaments,12,second_group_stage,boolean,Whether there was a second group stage. Coded {1} if there was a second group stage and {0} otherwise.
+1,tournaments,13,final_round,boolean,Whether there was a final round. Coded {1} if there was a final round and {0} otherwise.
+1,tournaments,14,round_of_16,boolean,Whether there was a round of 16 stage. Coded {1} if there was a round of 16 stage and {0} otherwise.
+1,tournaments,15,quarter_finals,boolean,Whether there was a quarter-finals stage. Coded {1} if there was a quarter-finals stage and {0} otherwise.
+1,tournaments,16,semi_finals,boolean,Whether there was a semi-finals stage. Coded {1} if there was a semi-finals stage and {0} otherwise.
+1,tournaments,17,third_place_match,boolean,Whether there was a third-place match. Coded {1} if there was a third-place match and {0} otherwise.
+1,tournaments,18,final,boolean,Whether there was a final match. Coded {1} if there was a final match and {0} otherwise.
+2,confederations,1,key_id,integer,The unique ID number for the observation.
+2,confederations,2,confederation_id,text,"The unique ID number for the confederation. Has the format {CF-#}, where the number is a counter that is assigned with the confederations sorted in alphabetical order."
+2,confederations,3,confederation_name,text,The name of the confederation.
+2,confederations,4,confederation_code,text,The abbreviation for the confederation.
+2,confederations,5,confederation_wikipedia_link,text,The Wikipedia link for the confederation.
+3,teams,1,key_id,integer,The unique ID number for the observation.
+3,teams,2,team_id,text,"The unique ID number for the team. Has the format {T-##}, where the number is a counter that is assigned with the data sorted by the year of the team's first tournament and then by the team's name."
+3,teams,3,team_name,text,The name of the team.
+3,teams,4,team_code,text,The 3-letter code for the team.
+,teams,5,mens_team,boolean,Whether the country's men's team has qualified for a tournament.
+,teams,6,womens_team,boolean,Whether the country's women's team has qualified for a tournament.
+3,teams,7,federation_name,text,The name of the team's federation.
+3,teams,8,region_name,text,The name of the region that the country is located in.
+3,teams,9,confederation_id,text,The unique ID number for the confederation. References {confederation_id} in the {confederations} dataset.
+3,teams,10,confederation_name,text,The name of the confederation.
+3,teams,11,confederation_code,text,The abbreviation for the confederation.
+3,teams,12,mens_team_wikipedia_link,text,The Wikipedia link for country's men's team. Coded {not applicable} if the country's men's team has not qualified for a tournament.
+,teams,13,womens_team_wikipedia_link,text,The Wikipedia link for country's women's team. Coded {not applicable} if the country's women's team has not qualified for a tournament.
+3,teams,14,federation_wikipedia_link,text,The Wikipedia link of the team's federation.
+4,players,1,key_id,integer,The unique ID number for the observation.
+4,players,2,player_id,text,"The unique ID number for the player. Has the format {P-#####}, where the number is a randomly drawn, uniquely identifying number."
+4,players,3,family_name,text,The family name of the player.
+4,players,4,given_name,text,The given name of the player.
+4,players,5,birth_date,date,The birth date of the player in the format {YYYY-MM-DD}.
+,players,6,female,boolean,Whether the player is female. Coded {1} if the player is female and {0} if the player is male.
+4,players,7,goal_keeper,boolean,Whether the player was a goal keeper. Coded {1} if the player was a goal keeper and {0} otherwise.
+4,players,8,defender,boolean,Whether the player was a defender. Coded {1} if the player was a defender and {0} otherwise.
+4,players,9,midfielder,boolean,Whether the player was a midfielder. Coded {1} if the player was a midfielder and {0} otherwise.
+4,players,10,forward,boolean,Whether the player was a forward. Coded {1} if the player was a forward and {0} otherwise.
+4,players,11,count_tournaments,integer,The number of tournaments that the player participated in.
+4,players,12,list_tournaments,text,"A list of tournaments that the player participated in, separated by a comma."
+4,players,13,player_wikipedia_link,text,The name of the team of the player.
+5,managers,1,key_id,integer,The unique ID number for the observation.
+5,managers,2,manager_id,text,"The unique ID number for the manager. Has the format {M-####}, where the number is a counter that is assigned with the data sorted by the year of the manager's first appearance, then by the manager's family name, and then by the manager's given name."
+5,managers,3,family_name,text,The family name of the manager.
+5,managers,4,given_name,text,The given name of the manager.
+,managers,5,female,boolean,Whether the manager is female. Coded {1} if the manager is female and {0} if the manager is male.
+5,managers,6,country_name,text,The name of the manager's home country.
+5,managers,7,manager_wikipedia_link,text,The Wikipedia link for the manager.
+6,referees,1,key_id,integer,The unique ID number for the observation.
+6,referees,2,referee_id,text,"The unique ID number for the referee. Has the format {R-####}, where the number is a counter that is assigned with the data sorted by the year of the referee's first appearance, then by the referee's family name, and then by the referee's given name."
+6,referees,3,family_name,text,The family name of the referee.
+6,referees,4,given_name,text,The given name of the referee.
+,referees,5,female,boolean,Whether the referee is female. Coded {1} if the referee is female and {0} if the referee is male.
+6,referees,6,country_name,text,The name of the referee's home country.
+6,referees,7,confederation_id,text,The unique ID number for the confederation. References {confederation_id} in the {confederations} dataset.
+6,referees,8,confederation_name,text,The name of the confederation.
+6,referees,9,confederation_code,text,The abbreviation for the confederation.
+6,referees,10,referee_wikipedia_link,text,The Wikipedia link for the referee.
+7,stadiums,1,key_id,integer,The unique ID number for the observation.
+7,stadiums,2,stadium_id,text,"The unique ID number for the stadium. Has the format {S-###}, where the number is a count that is assigned with the data sorted by country, then by city, then by the name of the stadium."
+7,stadiums,3,stadium_name,text,The name of the stadium.
+7,stadiums,4,city_name,text,The city in which the match was played.
+7,stadiums,5,country_name,text,The name of the country in which the stadium is located.
+7,stadiums,6,stadium_capacity,integer,The approximate capacity of the stadium.
+7,stadiums,7,stadium_wikipedia_link,text,The Wikipedia link for the stadium.
+7,stadiums,8,city_wikipedia_link,text,The Wikipedia link for the city in which the match was played.
+8,matches,1,key_id,integer,The unique ID number for the observation.
+8,matches,2,tournament_id,text,The unique ID number for the tournament. References {tournament_id} in the {tournaments} dataset.
+8,matches,3,tournament_name,text,The name of the tournament.
+8,matches,4,match_id,text,"The unique ID number for the match. Has the format {M-####-##}, where the first number is the year of the tournament and the second number is a within-tournament counter that is assigned with the data sorted by the date of the match, then by the time of the match, then by the name of the group, and then by name of the home team."
+8,matches,5,match_name,text,The name of the match. 
+8,matches,6,stage_name,enum,"The stage of the tournament in which the match occurred. The possible values are: {first round}, {second round}, {group stage}, {round of sixteen}, {quarter-finals}, {semi-finals}, {third place match}, {final}. Note that not all values are applicable to all tournaments."
+8,matches,7,group_name,text,The name of the group.
+8,matches,8,group_stage,boolean,Whether the match is a group stage match. Coded {1} if the match is a group stage match and {0} otherwise.
+8,matches,9,knockout_stage,boolean,Whether the match is a knockout stage match. Coded {1} if the match is a knockout stage match and {0} otherwise.
+8,matches,10,replayed,boolean,Whether the match was replayed. Coded {1} if the match was replayed and {0} otherwise.
+8,matches,11,replay,boolean,Whether the match was a replay. Coded {1} if the match was a replay and {0} otherwise.
+8,matches,12,match_date,date,The date of the match in the format {YYYY-MM-DD}.
+8,matches,13,match_time,integer,The time of the match in the format {HH:MM}.
+8,matches,14,stadium_id,text,The unique ID number for the stadium. References {stadium_id} in the {stadiums} dataset.
+8,matches,15,stadium_name,text,The name of the stadium.
+8,matches,16,city_name,text,The city in which the match was played.
+8,matches,17,country_name,text,The name of the country in which the match was played.
+8,matches,18,home_team_id,text,The unique ID number for the home team. References {team_id} in the {teams} dataset.
+8,matches,19,home_team_name,text,The name of the home team. See the {teams} dataset.
+8,matches,20,home_team_code,text,The 3-letter code for the home team.
+8,matches,21,away_team_id,text,The unique ID number for the away team. References {team_id} in the {teams} dataset.
+8,matches,22,away_team_name,text,The name of the away team. See the {teams} dataset.
+8,matches,23,away_team_code,text,The 3-letter code for the away team.
+8,matches,24,score,text,"The score of the match in the format {#-#}, where the first number is the score of the home team and the second number is the score of the away team."
+8,matches,25,home_team_score,integer,The score of the home team.
+8,matches,26,away_team_score,integer,The score of the away team.
+8,matches,27,home_team_score_margin,integer,The score margin for the home team.
+8,matches,28,away_team_score_margin,integer,The score margin for the away team.
+8,matches,29,extra_time,boolean,Whether the match went to extra time. Coded {1} if the match went to extra time and {0} otherwise.
+8,matches,30,penalty_shootout,boolean,Whether the match ended in a penalty shootout. Coded {1} if the match ended in a penalty shootout and {0} otherwise.
+8,matches,31,score_penalties,text,The score of the penalty shootout in the format {#-#}. Coded {0-0} if there was not a penalty shootout.
+8,matches,32,home_team_score_penalties,integer,The score of the home team in the penalty shootout. Coded {NA} if there was not a penalty shootout.
+8,matches,33,away_team_score_penalties,integer,The score of the away team in the penalty shootout. Coded {NA} if there was not a penalty shootout.
+8,matches,34,result,enum,"The result of the match. The possible values are: {home team win}, {away team win}, {draw}, {replayed}."
+8,matches,35,home_team_win,boolean,Whether the home team won the match. Coded {1} if the home team won the match and {0} otherwise.
+8,matches,36,away_team_win,boolean,Whether the home team won the match. Coded {1} if the home team won the match and {0} otherwise.
+8,matches,37,draw,boolean,Whether the match ended in a draw. Coded {1} of the match ended in a draw and {0} otherwise.
+9,awards,1,key_id,integer,The unique ID number for the observation.
+9,awards,2,award_id,text,"The unique ID number for the award. Has the format {A-#}, where the number is a counter."
+9,awards,3,award_name,enum,"The name of the award. The possible values are: {Golden Ball}, {Silver Ball}, {Bronze Ball}, {Golden Boot}, {Silver Boot}, {Bronze Boot}, {Golden Glove}, {Best Young Player}. "
+9,awards,4,award_description,text,A description of the award.
+9,awards,5,year_introduced,integer,The year the award was first introduced.
+10,qualified_teams,1,key_id,integer,The unique ID number for the observation.
+10,qualified_teams,2,tournament_id,text,The unique ID number for the tournament. References {tournament_id} in the {tournaments} dataset.
+10,qualified_teams,3,tournament_name,text,The name of the tournament.
+10,qualified_teams,4,team_id,text,The unique ID number for the team. References {team_id} in the {teams} dataset.
+10,qualified_teams,5,team_name,text,The name of the team.
+10,qualified_teams,6,team_code,text,The 3-letter code for the team.
+10,qualified_teams,7,count_matches,integer,The number of matches that the team played in the tournament.
+10,qualified_teams,8,performance,text,The furthest stage of the tournament reached by the team.
+11,squads,1,key_id,integer,The unique ID number for the observation.
+11,squads,2,tournament_id,text,The unique ID number for the tournament. References {tournament_id} in the {tournaments} dataset.
+11,squads,3,tournament_name,text,The name of the tournament.
+11,squads,4,team_id,text,The unique ID number for the team. References {team_id} in the {teams} dataset.
+11,squads,5,team_name,text,The name of the team of the player.
+11,squads,6,team_code,text,The 3-letter code for the team.
+11,squads,7,player_id,text,The unique ID number for the player. References {player_id} in the {players} dataset.
+11,squads,8,family_name,text,The family name of the player.
+11,squads,9,given_name,text,The given name of the player.
+11,squads,10,shirt_number,integer,The shirt number of the player.
+11,squads,11,position_name,enum,"The position of the player. The possible values are: {goal keeper}, {defender}, {midfielder}, {forward}."
+11,squads,12,position_code,enum,"The code for the position of the player. The possible values are: {GK}, {DF}, {MF}, {FW}."
+12,manager_appointments,1,key_id,integer,The unique ID number for the observation.
+12,manager_appointments,2,tournament_id,text,The unique ID number for the tournament. References {tournament_id} in the {tournaments} dataset.
+12,manager_appointments,3,tournament_name,text,The name of the tournament.
+12,manager_appointments,4,team_id,text,The unique ID number for the team. References {team_id} in the {teams} dataset.
+12,manager_appointments,5,team_name,text,The name of the team of the manager.
+12,manager_appointments,6,team_code,text,The 3-letter code for the team.
+12,manager_appointments,7,manager_id,text,The unique ID number for the manager. References {manager_id} in the {managers} dataset.
+12,manager_appointments,8,family_name,text,The family name of the manager.
+12,manager_appointments,9,given_name,text,The given name of the manager.
+12,manager_appointments,10,country_name,text,The name of the manager's home country.
+13,referee_appointments,1,key_id,integer,The unique ID number for the observation.
+13,referee_appointments,2,tournament_id,text,The unique ID number for the tournament. References {tournament_id} in the {tournaments} dataset.
+13,referee_appointments,3,tournament_name,text,The name of the tournament.
+13,referee_appointments,4,referee_id,text,The unique ID number for the referee. References {referee_id} in the {referees} dataset.
+13,referee_appointments,5,family_name,text,The family name of the referee.
+13,referee_appointments,6,given_name,text,The given name fo the referee.
+13,referee_appointments,7,country_name,text,The name of the referee's home country.
+13,referee_appointments,8,confederation_id,text,The unique ID number for the confederation. References {confederation_id} in the {confederations} dataset.
+13,referee_appointments,9,confederation_name,text,The name of the confederation.
+13,referee_appointments,10,confederation_code,text,The abbreviation for the confederation.
+14,team_appearances,1,key_id,integer,The unique ID number for the observation.
+14,team_appearances,2,tournament_id,text,The unique ID number for the tournament. References {tournament_id} in the {tournaments} dataset.
+14,team_appearances,3,tournament_name,text,The name of the tournament.
+14,team_appearances,4,match_id,text,The unique ID number for the match. References {match_id} in the {matches} dataset.
+14,team_appearances,5,match_name,text,The name of the match.
+14,team_appearances,6,stage_name,enum,"The stage of the tournament in which the match occurred. The possible values are: {first round}, {second round}, {group stage}, {round of sixteen}, {quarter-finals}, {semi-finals}, {third place match}, {final}. Note that not all values are applicable to all tournaments."
+14,team_appearances,7,group_name,text,The name of the group.
+14,team_appearances,8,group_stage,boolean,Whether the match is a group stage match. Coded {1} if the match is a group stage match and {0} otherwise.
+14,team_appearances,9,knockout_stage,boolean,Whether the match is a knockout stage match. Coded {1} if the match is a knockout stage match and {0} otherwise.
+14,team_appearances,10,replayed,boolean,Whether the match was replayed. Coded {1} if the match was replayed and {0} otherwise.
+14,team_appearances,11,replay,boolean,Whether the match was a replay. Coded {1} if the match was a replay and {0} otherwise.
+14,team_appearances,12,match_date,date,The date of the match in the format {YYYY-MM-DD}.
+14,team_appearances,13,match_time,integer,The time of the match in the format {HH:MM}.
+14,team_appearances,14,stadium_id,text,The unique ID number for the stadium. References {stadium_id} in the {stadiums} dataset.
+14,team_appearances,15,stadium_name,text,The name of the stadium.
+14,team_appearances,16,city_name,text,The city in which the match was played.
+14,team_appearances,17,country_name,text,The name of the country in which the match was played.
+14,team_appearances,18,team_id,text,The unique ID number for the team. References {team_id} in the {teams} dataset.
+14,team_appearances,19,team_name,text,The name of the team.
+14,team_appearances,20,team_code,text,The 3-letter code for the team.
+14,team_appearances,21,opponent_id,text,The unique ID number for the team's opponent. References {team_id} in the {teams} dataset.
+14,team_appearances,22,opponent_name,text,The name of the team's opponent.
+14,team_appearances,23,opponent_code,text,The 3-letter code for the team's opponent.
+14,team_appearances,24,home_team,boolean,Whether the team was the home team. Coded {1} if the team was the home team and {0} otherwise.
+14,team_appearances,25,away_team,boolean,Whether the team was the away team. Coded {1} if the team was the away team and {0} otherwise.
+14,team_appearances,26,goals_for,integer,The number of goals scored by the team.
+14,team_appearances,27,goals_against,integer,The number of goals scored against the team.
+14,team_appearances,28,goal_differential,integer,The team's goal differential.
+14,team_appearances,29,extra_time,boolean,Whether the match went to extra time. Coded {1} if the match went to extra time and {0} otherwise.
+14,team_appearances,30,penalty_shootout,boolean,Whether the match ended in a penalty shootout. Coded {1} if the match ended in a penalty shootout and {0} otherwise.
+14,team_appearances,31,penalties_for,integer,"The number of penalties scored by the opponent, if the match ended in a penalty shootout. Coded {0} if there was not a shootout."
+14,team_appearances,32,penalties_against,integer,"The number of penalties scored by the team, if the match ended in a penalty shootout. Coded {0} if there was not a shootout."
+14,team_appearances,33,result,enum,"The result of the match. The possible values are: {home team win}, {away team win}, {draw}, {replayed}."
+14,team_appearances,34,win,boolean,Whether the team won the match. Coded {1} if the team won the match and {0} otherwise.
+14,team_appearances,35,lose,boolean,Whether the team lost the match. Coded {1} if the team lost the match and {0} otherwise.
+14,team_appearances,36,draw,boolean,Whether the match ended in a draw. Coded {1} of the match ended in a draw and {0} otherwise.
+15,player_appearances,1,key_id,integer,The unique ID number for the observation.
+15,player_appearances,2,tournament_id,text,The unique ID number for the tournament. References {tournament_id} in the {tournaments} dataset.
+15,player_appearances,3,tournament_name,text,The name of the tournament.
+15,player_appearances,4,match_id,text,The unique ID number for the match. References {match_id} in the {matches} dataset.
+15,player_appearances,5,match_name,text,The name of the match.
+15,player_appearances,6,match_date,date,The date of the match in the format {YYYY-MM-DD}.
+15,player_appearances,7,stage_name,enum,"The stage of the tournament in which the match occurred. The possible values are: {first round}, {second round}, {group stage}, {round of sixteen}, {quarter-finals}, {semi-finals}, {third place match}, {final}. Note that not all values are applicable to all tournaments."
+15,player_appearances,8,group_name,text,The name of the group.
+15,player_appearances,9,team_id,text,The unique ID number for the team of the player. References {team_id} in the {teams} dataset.
+15,player_appearances,10,team_name,text,The name of the team of the player.
+15,player_appearances,11,team_code,text,The 3-letter code for the team of the player.
+15,player_appearances,12,home_team,boolean,Whether the team was the home team. Coded {1} if the team was the home team and {0} otherwise.
+15,player_appearances,13,away_team,boolean,Whether the team was the away team. Coded {1} if the team was the away team and {0} otherwise.
+15,player_appearances,14,player_id,text,The unique ID number for the player. References {player_id} in the {players} dataset.
+15,player_appearances,15,family_name,text,The family name of the player.
+15,player_appearances,16,given_name,text,The given name of the player.
+15,player_appearances,17,shirt_number,integer,The shirt number of the player.
+15,player_appearances,18,position_name,text,The name of the position of the player.
+15,player_appearances,19,position_code,text,A 2-letter or 3-letter code that indicates the position of the player.
+15,player_appearances,20,starter,boolean,Whether the player started the match. Coded {1} if the player started the match and {0} otherwise.
+15,player_appearances,21,substitute,boolean,Whether the player was a substitute. Coded {1} if the player was a substitute and {0} otherwise.
+16,manager_appearances,1,key_id,integer,The unique ID number for the observation.
+16,manager_appearances,2,tournament_id,text,The unique ID number for the tournament. References {tournament_id} in the {tournaments} dataset.
+16,manager_appearances,3,tournament_name,text,The name of the tournament.
+16,manager_appearances,4,match_id,text,The unique ID number for the match. References {match_id} in the {matches} dataset.
+16,manager_appearances,5,match_name,text,The name of the match. 
+16,manager_appearances,6,match_date,date,The date of the match in the format {YYYY-MM-DD}.
+16,manager_appearances,7,stage_name,enum,"The stage of the tournament in which the match occurred. The possible values are: {first round}, {second round}, {group stage}, {round of sixteen}, {quarter-finals}, {semi-finals}, {third place match}, {final}. Note that not all values are applicable to all tournaments."
+16,manager_appearances,8,group_name,text,The name of the group.
+16,manager_appearances,9,team_id,text,The unique ID number for the team of the manager. References {team_id} in the {teams} dataset.
+16,manager_appearances,10,team_name,text,The name of the team of the manager.
+16,manager_appearances,11,team_code,text,The 3-letter code for the team of the manager.
+16,manager_appearances,12,home_team,boolean,Whether the team was the home team. Coded {1} if the team was the home team and {0} otherwise.
+16,manager_appearances,13,away_team,boolean,Whether the team was the away team. Coded {1} if the team was the away team and {0} otherwise.
+16,manager_appearances,14,manager_id,text,The unique ID number for the manager. References {manager_id} in the {managers} dataset.
+16,manager_appearances,15,family_name,text,The family name of the manager.
+16,manager_appearances,16,given_name,text,The given name of the manager.
+16,manager_appearances,17,country_name,text,The name of the manager's home country.
+17,referee_appearances,1,key_id,integer,The unique ID number for the observation.
+17,referee_appearances,2,tournament_id,text,The unique ID number for the tournament. References {tournament_id} in the {tournaments} dataset.
+17,referee_appearances,3,tournament_name,text,The name of the tournament.
+17,referee_appearances,4,match_id,text,The unique ID number for the match. References {match_id} in the {matches} dataset.
+17,referee_appearances,5,match_name,text,The name of the match.
+17,referee_appearances,6,match_date,date,The date of the match in the format {YYYY-MM-DD}.
+17,referee_appearances,7,stage_name,enum,"The stage of the tournament in which the match occurred. The possible values are: {first round}, {second round}, {group stage}, {round of sixteen}, {quarter-finals}, {semi-finals}, {third place match}, {final}. Note that not all values are applicable to all tournaments."
+17,referee_appearances,8,group_name,text,The name of the group.
+17,referee_appearances,9,referee_id,text,The unique ID number for the referee. References {referee_id} in the {referees} dataset.
+17,referee_appearances,10,family_name,text,The family name of the referee.
+17,referee_appearances,11,given_name,text,The given name of the referee.
+17,referee_appearances,12,country_name,text,The name of the referee's home country.
+17,referee_appearances,13,confederation_id,text,The unique ID number for the confederation. References {confederation_id} in the {confederations} dataset.
+17,referee_appearances,14,confederation_name,text,The name of the confederation.
+17,referee_appearances,15,confederation_code,text,The abbreviation for the confederation.
+18,goals,1,key_id,integer,The unique ID number for the observation.
+18,goals,2,goal_id,text,"The unique ID number for the goal. Has the format {G-####}, where the number is a counter that is assigned with the data sorted by the match ID, then the minute of the goal."
+18,goals,3,tournament_id,text,The unique ID number for the tournament. References {tournament_id} in the {tournaments} dataset.
+18,goals,4,tournament_name,text,The name of the tournament.
+18,goals,5,match_id,text,The unique ID number for the match in which the goal occurred. References {match_id} in the {matches} dataset.
+18,goals,6,match_name,text,The name of the match in which the goal occurred.
+18,goals,7,match_date,date,The date of the match in the format {YYYY-MM-DD}.
+18,goals,8,stage_name,enum,"The stage of the tournament in which the match occurred. The possible values are: {first round}, {second round}, {group stage}, {round of sixteen}, {quarter-finals}, {semi-finals}, {third place match}, {final}. Note that not all values are applicable to all tournaments."
+18,goals,9,group_name,text,The name of the group.
+18,goals,10,team_id,text,"The unique ID number for the team that scored the goal. References {team_id} in the {teams} dataset. For own goals, this is the team that is awarded the goal, not the team of the player who scored the own goal."
+18,goals,11,team_name,text,The name of the team of the player who scored the goal.
+18,goals,12,team_code,text,The 3-letter code for the team of the player who scored the goal.
+18,goals,13,home_team,boolean,Whether the team was the home team. Coded {1} if the team was the home team and {0} otherwise.
+18,goals,14,away_team,boolean,Whether the team was the away team. Coded {1} if the team was the away team and {0} otherwise.
+18,goals,15,player_id,text,The unique ID number for the player who scored the goal. References {player_id} in the {players} dataset.
+18,goals,16,family_name,text,The family name of the player who scored the goal.
+18,goals,17,given_name,text,The given name of the player who scored the goal.
+18,goals,18,shirt_number,integer,The shirt number of the player who scored the goal.
+18,goals,19,player_team_id,text,"The unique ID number for the team of the player who scored the goal. References {team_id} in the {teams} dataset. For own goals, this is the team of the player who scored the own goal, not the team that is awarded the goal."
+18,goals,20,player_team_name,text,The name of the team of the player who scored the goal.
+18,goals,21,player_team_code,text,The 3-letter code for the team of the player who scored the goal.
+18,goals,22,minute_label,text,The minute of the match in which the goal occurred in the format {#'} or {#'+#'}.
+18,goals,23,minute_regulation,integer,The minute of regulation time in which the substitution occurred.
+18,goals,24,minute_stoppage,integer,The minute of stoppage time in which the goal occurred. Coded {0} if the substitution did not occur during stoppage time.
+18,goals,25,match_period,enum,"The period of the match in which the goal occurred. The possible values are: {first half}, {first half, stoppage time}, {second half}, {second half, stoppage time}, {extra time, first half}, {extra time, first half, stoppage time}, {extra time, second half}, {extra time, second half, stoppage time}, {after extra time}."
+18,goals,26,own_goal,boolean,Whether the goal was an own goal. Coded {1} if the goal was an own goal and {0} otherwise.
+18,goals,27,penalty,boolean,"Whether the goal was a penalty that occurred during the game, as opposed to during a penalty shootout. Coded {1} if the goal was a penalty that occurred during the game and {0} otherwise."
+19,penalty_kicks,1,key_id,integer,The unique ID number for the observation.
+19,penalty_kicks,2,penalty_kick_id,text,"The unique ID number for the penalty kick. Has the format {PK-####}, where the number is a counter that is assigned with the data sorted by the match ID, then the minute of the penalty kick."
+19,penalty_kicks,3,tournament_id,text,The unique ID number for the tournament. References {tournament_id} in the {tournaments} dataset.
+19,penalty_kicks,4,tournament_name,text,The name of the tournament.
+19,penalty_kicks,5,match_id,text,The unique ID number for the match in which the penalty kick occurred. References {match_id} in the {matches} dataset.
+19,penalty_kicks,6,match_name,text,The name of match in which the penalty kick occurred.
+19,penalty_kicks,7,match_date,date,The date of the match in the format {YYYY-MM-DD}.
+19,penalty_kicks,8,stage_name,enum,"The stage of the tournament in which the match occurred. The possible values are: {first round}, {second round}, {group stage}, {round of sixteen}, {quarter-finals}, {semi-finals}, {third place match}, {final}. Note that not all values are applicable to all tournaments."
+19,penalty_kicks,9,group_name,text,The name of the group.
+19,penalty_kicks,10,team_id,text,The unique ID number for the team of the player who took the penalty kick. References {team_id} in the {teams} dataset.
+19,penalty_kicks,11,team_name,text,The name of the team of the player who took the penalty kick.
+19,penalty_kicks,12,team_code,text,The 3-letter code for the team of the player who took the penalty kick.
+19,penalty_kicks,13,home_team,boolean,Whether the team was the home team. Coded {1} if the team was the home team and {0} otherwise.
+19,penalty_kicks,14,away_team,boolean,Whether the team was the away team. Coded {1} if the team was the away team and {0} otherwise.
+19,penalty_kicks,15,player_id,text,The unique ID number for the player who took the penalty kick. References {player_id} in the {players} dataset.
+19,penalty_kicks,16,family_name,text,The family name of the player who took the penalty kick.
+19,penalty_kicks,17,given_name,text,The given name of the player who took the penalty kick.
+19,penalty_kicks,18,shirt_number,integer,The shirt number of the player who took the penalty kick. 
+19,penalty_kicks,19,converted,boolean,Whether the penalty kick was converted. Coded {1} if the penalty kick was converted and {0} otherwise.
+20,bookings,1,key_id,integer,The unique ID number for the observation.
+20,bookings,2,booking_id,text,"The unique ID number for the booking. Has the format {B-####}, where the number is a counter that is assigned with the data sorted by the match ID, then the minute of the booking."
+20,bookings,3,tournament_id,text,The unique ID number for the tournament. References {tournament_id} in the {tournaments} dataset.
+20,bookings,4,tournament_name,text,The name of the tournament.
+20,bookings,5,match_id,text,The unique ID number for the match in which the booking occurred. References {match_id} in the {matches} dataset.
+20,bookings,6,match_name,text,The name of the match in which the booking occurred.
+20,bookings,7,match_date,date,The date of the match in the format {YYYY-MM-DD}.
+20,bookings,8,stage_name,enum,"The stage of the tournament in which the match occurred. The possible values are: {first round}, {second round}, {group stage}, {round of sixteen}, {quarter-finals}, {semi-finals}, {third place match}, {final}. Note that not all values are applicable to all tournaments."
+20,bookings,9,group_name,text,The name of the group.
+20,bookings,10,team_id,text,The unique ID number for the team of the player who was booked. References {team_id} in the {teams} dataset.
+20,bookings,11,team_name,text,The name of the team of the player who was booked.
+20,bookings,12,team_code,text,The 3-letter code for the team of the player who was booked.
+20,bookings,13,home_team,boolean,Whether the team was the home team. Coded {1} if the team was the home team and {0} otherwise.
+20,bookings,14,away_team,boolean,Whether the team was the away team. Coded {1} if the team was the away team and {0} otherwise.
+20,bookings,15,player_id,text,The unique ID number for the player who was booked. References {player_id} in the {players} dataset.
+20,bookings,16,family_name,text,The family name of the player who was booked.
+20,bookings,17,given_name,text,The given name of the player who was booked.
+20,bookings,18,shirt_number,integer,The shirt number of the player who was booked.
+20,bookings,19,minute_label,text,The minute of the match in which the booking occurred in the format {#'} or {#'+#'}.
+20,bookings,20,minute_regulation,integer,The minute of regulation time in which the booking occurred.
+20,bookings,21,minute_stoppage,integer,The minute of stoppage time in which the booking occurred. Coded {0} if the substitution did not occur during stoppage time.
+20,bookings,22,match_period,enum,"The period of the match in which the booking occurred. The possible values are: {first half}, {first half, stoppage time}, {second half}, {second half, stoppage time}, {extra time, first half}, {extra time, first half, stoppage time}, {extra time, second half}, {extra time, second half, stoppage time}, {after extra time}."
+20,bookings,23,yellow_card,boolean,Whether the booking was a yellow card. Coded {1} if the card is a yellow card and {0} otherwise.
+20,bookings,24,red_card,boolean,Whether the booking was a red card. Coded {1} if the card is a red card and {0} otherwise.
+20,bookings,25,second_yellow_card,boolean,Whether the booking was a second yellow card. Coded {1} if the booking is a second yellow and {0} otherwise.
+20,bookings,26,sending_off,boolean,Whether the booking resulted in the player being sent off. Coded {1} if the player was sent off and {0} otherwise.
+21,substitutions,1,key_id,integer,The unique ID number for the observation.
+21,substitutions,2,substitution_id,text,"The unique ID number for the substitution. Has the format {S-####}, where the number is a counter that is assigned with the data sorted by the match ID, then the minute of the substitution, then whether the player is going off."
+21,substitutions,3,tournament_id,text,The unique ID number for the tournament. References {tournament_id} in the {tournaments} dataset.
+21,substitutions,4,tournament_name,text,The name of the tournament.
+21,substitutions,5,match_id,text,The unique ID number for the match in which the substitution occurred. References {match_id} in the {matches} dataset.
+21,substitutions,6,match_name,text,The name of the match in which the substitution occurred.
+21,substitutions,7,match_date,date,The date of the match in the format {YYYY-MM-DD}.
+21,substitutions,8,stage_name,enum,"The stage of the tournament in which the match occurred. The possible values are: {first round}, {second round}, {group stage}, {round of sixteen}, {quarter-finals}, {semi-finals}, {third place match}, {final}. Note that not all values are applicable to all tournaments."
+21,substitutions,9,group_name,text,The name of the group.
+21,substitutions,10,team_id,text,The unique ID number for the team of the player who was substituted. References {team_id} in the {teams} dataset.
+21,substitutions,11,team_name,text,The name of the team of the player who was substituted.
+21,substitutions,12,team_code,text,The 3-letter code for the team of the player who was substituted.
+21,substitutions,13,home_team,boolean,Whether the team was the home team. Coded {1} if the team was the home team and {0} otherwise.
+21,substitutions,14,away_team,boolean,Whether the team was the away team. Coded {1} if the team was the away team and {0} otherwise.
+21,substitutions,15,player_id,text,The unique ID number for the player who was substituted. References {player_id} in the {players} dataset.
+21,substitutions,16,family_name,text,The family name of the player who was substituted.
+21,substitutions,17,given_name,text,The given name of the player who was substituted.
+21,substitutions,18,shirt_number,integer,The shirt number of the player who was substituted.
+21,substitutions,19,minute_label,text,The minute of the match in which the substitution occurred in the format {#'} or {#'+#'}.
+21,substitutions,20,minute_regulation,integer,The minute of regulation time in which the substitution occurred.
+21,substitutions,21,minute_stoppage,integer,The minute of stoppage time in which the substitution occurred. Coded {0} if the substitution did not occur during stoppage time.
+21,substitutions,22,match_period,enum,"The period of the match in which the substitution occurred. The possible values are: {first half}, {first half, stoppage time}, {second half}, {second half, stoppage time}, {extra time, first half}, {extra time, first half, stoppage time}, {extra time, second half}, {extra time, second half, stoppage time}, {after extra time}."
+21,substitutions,23,going_off,boolean,Whether the player was going off the field. Coded {1} if the player was going off and {0} otherwise.
+21,substitutions,24,coming_on,boolean,Whether the player was coming on the field. Coded {1} if the player was coming on and {0} otherwise.
+22,host_countries,1,key_id,integer,The unique ID number for the observation.
+22,host_countries,2,tournament_id,text,The unique ID number for the tournament. References {tournament_id} in the {tournaments} dataset.
+22,host_countries,3,tournament_name,text,The name of the tournament.
+22,host_countries,4,team_id,text,The unique ID number for the team. References {team_id} in the {teams} dataset.
+22,host_countries,5,team_name,text,The name of the team.
+22,host_countries,6,team_code,text,The 3-letter code for the team.
+22,host_countries,7,performance,text,The furthest stage of the tournament reached by the host country's team.
+23,tournament_stages,1,key_id,integer,The unique ID number for the observation.
+23,tournament_stages,2,tournament_id,text,The unique ID number for the tournament. References {tournament_id} in the {tournaments} dataset.
+23,tournament_stages,3,tournament_name,text,The name of the tournament.
+23,tournament_stages,4,stage_number,integer,The number of the stage.
+23,tournament_stages,5,stage_name,enum,"The stage of the tournament in which the match occurred. The possible values are: {first round}, {second round}, {group stage}, {round of sixteen}, {quarter-finals}, {semi-finals}, {third place match}, {final}. Note that not all values are applicable to all tournaments."
+23,tournament_stages,6,group_stage,boolean,Whether the match is a group stage match. Coded {1} if the match is a group stage match and {0} otherwise.
+23,tournament_stages,7,knockout_stage,boolean,Whether there was a knockout stage. Coded {1} if there was a knockout stage and {0} otherwise.
+23,tournament_stages,8,unbalanced_groups,boolean,Whether there were unbalanced groups. Coded {1} if there were unbalanced groups and {0} otherwise.
+23,tournament_stages,9,start_date,date,The start date of the stage in the format {YYYY-MM-DD}.
+23,tournament_stages,10,end_date,date,The end date of the stage in the format {YYYY-MM-DD}.
+23,tournament_stages,11,count_matches,integer,The number of matches in the stage.
+23,tournament_stages,12,count_teams,integer,The number of teams that participated in the stage.
+23,tournament_stages,13,count_scheduled,integer,The number of games that were scheduled in the stage.
+23,tournament_stages,14,count_replays,integer,The number of replays in the stage.
+23,tournament_stages,15,count_playoffs,integer,The number of playoff games in the stage.
+23,tournament_stages,16,count_walkovers,integer,The number of walkovers in the stage.
+24,groups,1,key_id,integer,The unique ID number for the observation.
+24,groups,2,tournament_id,text,The unique ID number for the tournament. References {tournament_id} in the {tournaments} dataset.
+24,groups,3,tournament_name,text,The name of the tournament.
+24,groups,4,stage_number,integer,The number of the stage.
+24,groups,5,stage_name,enum,"The stage of the tournament in which the match occurred. The possible values are: {first round}, {second round}, {group stage}, {round of sixteen}, {quarter-finals}, {semi-finals}, {third place match}, {final}. Note that not all values are applicable to all tournaments."
+24,groups,6,group_name,text,The name of the group.
+24,groups,7,count_teams,integer,The number of teams in the group.
+25,group_standings,1,key_id,integer,The unique ID number for the observation.
+25,group_standings,2,tournament_id,text,The unique ID number for the tournament. References {tournament_id} in the {tournaments} dataset.
+25,group_standings,3,tournament_name,text,The name of the tournament.
+25,group_standings,4,stage_number,integer,The number of the stage.
+25,group_standings,5,stage_name,enum,"The stage of the tournament in which the match occurred. The possible values are: {first round}, {second round}, {group stage}, {round of sixteen}, {quarter-finals}, {semi-finals}, {third place match}, {final}. Note that not all values are applicable to all tournaments."
+25,group_standings,6,group_name,text,The name of the group.
+25,group_standings,7,position,integer,The team's final position in the group.
+25,group_standings,8,team_id,text,The unique ID number for the team. References {team_id} in the {teams} dataset.
+25,group_standings,9,team_name,text,The name of the team.
+25,group_standings,10,team_code,text,The 3-letter code for the team.
+25,group_standings,11,played,integer,The number of matches that the team played in the group.
+25,group_standings,12,wins,integer,The number of matches that the team won in the group stage.
+25,group_standings,13,draws,integer,The number of matches that the team drew in the group stage.
+25,group_standings,14,losses,integer,The number of matches that the team lost in the group stage.
+25,group_standings,15,goals_for,integer,The number of goals scored by the team in the group stage.
+25,group_standings,16,goals_against,integer,The number of goals scored against the team in the group stage.
+25,group_standings,17,goal_difference,integer,The team's goal difference in the group stage.
+25,group_standings,18,points,integer,The number of points that the team earned in the group.
+25,group_standings,19,advanced,boolean,Whether the team advanced out of the group. Coded {1} if the team advanced and {0} otherwise.
+26,tournament_standings,1,key_id,integer,The unique ID number for the observation.
+26,tournament_standings,2,tournament_id,text,The unique ID number for the tournament. References {tournament_id} in the {tournaments} dataset.
+26,tournament_standings,3,tournament_name,text,The name of the tournament.
+26,tournament_standings,4,position,integer,The place of the team in the final standings.
+26,tournament_standings,5,team_id,text,The unique ID number for the team. References {team_id} in the {teams} dataset.
+26,tournament_standings,6,team_name,text,The name of the team.
+26,tournament_standings,7,team_code,text,The 3-letter code for the team.
+27,award_winners,1,key_id,integer,The unique ID number for the observation.
+27,award_winners,2,tournament_id,text,The unique ID number for the tournament. References {tournament_id} in the {tournaments} dataset.
+27,award_winners,3,tournament_name,text,The name of the tournament.
+27,award_winners,4,award_id,text,The unique ID number for the award. References {award_id} in the {awards} dataset.
+27,award_winners,5,award_name,enum,"The name of the award. The possible values are: {Golden Ball}, {Silver Ball}, {Bronze Ball}, {Golden Boot}, {Silver Boot}, {Bronze Boot}, {Golden Glove}, {Best Young Player}. "
+27,award_winners,6,shared,boolean,Whether the award was shared between multiple players. Coded {1} if the award was shared and {0} otherwise.
+27,award_winners,7,player_id,text,The unique ID number for the player who won the award. References {player_id} in the {players} dataset.
+27,award_winners,8,family_name,text,The family name of the player who won the award.
+27,award_winners,9,given_name,text,The given name of the player who won the award.
+27,award_winners,10,team_id,text,The unique ID number for the team. References {team_id} in the {teams} dataset.
+27,award_winners,11,team_name,text,The name of the team of the player who won the award.
+27,award_winners,12,team_code,text,The 3-letter code for the team of the player who won the award.

hierarchy_eval.py

@@ -0,0 +1,524 @@
+"""
+hierarchy_eval.py — shared, reference-free hierarchy evaluation for the TFM.
+
+WHY REFERENCE-FREE?
+-------------------
+In all three approaches the dataset's group column is a *construction input*
+(Gonçalves text object in baseline / Approach 1; explicit group-anchored L1/L2
+in Approach 2).  An input cannot also serve as the ground truth — measuring the
+hierarchy against the group column is therefore circular (for Approach 2 it is
+circular by design).  The defensible evaluation is reference-free.
+
+PRIMARY METRICS (no gold standard required)
+-------------------------------------------
+  • Parent–child coherence   — TraCo (Wu et al., AAAI 2024, arXiv:2401.14113)
+  • Sibling diversity        — TraCo (same paper)
+  • NPMI label coherence     — Lau et al., EACL 2014 (aclanthology.org/E14-1056);
+                               orig. Mimno et al., EMNLP 2010
+  • Structural statistics    — HiExpan-style reporting (Shen et al., KDD 2018)
+
+SECONDARY (descriptive, explicitly caveated)
+--------------------------------------------
+  • Group-structure preservation (NMI / ARI / Purity vs the group column).
+    Reported only as "how much the discovered hierarchy still reflects the
+    pre-existing group column that was used as input" — NOT an accuracy metric.
+
+All metrics are computed the same way for every approach, so cross-approach
+comparison is fair.
+"""
+from __future__ import annotations
+
+import re
+from collections import Counter
+
+import numpy as np
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Tree helpers
+# ──────────────────────────────────────────────────────────────────────────────
+def build_parent_map(nodes: list) -> dict:
+    pm: dict = {}
+    for n in nodes:
+        for c in n.get('related', []):
+            cid = int(c)
+            if cid not in pm:
+                pm[cid] = int(n['id'])
+    return pm
+
+
+def structural_stats(nodes: list) -> dict:
+    pm = build_parent_map(nodes)
+
+    def depth_of(nid: int) -> int:
+        d = 0
+        while nid in pm:
+            nid = pm[nid]; d += 1
+        return d
+
+    agg   = [n for n in nodes if n.get('type') == 'aggregation']
+    leafs = [n for n in nodes if n.get('type') == 'attribute']
+    depths   = [depth_of(int(n['id'])) for n in leafs]
+    branches = [len(n.get('related', [])) for n in agg]
+    singletons = sum(1 for b in branches if b == 1)
+    return {
+        'n_aggregation_nodes':  len(agg),
+        'max_depth':            int(max(depths, default=0)),
+        'avg_leaf_depth':       round(float(np.mean(depths)), 2) if depths else 0.0,
+        'avg_branching_factor': round(float(np.mean(branches)), 2) if branches else 0.0,
+        'singleton_nodes_%':    round(100.0 * singletons / max(len(agg), 1), 1),
+    }
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Encoder — SBERT if available, TF-IDF fallback.  Loaded once, reused.
+# ──────────────────────────────────────────────────────────────────────────────
+_SBERT = None
+_SBERT_TRIED = False
+
+
+def _get_sbert():
+    global _SBERT, _SBERT_TRIED
+    if _SBERT_TRIED:
+        return _SBERT
+    _SBERT_TRIED = True
+    try:
+        from sentence_transformers import SentenceTransformer
+        _SBERT = SentenceTransformer('all-MiniLM-L6-v2')
+    except Exception:
+        _SBERT = None
+    return _SBERT
+
+
+def encode(texts: list):
+    """Return (unit-normalised vectors, backend_name)."""
+    texts = [str(t) if str(t).strip() else '_' for t in texts]
+    model = _get_sbert()
+    if model is not None:
+        v = model.encode(texts, normalize_embeddings=True, show_progress_bar=False)
+        return np.asarray(v, dtype=float), 'SBERT (all-MiniLM-L6-v2)'
+    from sklearn.feature_extraction.text import TfidfVectorizer
+    X = TfidfVectorizer(stop_words='english', max_features=2000,
+                        min_df=1).fit_transform(texts).toarray().astype(float)
+    norms = np.linalg.norm(X, axis=1, keepdims=True)
+    return X / np.where(norms == 0, 1.0, norms), 'TF-IDF (SBERT unavailable)'
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# TraCo reference-free metrics  (Wu et al., AAAI 2024)
+# ──────────���───────────────────────────────────────────────────────────────────
+def traco_metrics(nodes: list) -> dict:
+    """Parent–child coherence and sibling diversity over node *labels*."""
+    usable = [n for n in nodes if n.get('type') in ('aggregation', 'attribute')]
+    if len(usable) < 2:
+        return {'pc_coherence': 0.0, 'sibling_diversity': 0.0, 'encoder': 'n/a'}
+
+    ids    = [int(n['id']) for n in usable]
+    labels = [str(n.get('name', '')) for n in usable]
+    vecs, backend = encode(labels)
+    id2v = {i: vecs[k] for k, i in enumerate(ids)}
+
+    pc_sims, sib_divs = [], []
+    for n in nodes:
+        if n.get('type') == 'root':
+            continue
+        pid = int(n['id'])
+        if pid not in id2v:
+            continue
+        children = [int(c) for c in n.get('related', []) if int(c) in id2v]
+        for cid in children:
+            pc_sims.append(float(np.dot(id2v[pid], id2v[cid])))
+        if len(children) >= 2:
+            cv = np.array([id2v[c] for c in children])
+            S  = cv @ cv.T
+            nc = len(children)
+            divs = [1.0 - float(S[i, j]) for i in range(nc) for j in range(i + 1, nc)]
+            sib_divs.append(float(np.mean(divs)))
+
+    return {
+        'pc_coherence':      round(float(np.mean(pc_sims)),  4) if pc_sims  else 0.0,
+        'sibling_diversity': round(float(np.mean(sib_divs)), 4) if sib_divs else 0.0,
+        'encoder':           backend,
+    }
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# NPMI label coherence  (Lau et al., EACL 2014; Mimno et al., EMNLP 2010)
+# Reference corpus = the variable descriptions themselves.
+# ──────────────────────────────────────────────────────────────────────────────
+_TOKEN_RE = re.compile(r'[a-z][a-z]{2,}')
+_STOP = set(
+    'the a an and or of to in for on with by at from as is are be this that these '
+    'those it its was were has have had not no than then so such can will may '
+    'group description name label value type using used per each'.split()
+)
+
+
+def _tokens(text: str) -> set:
+    return {w for w in _TOKEN_RE.findall(str(text).lower()) if w not in _STOP}
+
+
+def npmi_coherence(nodes: list, corpus_texts: list, topn: int = 5) -> float:
+    """Average NPMI of each aggregation node's label terms over the corpus.
+
+    Returns a value in roughly [-1, 1]; higher = node labels use term
+    combinations that genuinely co-occur in the data (meaningful, not random).
+    """
+    docs = [_tokens(t) for t in corpus_texts]
+    docs = [d for d in docs if d]
+    N = len(docs)
+    if N < 2:
+        return 0.0
+
+    df: Counter = Counter()
+    for d in docs:
+        for w in d:
+            df[w] += 1
+
+    # Collect the term sets we actually need (node labels)
+    label_termsets: list = []
+    needed_terms: set = set()
+    for n in nodes:
+        if n.get('type') != 'aggregation':
+            continue
+        terms = [w for w in _tokens(n.get('name', '')) if df.get(w, 0) > 0]
+        terms = sorted(terms, key=lambda w: df[w], reverse=True)[:topn]
+        if len(terms) >= 2:
+            label_termsets.append(terms)
+            needed_terms.update(terms)
+
+    if not label_termsets:
+        return 0.0
+
+    # Pair co-occurrence counts (only for needed pairs)
+    needed_pairs = set()
+    for terms in label_termsets:
+        for i in range(len(terms)):
+            for j in range(i + 1, len(terms)):
+                needed_pairs.add(frozenset((terms[i], terms[j])))
+
+    co: Counter = Counter()
+    for d in docs:
+        present = d & needed_terms
+        if len(present) < 2:
+            continue
+        pl = list(present)
+        for i in range(len(pl)):
+            for j in range(i + 1, len(pl)):
+                pair = frozenset((pl[i], pl[j]))
+                if pair in needed_pairs:
+                    co[pair] += 1
+
+    eps = 1e-12
+    node_scores: list = []
+    for terms in label_termsets:
+        pair_npmis: list = []
+        for i in range(len(terms)):
+            for j in range(i + 1, len(terms)):
+                wi, wj = terms[i], terms[j]
+                c_ij = co.get(frozenset((wi, wj)), 0)
+                p_ij = (c_ij + eps) / N
+                p_i  = df[wi] / N
+                p_j  = df[wj] / N
+                pmi  = np.log(p_ij / (p_i * p_j + eps) + eps)
+                npmi = pmi / (-np.log(p_ij + eps))
+                pair_npmis.append(float(npmi))
+        if pair_npmis:
+            node_scores.append(float(np.mean(pair_npmis)))
+
+    return round(float(np.mean(node_scores)), 4) if node_scores else 0.0
+
+
+# ─────────────��────────────────────────────────────────────────────────────────
+# Secondary (descriptive, caveated): group-structure preservation
+# ──────────────────────────────────────────────────────────────────────────────
+def _depth1_assignments(nodes: list, can) -> list:
+    pm = build_parent_map(nodes)
+
+    def depth1(nid: int) -> int:
+        while pm.get(nid, -1) not in (-1, 0):
+            nid = pm[nid]
+        return nid
+
+    lid_to_nid = {n['metadata']['leaf_id']: int(n['id'])
+                  for n in nodes if n.get('type') == 'attribute' and 'metadata' in n}
+    return [depth1(lid_to_nid[lid]) if lid in lid_to_nid else -1
+            for lid in can['_leaf_id']]
+
+
+def _purity(y_true, y_pred) -> float:
+    clusters: dict = {}
+    for t, p in zip(y_true, y_pred):
+        clusters.setdefault(p, []).append(t)
+    correct = sum(Counter(v).most_common(1)[0][1] for v in clusters.values())
+    return correct / max(len(y_true), 1)
+
+
+def group_preservation(nodes: list, can) -> dict:
+    """NMI / ARI / Purity of the depth-1 partition vs the group column.
+
+    CAVEAT: the group column is a construction input in every approach, so this
+    is a descriptive 'structure preservation' figure, NOT an accuracy metric.
+    """
+    from sklearn.metrics import normalized_mutual_info_score, adjusted_rand_score
+    from sklearn.preprocessing import LabelEncoder
+    import pandas as pd
+
+    y_true_raw = can['_group_path'].apply(
+        lambda x: str(x).split(' > ')[0].strip()
+        if pd.notna(x) and str(x) not in ('', 'nan') else 'Ungrouped'
+    ).tolist()
+    y_pred_raw = _depth1_assignments(nodes, can)
+
+    y_true = LabelEncoder().fit_transform(y_true_raw)
+    y_pred = LabelEncoder().fit_transform(y_pred_raw)
+    return {
+        'NMI':    round(float(normalized_mutual_info_score(
+                     y_true, y_pred, average_method='arithmetic')), 4),
+        'ARI':    round(float(adjusted_rand_score(y_true, y_pred)), 4),
+        'Purity': round(_purity(y_true_raw, y_pred_raw), 4),
+    }
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Gold-standard comparison — Edge-F1 / Ancestor-F1
+#
+# HiExpan (Shen et al., KDD 2018) scores a system taxonomy against a hand-built
+# gold taxonomy with Edge-F1 (direct parent–child links) and Ancestor-F1 (all
+# ancestor links).  Because our internal-node *labels* differ between the gold
+# tree and each system, we use the label-free leaf-pair formulation (the
+# pair-counting tradition, Fowlkes & Mallows 1983):
+#
+#   • Edge-F1     — over pairs of leaves that share the same IMMEDIATE parent
+#                   (i.e. they are siblings).  Strict: rewards correct granularity.
+#   • Ancestor-F1 — over pairs of leaves that share ANY non-root ancestor
+#                   (i.e. they are grouped together somewhere).  Lenient.
+#
+# Leaves are matched between gold and system by their attribute-node NAME (the
+# variable label) — the one field all three approaches expose for every leaf.
+# Only leaves present in BOTH the gold subset and the system tree are scored, so
+# a gold subset of 50–100 variables fairly evaluates a full hierarchy.
+# ──────────────────────────────────────────────────────────────────────────────
+def _pred_leaf_lineage(nodes: list) -> dict:
+    """leaf name → list of ancestor node ids (root-most first, excl. root & leaf)."""
+    pm = build_parent_map(nodes)
+    id_to_node = {int(n['id']): n for n in nodes}
+    lineage: dict = {}
+    for n in nodes:
+        if n.get('type') != 'attribute':
+            continue
+        name = str(n.get('name', ''))
+        cur  = int(n['id'])
+        anc, seen = [], set()
+        while cur in pm and cur not in seen:
+            seen.add(cur)
+            cur = pm[cur]
+            nd = id_to_node.get(cur)
+            if nd is None or nd.get('type') == 'root':
+                break
+            anc.append(cur)
+        anc.reverse()
+        lineage[name] = anc
+    return lineage
+
+
+def _gold_leaf_lineage(gold_df) -> dict:
+    """leaf name → list of cumulative path-prefix strings (the gold ancestors)."""
+    lineage: dict = {}
+    for _, r in gold_df.iterrows():
+        name = str(r['leaf_label'])
+        path = str(r.get('gold_path', '') or '')
+        comps = [c.strip() for c in path.split('>')
+                 if c.strip() and c.strip().lower() != 'ungrouped']
+        anc, pref = [], ''
+        for c in comps:
+            pref = c if not pref else f'{pref} > {c}'
+            anc.append(pref)
+        lineage[name] = anc
+    return lineage
+
+
+def _sibling_pairs(lineage: dict) -> set:
+    from collections import defaultdict
+    groups: dict = defaultdict(list)
+    for name, anc in lineage.items():
+        if anc:
+            groups[anc[-1]].append(name)
+    pairs: set = set()
+    for members in groups.values():
+        m = sorted(members)
+        for i in range(len(m)):
+            for j in range(i + 1, len(m)):
+                pairs.add((m[i], m[j]))
+    return pairs
+
+
+def _cogrouped_pairs(lineage: dict) -> set:
+    from collections import defaultdict
+    occ: dict = defaultdict(set)
+    for name, anc in lineage.items():
+        for a in anc:
+            occ[a].add(name)
+    pairs: set = set()
+    for members in occ.values():
+        m = sorted(members)
+        for i in range(len(m)):
+            for j in range(i + 1, len(m)):
+                pairs.add((m[i], m[j]))
+    return pairs
+
+
+def _prf(pred_set: set, gold_set: set) -> dict:
+    if not pred_set and not gold_set:
+        return {'precision': 1.0, 'recall': 1.0, 'f1': 1.0}
+    tp = len(pred_set & gold_set)
+    p = tp / len(pred_set) if pred_set else 0.0
+    r = tp / len(gold_set) if gold_set else 0.0
+    f = 2 * p * r / (p + r) if (p + r) > 0 else 0.0
+    return {'precision': round(p, 4), 'recall': round(r, 4), 'f1': round(f, 4)}
+
+
+def gold_comparison(nodes: list, gold_df) -> dict:
+    """Edge-F1 and Ancestor-F1 of a system tree vs a hand-built gold tree."""
+    pred = _pred_leaf_lineage(nodes)
+    gold = _gold_leaf_lineage(gold_df)
+    shared = set(pred) & set(gold)
+    pred = {k: v for k, v in pred.items() if k in shared}
+    gold = {k: v for k, v in gold.items() if k in shared}
+    return {
+        'n_matched_leaves': len(shared),
+        'edge_f1':     _prf(_sibling_pairs(pred),   _sibling_pairs(gold)),
+        'ancestor_f1': _prf(_cogrouped_pairs(pred), _cogrouped_pairs(gold)),
+    }
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# Granularity-tolerant, label-independent structural F1  (set-overlap matching)
+#
+# Edge-F1 punishes a system for adding *correct* extra depth, because two leaves
+# that gold lists as siblings stop being immediate siblings once the system
+# refines them into sub-tiers.  That makes edge-F1 unfair to deliberately deeper
+# trees (Approaches 1 & 2).  Set-overlap F1 fixes this: it matches each gold
+# cluster (the set of leaves under a gold path-prefix) to the system node whose
+# leaf set overlaps it most (Jaccard), regardless of that node's depth or label.
+#
+#   • precision — for each system aggregation node, its best Jaccard with any
+#                 gold cluster, averaged.  Low when the system invents groups
+#                 gold does not have (e.g. one node per delay value = over-split).
+#   • recall    — for each gold cluster, its best Jaccard with any system node,
+#                 averaged.  Low when the system fails to recover a gold group.
+#
+# This is the cluster-matching / overlap-F1 tradition (e.g. ontology alignment,
+# hierarchical-clustering evaluation).  Label-free, so it compares the three
+# approaches fairly even though their internal-node labels differ.
+# ──────────────────────────────────────────────────────────────────────────────
+def _system_clusters(nodes: list) -> list:
+    """Each aggregation node → frozenset of leaf NAMES in its subtree (size ≥ 2)."""
+    id_to_node = {int(n['id']): n for n in nodes}
+    out: list = []
+    for n in nodes:
+        if n.get('type') != 'aggregation':
+            continue
+        leaves: list = []
+        stack = [int(n['id'])]
+        seen: set = set()
+        while stack:
+            x = stack.pop()
+            if x in seen:
+                continue
+            seen.add(x)
+            nd = id_to_node.get(x)
+            if nd is None:
+                continue
+            if nd.get('type') == 'attribute':
+                leaves.append(str(nd.get('name', '')))
+            else:
+                stack.extend(int(c) for c in nd.get('related', []))
+        s = frozenset(leaves)
+        if len(s) >= 2:
+            out.append(s)
+    return out
+
+
+def _gold_clusters(gold_df) -> list:
+    """Each gold path-prefix → frozenset of leaf NAMES under it (size ≥ 2)."""
+    from collections import defaultdict
+    occ: dict = defaultdict(set)
+    for name, anc in _gold_leaf_lineage(gold_df).items():
+        for a in anc:
+            occ[a].add(name)
+    return [frozenset(v) for v in occ.values() if len(v) >= 2]
+
+
+def set_overlap_f1(nodes: list, gold_df) -> dict:
+    """Granularity-tolerant, label-free hierarchical F1 via best leaf-set Jaccard."""
+    pred_names = set(_pred_leaf_lineage(nodes))
+    gold_names = {str(x) for x in gold_df['leaf_label']}
+    shared = pred_names & gold_names
+    if len(shared) < 2:
+        return {'precision': 0.0, 'recall': 0.0, 'f1': 0.0}
+
+    sys_cl  = [c & shared for c in _system_clusters(nodes)]
+    sys_cl  = [c for c in sys_cl if len(c) >= 2]
+    gold_cl = [c & shared for c in _gold_clusters(gold_df)]
+    gold_cl = [c for c in gold_cl if len(c) >= 2]
+    if not sys_cl or not gold_cl:
+        return {'precision': 0.0, 'recall': 0.0, 'f1': 0.0}
+
+    def jac(a: frozenset, b: frozenset) -> float:
+        u = len(a | b)
+        return len(a & b) / u if u else 0.0
+
+    prec = float(np.mean([max(jac(s, g) for g in gold_cl) for s in sys_cl]))
+    rec  = float(np.mean([max(jac(s, g) for s in sys_cl) for g in gold_cl]))
+    f1   = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
+    return {'precision': round(prec, 4), 'recall': round(rec, 4), 'f1': round(f1, 4)}
+
+
+def refinement_breakdown(nodes: list, gold_df) -> dict:
+    """Decompose edge-F1 disagreements into harmless refinement vs real errors.
+
+    • wrong_merge_rate — system sibling pairs that gold does NOT co-group anywhere
+      (genuine mistakes: variables wrongly placed together).
+    • refinement_rate  — gold sibling pairs the system keeps co-grouped but at a
+      FINER level (split into sub-tiers).  These are deeper-but-consistent, the
+      thing edge-F1 unfairly penalises.
+    • missed_rate      — gold sibling pairs the system fails to co-group at all
+      (real recall failures).
+    """
+    pred = _pred_leaf_lineage(nodes)
+    gold = _gold_leaf_lineage(gold_df)
+    shared = set(pred) & set(gold)
+    pred = {k: v for k, v in pred.items() if k in shared}
+    gold = {k: v for k, v in gold.items() if k in shared}
+
+    sys_sib = _sibling_pairs(pred)
+    sys_cog = _cogrouped_pairs(pred)
+    gold_sib = _sibling_pairs(gold)
+    gold_cog = _cogrouped_pairs(gold)
+
+    wrong_merge = len(sys_sib - gold_cog)
+    refined     = len((gold_sib & sys_cog) - sys_sib)
+    missed      = len(gold_sib - sys_cog)
+    return {
+        'wrong_merge_rate': round(wrong_merge / len(sys_sib), 4) if sys_sib else 0.0,
+        'refinement_rate':  round(refined / len(gold_sib), 4) if gold_sib else 0.0,
+        'missed_rate':      round(missed / len(gold_sib), 4) if gold_sib else 0.0,
+    }
+
+
+# ──────────────────────────────────────────────────────────────────────────────
+# One-call bundle
+# ──────────────────────────────────────────────────────────────────────────────
+def evaluate(nodes: list, corpus_texts: list | None = None, can=None,
+             gold_df=None) -> dict:
+    """Compute the full metric bundle for one hierarchy."""
+    out: dict = {}
+    out.update(traco_metrics(nodes))
+    out['npmi_coherence'] = (npmi_coherence(nodes, corpus_texts)
+                             if corpus_texts is not None else None)
+    out.update({f'struct_{k}': v for k, v in structural_stats(nodes).items()})
+    if can is not None:
+        out['group_preservation'] = group_preservation(nodes, can)
+    if gold_df is not None:
+        out['gold'] = gold_comparison(nodes, gold_df)
+    return out

launcher.py

@@ -0,0 +1,137 @@
+"""
+launcher.py — start Baseline, Approach 1 and Approach 2 on different ports,
+              open them in browser tabs, and shut down all at once when you
+              press Enter.
+
+Usage:
+    python launcher.py
+
+Each app has its own file uploader — upload a different CSV to each tab to
+compare approaches side by side.
+"""
+
+from __future__ import annotations
+import socket
+import subprocess
+import sys
+import time
+import webbrowser
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+
+JOBS = [
+    ('baseline.py',   8501, 'Baseline'),
+    ('approach_1.py', 8502, 'Approach 1'),
+    ('approach_2.py', 8503, 'Approach 2'),
+]
+
+# TIP: to compare TWO datasets at once you do NOT need extra ports. Streamlit
+# gives every browser tab its own independent session (separate upload + state),
+# so just open the same URL twice — e.g. open http://localhost:8501 in two tabs,
+# load AI-MIND in one and HCP in the other.
+
+OPEN_BROWSER      = True
+STARTUP_WAIT_SECS = 5
+
+
+def _port_in_use(port: int) -> bool:
+    """Return True if something is already listening on this port."""
+    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
+        s.settimeout(0.5)
+        return s.connect_ex(('127.0.0.1', port)) == 0
+
+
+def _kill_tree(p: subprocess.Popen) -> None:
+    """Kill a process and all its children (works reliably on Windows and POSIX)."""
+    if sys.platform == 'win32':
+        subprocess.call(
+            ['taskkill', '/F', '/T', '/PID', str(p.pid)],
+            stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
+        )
+    else:
+        try:
+            import os, signal
+            os.killpg(os.getpgid(p.pid), signal.SIGTERM)
+        except Exception:
+            p.terminate()
+    try:
+        p.wait(timeout=5)
+    except subprocess.TimeoutExpired:
+        p.kill()
+
+
+def main() -> int:
+    # Validate scripts
+    missing = [s for s, _, _ in JOBS if not (HERE / s).is_file()]
+    if missing:
+        print(f'ERROR: missing files: {missing}')
+        return 1
+
+    # Abort if any port is already occupied — prevents the duplicate-tab problem
+    busy = [(label, port) for _, port, label in JOBS if _port_in_use(port)]
+    if busy:
+        for label, port in busy:
+            print(f'ERROR: port {port} ({label}) is already in use.')
+        print('\nKill the existing servers first (Task Manager → python.exe → End Task),')
+        print('then run launcher.py again.')
+        return 1
+
+    procs: list[subprocess.Popen] = []
+    print(f'Working directory: {HERE}')
+    print(f'Launching {len(JOBS)} Streamlit instance(s)…\n')
+
+    for script, port, label in JOBS:
+        cmd = [
+            sys.executable, '-m', 'streamlit', 'run', str(HERE / script),
+            '--server.port', str(port),
+            '--server.headless', 'true',       # suppress Streamlit's own browser open
+            '--browser.gatherUsageStats', 'false',
+        ]
+        try:
+            # Do NOT use CREATE_NEW_PROCESS_GROUP — it breaks taskkill /T
+            p = subprocess.Popen(cmd)
+            procs.append(p)
+            print(f'  ✓ {label:<12} pid={p.pid:<6} → http://localhost:{port}')
+        except Exception as e:
+            print(f'  ✗ FAILED {label}: {e}')
+
+    if not procs:
+        print('Nothing started.')
+        return 1
+
+    # Wait for each server to actually be reachable before opening the browser
+    print(f'\nWaiting for servers to come up (max {STARTUP_WAIT_SECS}s each)…')
+    for _, port, label in JOBS:
+        for _ in range(STARTUP_WAIT_SECS * 2):
+            if _port_in_use(port):
+                print(f'  ✓ {label} ready')
+                break
+            time.sleep(0.5)
+        else:
+            print(f'  ⚠ {label} did not respond in time — opening anyway')
+
+    if OPEN_BROWSER:
+        print('\nOpening browser tabs…')
+        for _, port, label in JOBS:
+            url = f'http://localhost:{port}'
+            webbrowser.open_new_tab(url)
+            print(f'  • {label}  →  {url}')
+            time.sleep(0.3)   # small gap so tabs open in order
+
+    print('\nAll servers running.')
+    print('Press Enter (in THIS terminal) to stop all servers and exit.\n')
+    try:
+        input()
+    except KeyboardInterrupt:
+        pass
+
+    print('\nStopping servers…')
+    for p in procs:
+        _kill_tree(p)
+    print('Done.')
+    return 0
+
+
+if __name__ == '__main__':
+    raise SystemExit(main())

requirements.txt

@@ -0,0 +1,17 @@
+streamlit>=1.30
+pandas>=2.0
+numpy>=1.24
+scikit-learn>=1.3
+plotly>=5.18
+sentence-transformers>=2.5
+requests>=2.31
+openpyxl>=3.1
+
+# Approach 2 — semantic aspect discovery (recent SOTA, NeurIPS 2024)
+# Optional but recommended.  Pulls torch as a transitive dependency.
+fastopic>=0.0.5
+
+# Approach 2 — local LLM label refinement (TopicTag-style, evidence-grounded).
+# Uses an Ollama server on localhost:11434 with the OpenAI-compatible /v1
+# endpoint.  See README for `ollama pull` instructions.
+openai>=1.30