---
title: leaderboard-analytics-service
emoji: 📊
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: "6.0.0"
python_version: "3.11"
app_file: app.py
pinned: false
---

# Leaderboard Analytics Metrics Spec

This project analyzes user behavior on the MTEB leaderboard page from event logs in MongoDB.

The primary purpose of this document is to define **what is measured**, **where each metric comes from**, and **how each metric is calculated**.

---

## Data Contract

All analytics are based on the `events` collection and the following stable fields:

- Core dimensions: `event_name`, `timestamp`, `session_id`
- Preferred event time: `ts` as a MongoDB Date
- Behavior context: `benchmark`, `filters`
- Visitor identity (approximate): `properties.visitor_id`
- Change context: `properties.old_value`, `properties.new_value`, `properties.filter_name`

Important event names:

- `page_view`
- `benchmark_change`
- `filter_change_`* (dynamic names, such as `filter_change_task_type`)
- `table_download` (currently may be missing in some deployments)

---

## Metrics Dictionary

### 1) PV (Page Views)

- **Definition**: Number of page view events.
- **Source fields**: `event_name`
- **Calculation**:
  - Filter events where `event_name == "page_view"`
  - PV = count of matched events

### 2) Sessions

- **Definition**: Number of unique interaction sessions.
- **Source fields**: `session_id`
- **Calculation**:
  - Sessions = count of distinct non-empty `session_id` values in the selected time range

### 3) UV (Unique Visitors, Approximate)

- **Definition**: Number of unique visitors identified by hashed fingerprint.
- **Source fields**: `properties.visitor_id`
- **Calculation**:
  - Remove null/empty `properties.visitor_id`
  - UV = count of distinct `properties.visitor_id` values in the selected time range

### 4) Sessions Per Visitor

- **Definition**: Average number of sessions per visitor.
- **Source fields**: derived from Sessions and UV
- **Calculation**:
  - Sessions Per Visitor = `Sessions / UV`
  - If UV is 0, result is 0

### 5) Session Depth (Events Per Session)

- **Definition**: Average interaction intensity per session.
- **Source fields**: all events, `session_id`
- **Calculation**:
  - Total Events = count of all events in range
  - Session Depth = `Total Events / Sessions`
  - If Sessions is 0, result is 0

---

## Behavior Metrics

### 6) Benchmark Popularity

- **Definition**: Frequency of selected benchmarks.
- **Source fields**: `event_name`, `properties.new_value`
- **Calculation**:
  - Filter `event_name == "benchmark_change"`
  - Group by `properties.new_value`
  - Popularity = event count per benchmark value

### 7) Filter Usage Distribution

- **Definition**: Usage volume by filter event type.
- **Source fields**: `event_name`
- **Calculation**:
  - Filter `event_name` matching regex `^filter_change_`
  - Group by `event_name`
  - Distribution = count per filter event

### 8) Filter Session Coverage

- **Definition**: Number of sessions that used each filter type.
- **Source fields**: `event_name`, `session_id`
- **Calculation**:
  - For each `filter_change_`* event type:
    - collect distinct non-empty `session_id`
    - coverage = distinct session count

---

## Funnel Metrics

Recommended session-level funnel:

1. `page_view`
2. `benchmark_change`
3. `filter_change_`*
4. `table_download`

### 9) Step Session Count

- **Definition**: Number of sessions that reached each ordered funnel step.
- **Source fields**: `session_id`, `event_name`, `ts` or `timestamp`
- **Calculation**:
  - Group events by `session_id`
  - Sort events by event time
  - Count each cumulative step only when it occurs after the previous required step

### 10) Step Conversion Rate

- **Definition**: Conversion from funnel step 1 (`page_view`) to each step.
- **Source fields**: derived from Step Session Count
- **Calculation**:
  - Conversion Rate(step N) = `StepN Sessions / Step1 Sessions * 100%`
  - If Step1 Sessions is 0, result is 0%

---

## Visitor Segmentation Metrics

### 11) New Visitors

- **Definition**: Visitors whose current period contains their first observed visit date.
- **Source fields**: `event_name`, `ts` or `timestamp`, `properties.visitor_id`
- **Calculation**:
  - Use `page_view` events only
  - For each `visitor_id`, find earliest timestamp (`first_seen`) from the full available dataset
  - If event date equals `first_seen` date, classify as `new`
  - Count distinct `visitor_id` by period

### 12) Returning Visitors

- **Definition**: Visitors seen after their first observed date.
- **Source fields**: same as New Visitors
- **Calculation**:
  - Use same first-seen logic
  - If event date is later than first-seen date, classify as `returning`
  - Count distinct `visitor_id` by period

---

## Time Aggregation Rules

All trend metrics support these granularities:

- `day` -> `%Y-%m-%d`
- `week` -> `%G-W%V` (ISO week)
- `month` -> `%Y-%m`

Time filtering rules:

- Prefer the indexed MongoDB Date field `ts`
- Fall back to converting legacy `timestamp` values when `ts` is not present
- Keep records where `start_time <= event time <= end_time`

Optional benchmark filtering:

- If benchmark filter is provided, add `benchmark == <value>` to match conditions

---

## Data Quality Notes

1. `visitor_id` is an approximate identifier, not a strict user identity.
2. For `filter_change_`*, `properties.new_value` may not always represent the actual final filter value; prefer `filters` snapshot for behavioral context.
3. If `table_download` is not instrumented, funnel step 4 will under-report by design.
4. Total UV and Sessions are distinct counts across the full selected time range. They are not calculated by summing per-period trend values.
5. Funnel steps are ordered by event time. A session only reaches a later step when that step happens after the previous required step.

---

## MongoDB Performance Notes

For production deployments, store event time as a MongoDB Date field named `ts`. Keeping only string timestamps forces aggregation pipelines to convert time values at query time and can reduce index usage.

Recommended indexes:

```javascript
db.events.createIndex({ ts: 1 })
db.events.createIndex({ ts: 1, benchmark: 1 })
db.events.createIndex({ event_name: 1, ts: 1 })
db.events.createIndex({ session_id: 1, ts: 1 })
db.events.createIndex({ "properties.visitor_id": 1, ts: 1 })
```

Legacy events with only `timestamp` remain supported, but backfilling `ts` is recommended before running this dashboard against large collections.

---

## Minimal Runtime Notes

Only required runtime inputs:

- MongoDB connection URI (`MONGO_URI`)
- Mongo database/collection names (defaults supported)

Local commands:

```bash
uv sync
uv run leaderboard-analytics
```

Run quality checks:

```bash
uv run ruff format --check .
uv run ruff check .
uv run pytest
```