Hugging Face's logo

Spaces:
SmileXing
/
leaderboard-analytics-service
Sleeping

App Files Community
4
leaderboard-analytics-service / README.md
SmileXing's picture
SmileXing
Optimization (#3)
19b1ae7
preview code
|
Raw
History Blame
6.87 kB
metadata 
title: leaderboard-analytics-service
emoji: 📊
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: 6.0.0
python_version: '3.11'
app_file: app.py
pinned: false

Leaderboard Analytics Metrics Spec

This project analyzes user behavior on the MTEB leaderboard page from event logs in MongoDB.

The primary purpose of this document is to define what is measured, where each metric comes from, and how each metric is calculated.

Data Contract

All analytics are based on the events collection and the following stable fields:

  • Core dimensions: event_name, timestamp, session_id
  • Preferred event time: ts as a MongoDB Date
  • Behavior context: benchmark, filters
  • Visitor identity (approximate): properties.visitor_id
  • Change context: properties.old_value, properties.new_value, properties.filter_name

Important event names:

  • page_view
  • benchmark_change
  • filter_change_* (dynamic names, such as filter_change_task_type)
  • table_download (currently may be missing in some deployments)

Metrics Dictionary

1) PV (Page Views)

  • Definition: Number of page view events.
  • Source fields: event_name
  • Calculation:
    • Filter events where event_name == "page_view"
    • PV = count of matched events

2) Sessions

  • Definition: Number of unique interaction sessions.
  • Source fields: session_id
  • Calculation:
    • Sessions = count of distinct non-empty session_id values in the selected time range

3) UV (Unique Visitors, Approximate)

  • Definition: Number of unique visitors identified by hashed fingerprint.
  • Source fields: properties.visitor_id
  • Calculation:
    • Remove null/empty properties.visitor_id
    • UV = count of distinct properties.visitor_id values in the selected time range

4) Sessions Per Visitor

  • Definition: Average number of sessions per visitor.
  • Source fields: derived from Sessions and UV
  • Calculation:
    • Sessions Per Visitor = Sessions / UV
    • If UV is 0, result is 0

5) Session Depth (Events Per Session)

  • Definition: Average interaction intensity per session.
  • Source fields: all events, session_id
  • Calculation:
    • Total Events = count of all events in range
    • Session Depth = Total Events / Sessions
    • If Sessions is 0, result is 0

Behavior Metrics

6) Benchmark Popularity

  • Definition: Frequency of selected benchmarks.
  • Source fields: event_name, properties.new_value
  • Calculation:
    • Filter event_name == "benchmark_change"
    • Group by properties.new_value
    • Popularity = event count per benchmark value

7) Filter Usage Distribution

  • Definition: Usage volume by filter event type.
  • Source fields: event_name
  • Calculation:
    • Filter event_name matching regex ^filter_change_
    • Group by event_name
    • Distribution = count per filter event

8) Filter Session Coverage

  • Definition: Number of sessions that used each filter type.
  • Source fields: event_name, session_id
  • Calculation:
    • For each filter_change_* event type:
      • collect distinct non-empty session_id
      • coverage = distinct session count

Funnel Metrics

Recommended session-level funnel:

  1. page_view
  2. benchmark_change
  3. filter_change_*
  4. table_download

9) Step Session Count

  • Definition: Number of sessions that reached each ordered funnel step.
  • Source fields: session_id, event_name, ts or timestamp
  • Calculation:
    • Group events by session_id
    • Sort events by event time
    • Count each cumulative step only when it occurs after the previous required step

10) Step Conversion Rate

  • Definition: Conversion from funnel step 1 (page_view) to each step.
  • Source fields: derived from Step Session Count
  • Calculation:
    • Conversion Rate(step N) = StepN Sessions / Step1 Sessions * 100%
    • If Step1 Sessions is 0, result is 0%

Visitor Segmentation Metrics

11) New Visitors

  • Definition: Visitors whose current period contains their first observed visit date.
  • Source fields: event_name, ts or timestamp, properties.visitor_id
  • Calculation:
    • Use page_view events only
    • For each visitor_id, find earliest timestamp (first_seen) from the full available dataset
    • If event date equals first_seen date, classify as new
    • Count distinct visitor_id by period

12) Returning Visitors

  • Definition: Visitors seen after their first observed date.
  • Source fields: same as New Visitors
  • Calculation:
    • Use same first-seen logic
    • If event date is later than first-seen date, classify as returning
    • Count distinct visitor_id by period

Time Aggregation Rules

All trend metrics support these granularities:

  • day -> %Y-%m-%d
  • week -> %G-W%V (ISO week)
  • month -> %Y-%m

Time filtering rules:

  • Prefer the indexed MongoDB Date field ts
  • Fall back to converting legacy timestamp values when ts is not present
  • Keep records where start_time <= event time <= end_time

Optional benchmark filtering:

  • If benchmark filter is provided, add benchmark == <value> to match conditions

Data Quality Notes

  1. visitor_id is an approximate identifier, not a strict user identity.
  2. For filter_change_*, properties.new_value may not always represent the actual final filter value; prefer filters snapshot for behavioral context.
  3. If table_download is not instrumented, funnel step 4 will under-report by design.
  4. Total UV and Sessions are distinct counts across the full selected time range. They are not calculated by summing per-period trend values.
  5. Funnel steps are ordered by event time. A session only reaches a later step when that step happens after the previous required step.

MongoDB Performance Notes

For production deployments, store event time as a MongoDB Date field named ts. Keeping only string timestamps forces aggregation pipelines to convert time values at query time and can reduce index usage.

Recommended indexes:

db.events.createIndex({ ts: 1 })
db.events.createIndex({ ts: 1, benchmark: 1 })
db.events.createIndex({ event_name: 1, ts: 1 })
db.events.createIndex({ session_id: 1, ts: 1 })
db.events.createIndex({ "properties.visitor_id": 1, ts: 1 })

Legacy events with only timestamp remain supported, but backfilling ts is recommended before running this dashboard against large collections.

Minimal Runtime Notes

Only required runtime inputs:

  • MongoDB connection URI (MONGO_URI)
  • Mongo database/collection names (defaults supported)

Local commands:

uv sync
uv run leaderboard-analytics

Run quality checks:

uv run ruff format --check .
uv run ruff check .
uv run pytest