Dataset freshness

Every weekly warehouse build appends one row per CMS dataset to a durable JSONL ledger at data/vintages/ledger.jsonl — capture time, upstream modified / issued / released dates, Parquet row and byte counts, and a schema hash. Because the ledger is committed to the repo, this history survives the ephemeral raw-data teardown each CI run does, and the mart at fct_dataset_freshness can answer three questions on top of it: how fresh is each dataset today, when did the upstream publisher last touch it, and what has changed between runs.

How to read this page today

The ledger currently holds

Per-dataset latest state

The primary view: for each dataset, the latest run_date, when the upstream publisher last modified the data, days since each of those, and the Parquet-footer row count. days_since_capture and days_since_upstream_modified are computed at build time against current_date, not at query time — the numbers age on disk between weekly rebuilds. healthcare_gov_glossary is the sole dataset with no upstream modified date; its publisher exposes none.

Coverage by source family

The 43 datasets group into six extraction families. dkan_provider_data (the CMS Care Compare provider files) is the largest cohort; dkan_data_api_bulk (spending-and-utilization bulk feeds) accounts for most of the aggregate row count. Upstream freshness varies markedly across families — the QHP landscape files (dkan_healthcare_gov_zip) are the freshest cohort at an average of about 32 days since publisher modified, while the bulk fact tables are the oldest, averaging over 100 days.

Upstream publication window

The current backfill captures upstream modified dates ranging from

Row-count landscape

Dataset sizes span eight orders of magnitude — from home_health_national at one row (a national roll-up) to medicare_part_d_prescribers_by_provider_and_drug at roughly 28 million rows. Because BarChart doesn't expose a log y-axis on this Evidence build, the section shows two views: a bucketed distribution across decades, and the top ten datasets by raw row count.

Change tracking (fills in over time)

The mart carries three between-run signals — row_count_delta, schema_changed, and n_distinct_modified — but each needs at least two ledger rows for a given dataset to say anything. Today every dataset has n_runs = 1, so the first two columns are NULL / false for the whole table by construction and there is nothing to plot. n_distinct_modified counts distinct non-null upstream dates observed to date; today that's exactly one for every dataset except healthcare_gov_glossary (whose publisher exposes no modified date, so it stays at zero). As weekly builds append, this section will grow the distributions and comparisons that only history can answer.

Caveats

• Single-run backfill. The ledger's 211 rows all come from the initial backfill between September 6, 2026 and September 28, 2026 . `run_date` reflects the day each dataset's sidecar was captured, not a uniform release schedule; from now on the weekly warehouse job appends one row per dataset per run. `row_count_delta`, `schema_changed`, and the distribution of `n_runs` fill in as history accrues.
• Upstream modified is CMS metadata. The modified column comes from the publisher's own dataset metadata (DKAN, healthcare.gov, etc.) — it can lag or lead the moment the underlying data actually changed, and one publisher ( 1 dataset: healthcare_gov_glossary) exposes no modified date at all. Treat "days since upstream modified" as the metadata age, not the data-content age.
• Build-time staleness. days_since_capture and days_since_upstream_modified are computed against current_date at dbt build time, not query time, so the numbers age on disk between weekly rebuilds. Interpret them as "days since capture as of the last successful warehouse build".
• Row counts are Parquet-footer sums. row_count sums Parquet num_rows across the files that make up each dataset at capture time. It counts raw rows in the staging extract — not distinct entities, not post-dedupe mart rows, not filtered scopes — so it is best read as data-volume signal, not analytical population size.
• Bot-PR cadence. Weekly builds publish the ledger back to main via a bot pull request, so the committed ledger this page reads can trail the deployed site by up to a week if the PR hasn't merged yet. The mart is rebuilt on every deploy, so the deployed page and the mart-on-disk always agree, but both can predate the most recent uncommitted capture.