Skip to main content

directly_written_records

Cache records that must be migrated to a fixed version whenever the cache opens.

Most cache records reach their target version because a DAG step declares a cache_table/cache_table_version binding and flows/dag/setup pins it before the run. These records are the exception, for either of two reasons.

  • No step binding names them. They are written by direct store calls and read directly, so no CacheAccessor and no binding covers them — e.g. trials_published_data_pointer, flipped by patient_eligibility via set_published_pointer and read by the patient-data API.
  • A step binding names them, but a reader-only process reads a column above v1. A step's binding is only applied on a pod that runs the DAG, so it does nothing for a process that only reads — e.g. patient_level_eligibility, whose ehr_retrieved_at column (added in v2) the patient-data API reads per patient and whose partitions it uses to scope the unpartitioned patients listing (the table has since moved on to v3 for an unrelated column, score, which the API does not read directly — but the entry here must still track whatever version the step itself pins), and model_inferences, whose v2 source_file_hash the API reaches through the full-row select(ModelInference) behind patient_api/scans/segmentations.py. Pinned here at the same version the step binds, so the two cannot disagree.

ensure_types migrates only what something pins. A record named by neither a step binding nor this map therefore stays at whatever version created it, and the first read of a column added in a later version fails with no such column on any pre-existing cache — which is exactly what happens to a reader (the patient-data API) that opens a cache no publishing DAG has migrated on this pod. So both paths pin this map:

  • flows/dag/setup._pinned_type_versions merges it into a run's step-derived pins, so a publishing pod migrates these records at DAG setup.
  • runtimes/file_metadata.open_background_cache passes it to ensure_types on every open, so a reader opening the pod cache stands these records at their target version without depending on a DAG having run first.

Hardcoding trades the registry's discovery-time validation for an explicit list, so the validation moves to a test: tests/bitfount/flows/dag/test_setup.py::test_every_directly_written_record_exists_at_that_version resolves every entry against the record registry, so a typo or a version that does not exist fails as a test rather than as no such column on someone's cache.

An entry can carry a real migration cost onto the reader's open: model_inferences reaches v3 by building the file_id index, and that table holds one row per inference per file, so a cache still at v1 or v2 with rows pays an index build proportional to row count when the API first opens it. A publishing DAG would have paid the same cost, and the alternative is the no such column failure above, so the cost is accepted rather than avoided.