Skip to content
data-updater-v2

data_updater_v2.jobs.daily_prices

daily_prices.py — the raw daily bars, one class per clock.

ListedDaily ETFs and equities: the NYSE’s sessions, EODHD as traded, FMP for what it lacks (only with the checked splits)

listed_daily_build    by hand            build_etf_daily ──────▶ check_etf_daily
                                         build_equity_daily ───▶ check_equity_daily
listed_daily_update   listed_daily_ready update_etf_daily ─────▶ check_etf_daily
                                         update_equity_daily ──▶ check_equity_daily
listed_daily_repair   02:00-04:00 NY     backfill_etf_daily ───▶ check_etf_daily
                                         backfill_equity_daily ▶ check_equity_daily

A build fetches a family’s histories in batches and writes each batch before fetching the next (whole histories are ~11,000 bars a listing): one receipt a batch, carrying the gaps. An update fetches the last finished session for every active listing at once. A backfill does what the issues ask (daily_backfill: the daily issues with an action, and the universe’s new listings). The check reads what the table owes from the tables alone and opens, closes and supersedes the issues. The ready sensor fires once the listed universe has landed for the session and EODHD’s file holds it whole, seen the same size twice (its 2026-08-28 rule); the repair sensor fires while the table owes work.

Classes

CryptoDaily

Bases: DataJob

The coins’ daily bars, one a UTC day: built in batches, updated a day at a time, backfilled from the issues, checked against the tables. Carried state: the base’s, with params this job’s CryptoDailyParams.

backfill_crypto_daily

backfill_crypto_daily(cfg: RunConfig) -> None

What the crypto table owes, from the issues: the live coins with an open daily issue that asks work, and the universe’s new coins — each re-checked now, grouped into windows (daily_backfill), fetched and written, one receipt a window; the check then closes what landed.

build_crypto_daily

build_crypto_daily(cfg: RunConfig) -> None

Every active coin’s daily history (Bitstamp’s own candles where it lists the pair, EODHD under its proven code, FMP for the rest — each next one for the days the one before lacks), in batches, each written before the next is fetched: one receipt a batch, carrying the gaps; the fetch failures opened as issues.

check_crypto_daily

check_crypto_daily(cfg: RunConfig) -> None

What the crypto table owes, read from the tables alone, every UTC day owed: opened, closed where it no longer holds, superseded where the era ended. A subset run checks its own coins only.

crypto_daily_ready

crypto_daily_ready(context: Any) -> Any

Every 15 minutes from CLOSED after the UTC day ends: the closed day, not yet landed — once the crypto catalog has landed since the day before, Bitstamp’s BTC candle for it exists (its own, final at the close) and EODHD’s CC file holds it (PUBLISHED of our on_eodhd coins present with a close), seen the same size on two consecutive ticks. One run a day; the day rides the run as its session.

crypto_daily_repair_due

crypto_daily_repair_due(context: Any) -> Any

Once a night, 02:00-04:00 New York (two hours: the night the clocks go forward has no 02:00), only while the crypto table owes work (what daily_backfill names).

update_crypto_daily

update_crypto_daily(cfg: RunConfig) -> None

The last finished UTC day’s bar for every active coin: Bitstamp per pair, EODHD’s CC file for the coins under its code, FMP for the rest and for a Bitstamp day with no trade. The file unread: an issue on the table and a skip (the backfill fills the day). The fetch failures opened as issues; the table’s unread issue closed.

CryptoDailyParams

Bases: JobParams

A crypto daily job’s row params: which coins, the build’s window and batch, and how the bars are written. Coins split nothing: no splits knob.

IndexDaily

Bases: DataJob

The index and commodity daily bars on the NYSE’s clock, each family its own jobs so one switches off by its row: built in batches, updated a session at a time, backfilled from the issues, checked against the tables. Carried state: the base’s, with params this job’s IndexDailyParams.

backfill_commodity_daily

backfill_commodity_daily(cfg: RunConfig) -> None

What the commodity table owes, as backfill_index_daily.

backfill_index_daily

backfill_index_daily(cfg: RunConfig) -> None

What the index table owes, from the issues (daily_backfill), fetched and written window by window; what no vendor could fill is closed as such; the check then closes what landed.

build_commodity_daily

build_commodity_daily(cfg: RunConfig) -> None

Every active commodity’s daily history (FMP alone), in batches, as build_index_daily.

build_index_daily

build_index_daily(cfg: RunConfig) -> None

Every active index’s daily history (FMP /stable first for the core indices, EODHD where its series is proven FMP’s — on FMP’s scale —, FMP /api/v3 for the rest; each next one for the days the one before lacks), in batches, each written before the next is fetched: one receipt a batch, carrying the gaps; the fetch failures opened as issues.

check_commodity_daily

check_commodity_daily(cfg: RunConfig) -> None

What the commodity table owes, as check_index_daily.

check_index_daily

check_index_daily(cfg: RunConfig) -> None

What the index table owes, read from the tables alone: opened, closed where it no longer holds, superseded where the era ended. A subset run checks its own indices only. An index’s calendar is not ours: it is behind only past one session.

commodity_daily_ready

commodity_daily_ready(context: Any) -> Any

Every 15 minutes: the newest NYSE session past its close by SETTLED that the commodity table does not have yet — once the commodity catalog has landed for it and FMP serves gold’s bar for the day (one call). One run a session.

commodity_daily_repair_due

commodity_daily_repair_due(context: Any) -> Any

Once a night, 02:00-04:00 New York, only while the commodity table owes work.

index_daily_ready

index_daily_ready(context: Any) -> Any

Every 15 minutes: the newest NYSE session past its close by SETTLED that the index table does not have yet — once the index catalog has landed for it and EODHD’s INDX file holds it (PUBLISHED of our on_eodhd indices present with a close), seen the same size on two consecutive ticks. One run a session.

index_daily_repair_due

index_daily_repair_due(context: Any) -> Any

Once a night, 02:00-04:00 New York, only while the index table owes work.

update_commodity_daily

update_commodity_daily(cfg: RunConfig) -> None

The last finished session’s bar for every active commodity (FMP), as update_index_daily.

update_index_daily

update_index_daily(cfg: RunConfig) -> None

The last finished session’s bar for every active index: FMP /stable for the core ones, EODHD’s INDX file for the rest it proves, FMP for what remains. The file unread: an issue on the table and a skip. The fetch failures opened as issues; the table’s unread issue closed.

IndexDailyParams

Bases: JobParams

An index or commodity daily job’s row params: which listings, the build’s window and batch, and how the bars are written. Nothing splits: no splits knob.

ListedDaily

Bases: DataJob

The ETF and equity daily bars: built in batches, updated a session at a time, backfilled from the issues, checked against the tables. Carried state: the base’s, with params this job’s ListedDailyParams.

backfill_equity_daily

backfill_equity_daily(cfg: RunConfig) -> None

What the equity table owes, as backfill_etf_daily.

backfill_etf_daily

backfill_etf_daily(cfg: RunConfig) -> None

What the ETF table owes, from the issues: the live eras with an open daily issue that asks work, and the universe’s open new listings — each re-checked now, grouped into windows (daily_backfill), fetched and written, one receipt a window; the check then closes what landed.

build_equity_daily

build_equity_daily(cfg: RunConfig) -> None

Every active equity’s daily history, as build_etf_daily.

build_etf_daily

build_etf_daily(cfg: RunConfig) -> None

Every active ETF’s daily history (EODHD as traded, its volume put back to shares; FMP only when the splits are handed in), in batches, each written before the next is fetched: one receipt a batch, carrying the gaps; the fetch failures opened as issues.

check_equity_daily

check_equity_daily(cfg: RunConfig) -> None

What the equity table owes, as check_etf_daily.

check_etf_daily

check_etf_daily(cfg: RunConfig) -> None

What the ETF table owes, read from the tables alone (no_history, late_history, behind, run_missed, missing_days, invalid_bars, extra_days, bars_after_end): opened, closed where it no longer holds, superseded where the era ended. A subset run checks its own listings only.

listed_daily_ready

listed_daily_ready(context: Any) -> Any

Every 15 minutes: the newest NYSE session past its close by SETTLED that the daily tables do not have yet — once the listed universe has landed for it and EODHD’s US file holds it (rows dated the day; PUBLISHED of our on_eodhd listings present with a close), seen the same size on two consecutive ticks, so a file still filling never lands as a partial day. One run a session; the day rides the run as its session.

listed_daily_repair_due

listed_daily_repair_due(context: Any) -> Any

Once a night, 02:00-04:00 New York (two hours: the night the clocks go forward has no 02:00), only while a daily table owes work: an open daily issue with an action, or an open new listing (what daily_backfill names).

refill_equity_daily

refill_equity_daily(cfg: RunConfig) -> None

The equities whose splits were corrected, as refill_etf_daily.

refill_etf_daily

refill_etf_daily(cfg: RunConfig) -> None

The ETFs whose splits were corrected (split_changed, the splits job’s): each one’s daily re-asked up to the day with the stored splits — EODHD-served bars come back the same, FMP-filled days and EODHD’s pre-split sessions on the corrected scale — upserted, one receipt a listing; the issue closed. A subset run refills its own listings only.

update_equity_daily

update_equity_daily(cfg: RunConfig) -> None

The last finished session’s bar for every active equity, as update_etf_daily.

update_etf_daily

update_etf_daily(cfg: RunConfig) -> None

The last finished session’s bar for every active ETF: EODHD’s file for the day, FMP for what it lacks. The file unread: an issue on the table and a skip (the backfill fills the day). The fetch failures opened as issues; the table’s unread issue closed.

ListedDailyParams

Bases: JobParams

A listed daily job’s row params: which listings, the build’s window and batch, the splits handed to the fetch, and how the bars are written (build_daily / update_daily / daily_backfill / write_prices’s own).