data_updater_v2.jobs.intraday

intraday.py — the raw minute bars, one class per clock, hot and cold as separate jobs.

ListedIntraday   ETFs and equities: the NYSE's sessions; FMP where it carries the listing
                 else EODHD, the other for its gaps; each day unified to the regular
                 session on the raw scale against the listing's daily bar (FMP's minutes
                 come split-adjusted; the scale is measured, exact with the stored splits)
CryptoIntraday   coins: the UTC day; Bitstamp where it lists the pair, then FMP, then
                 EODHD under its proven code; every minute kept as served, zero volume at
                 zero (Bitstamp and EODHD in coins, FMP in dollars)
IndexIntraday    indices: the NYSE's sessions; FMP, then EODHD where proven (its index
                 minutes are good only to 2024-07)

*_intraday_hot_build      by hand: the row's set, every history, a window at a time
*_intraday_hot_update     its ready sensor: the last finished session (UTC day for coins)
*_intraday_hot_repair     02:00-04:00 NY while the table owes work: the issues' windows
*_intraday_cold_*         the same steps on the row's cold store (common-db's cold leg,
                          not built yet: the DataClient refuses it by name); by hand

The store, the set and the width come from each job’s row (store, symbols, interval: the widths every vendor serves — 1min, 5min, an hour for coins; coarser widths are read from the stored minutes). A build walks a listing’s history window_days at a time and writes each window before the next is fetched: minutes are millions a listing.

The ready sensors probe, they do not fetch the set: the first two symbols of the job’s row (else the family’s named pair), each vendor on the probe’s route asked for that one day of minutes, the day counted whole when its first PROBE_MINUTES, PROBE_MINUTES in the middle and its last PROBE_MINUTES are there; the family’s daily must have landed for the day (the anchor). Listed and index: the newest session past its close; coins: from 00:05 UTC.

Classes

CryptoIntraday

Bases: DataJob

The coins’ minute bars, the UTC day: built a coin and a window at a time, updated a day at a time, backfilled from the issues, checked against the tables. Carried state: the base’s, with params this job’s CryptoIntradayParams.

backfill_crypto_intraday

backfill_crypto_intraday(cfg: RunConfig) -> None

What the coin minute table owes, from the issues (intraday_backfill), each window fetched against the coins’ daily bars and written; what no vendor could fill closed as such; the check then closes what landed.

build_crypto_intraday

build_crypto_intraday(cfg: RunConfig) -> None

Every coin of the set: its minutes from its first daily bar (or start) to the last finished UTC day (or end), window_days at a time — Bitstamp where it lists the pair, then FMP, then EODHD under its proven code, each next for the days the one before lacks, every minute kept as served and checked against the coin’s daily bar — each window written before the next is fetched: one receipt a window, carrying the gaps; the fetch failures opened.

check_crypto_intraday

check_crypto_intraday(cfg: RunConfig) -> None

What the coin minute table owes, read from the tables alone, every UTC day its daily bar has: opened, closed where it no longer holds, superseded where the era ended. A subset run checks its own coins only.

crypto_intraday_ready

crypto_intraday_ready(context: Any) -> Any

Every 15 minutes from CLOSED after the UTC day ends: the closed day, not yet landed — once the crypto daily has landed it (the anchor) and every vendor on the probes’ routes has the whole day of minutes (day_whole). The probes: the first two symbols of the hot update’s row, else BTC-USD and ETH-USD. One run a day.

crypto_intraday_repair_due

crypto_intraday_repair_due(context: Any) -> Any

Once a night, 02:00-04:00 New York, only while the coin minute table owes work or a source went unread.

update_crypto_intraday

update_crypto_intraday(cfg: RunConfig) -> None

The last finished UTC day’s minutes for every coin of the set: the main vendor’s day, its gaps that day asked of the next, checked against that day’s daily bar. A source unread: an issue on the table and a skip. The fetch failures opened; the table’s unread issue closed.

CryptoIntradayParams

Bases: JobParams

A crypto intraday job’s row params, as ListedIntradayParams without splits.

IndexIntraday

Bases: DataJob

The indices’ minute bars on the NYSE’s clock: built an index and a window at a time, updated a session at a time, backfilled from the issues, checked against the tables. Carried state: the base’s, with params this job’s IndexIntradayParams.

backfill_index_intraday

backfill_index_intraday(cfg: RunConfig) -> None

What the index minute table owes, from the issues (intraday_backfill), each window fetched against the indices’ daily bars and written; what no vendor could fill closed as such; the check then closes what landed.

build_index_intraday

build_index_intraday(cfg: RunConfig) -> None

Every index of the set: its minutes from its first daily bar (or start) to the last finished session (or end), window_days at a time — FMP, then EODHD where its series is proven FMP’s (its index minutes good only to 2024-07), each day checked against the index’s daily bar — each window written before the next is fetched: one receipt a window, carrying the gaps; the fetch failures opened.

check_index_intraday

check_index_intraday(cfg: RunConfig) -> None

What the index minute table owes, read from the tables alone: opened, closed where it no longer holds, superseded where the era ended. A subset run checks its own indices only.

index_intraday_ready

index_intraday_ready(context: Any) -> Any

Every 15 minutes: the newest session past its close that the index minute table does not have yet — once the index daily has landed it (the anchor) and FMP has the whole session of minutes for the probes (the first two symbols of the hot update’s row, else GSPC and NDX). One run a session, the day as its session.

index_intraday_repair_due

index_intraday_repair_due(context: Any) -> Any

Once a night, 02:00-04:00 New York, only while the index minute table owes work or a source went unread.

update_index_intraday

update_index_intraday(cfg: RunConfig) -> None

The last finished session’s minutes for every index of the set, checked against that day’s daily bar. A source unread: an issue on the table and a skip. The fetch failures opened; the table’s unread issue closed.

IndexIntradayParams

Bases: JobParams

An index intraday job’s row params, as CryptoIntradayParams on the NYSE’s sessions.

ListedIntraday

Bases: DataJob

The ETF and equity minute bars: built a listing and a window at a time, updated a session at a time, backfilled from the issues, checked against the tables. Carried state: the base’s, with params this job’s ListedIntradayParams.

backfill_equity_intraday

backfill_equity_intraday(cfg: RunConfig) -> None

What the equity minute table owes, as backfill_etf_intraday.

backfill_etf_intraday

backfill_etf_intraday(cfg: RunConfig) -> None

What the ETF minute table owes, from the issues (intraday_backfill: windows of at most max_days), each fetched against the listings’ daily bars and written, one receipt a window; what no vendor could fill closed as such; the check then closes what landed.

build_equity_intraday

build_equity_intraday(cfg: RunConfig) -> None

Every equity of the set, as build_etf_intraday.

build_etf_intraday

build_etf_intraday(cfg: RunConfig) -> None

Every ETF of the set: its minutes from its first daily bar (or start) to the last finished session (or end), window_days at a time — FMP where it carries the listing else EODHD, the other for its gaps, each day unified to the session on the raw scale against the listing’s daily bar, exact with the stored splits when handed in — each window written before the next is fetched: one receipt a window, carrying the gaps; the fetch failures opened. A listing with no daily bars yet is left to the daily job.

check_equity_intraday

check_equity_intraday(cfg: RunConfig) -> None

What the equity minute table owes, as check_etf_intraday.

check_etf_intraday

check_etf_intraday(cfg: RunConfig) -> None

What the ETF minute table owes, read from the tables alone (no_history, late_history, behind, run_missed, missing, short and edge days, invalid_bars, extra_days, bars_after_end): opened, closed where it no longer holds, superseded where the era ended. A subset run checks its own listings only.

listed_intraday_ready

listed_intraday_ready(context: Any) -> Any

Every 15 minutes: the newest session past its close that the minute tables do not have yet — once both daily tables have landed it (the anchor) and every vendor on the probes’ routes has the whole session of minutes (day_whole: its first, middle and last PROBE_MINUTES). The probes: the first two symbols of the hot update’s row, else SPY and QQQ. One run a session, the day as its session.

listed_intraday_repair_due

listed_intraday_repair_due(context: Any) -> Any

Once a night, 02:00-04:00 New York, only while a minute table owes work (what intraday_backfill names for the hot set) or a source went unread.

update_equity_intraday

update_equity_intraday(cfg: RunConfig) -> None

The last finished session’s minutes for every equity of the set, as update_etf_intraday.

update_etf_intraday

update_etf_intraday(cfg: RunConfig) -> None

The last finished session’s minutes for every ETF of the set: the main vendor’s day, its gaps that day asked of the other, checked against that day’s daily bar. A source unread: an issue on the table and a skip. The fetch failures opened; the table’s unread issue closed.

ListedIntradayParams

Bases: JobParams

A listed intraday job’s row params: the set, the store, the width, the build’s window and chunk, the splits handed to the fetch, and how the bars are written (build_intraday / update_intraday / intraday_backfill / write_prices’s).

Functions

day_whole

day_whole(bars: Any, start: dt.datetime, end: dt.datetime) -> bool

Whether a vendor’s day of minutes [start, end) holds a bar in each of its first PROBE_MINUTES, PROBE_MINUTES in the middle and its last PROBE_MINUTES.

last_session

last_session(sessions: Any, now: dt.datetime) -> dt.date | None

The newest session whose close is SETTLED past, from the calendar’s models.

probes

probes(params: dict[str, Any], kind: str) -> tuple[str, ...]

The symbols a ready sensor probes: the first two of the row’s set, else the family’s.

vendors_have

vendors_have(dc: DataClient, kind: Any, row: Any, day: dt.date, start: dt.datetime, end: dt.datetime, interval: Width) -> dict[str, bool]

Each vendor on the listing’s route asked for that one day of minutes: whether its day is whole (day_whole). An index asks FMP alone (EODHD’s index minutes are not kept).

windows

windows(first: dt.date, last: dt.date, days: int) -> list[tuple[dt.date, dt.date]]

[first, last] cut into consecutive windows of days days, the last one shorter.