Skip to content
data-updater-v2

data_updater_v2.jobs.metrics

metrics.py — the raw metrics: what each security is counted and valued at, day by day, one class per clock.

Metrics ETFs and equities: the NYSE’s sessions

metrics_build    by hand          build_equity_metrics      build_etf_metrics
metrics_update   metrics_ready    update_equity_metrics     update_etf_metrics

CryptoMetrics coins: the UTC day — Coin Metrics’ four tables, the night’s last landings

crypto_metrics_build    by hand                 build_crypto_{supply,valuation,network,exchange}
crypto_metrics_update   crypto_metrics_ready    update_crypto_{supply,valuation,network,exchange}

ShortMetrics ETFs and equities: the short side, mid-morning New York

short_build      by hand                build_short_interest   build_short_volume
                                        build_fails            build_borrow
short_update     10:00 NY, Mon-Fri      update_short_interest  update_short_volume
                                        update_fails           update_borrow

A build fetches a family’s histories in batches and writes each batch before fetching the next (DataClient.build_metrics: a stock’s SEC cover counts and FMP’s as-reported statement counts, kept when the latest agrees with today’s count once the stored splits since it are applied — the batch’s splits are read and handed in — else FMP’s market cap a day; a fund’s FMP market cap a day with today’s NAV and count): one receipt a batch, carrying what the fetch left unanswered; the fetch failures opened as issues. An update fetches the session its sensor stamped for every active listing at once (update_metrics: FMP’s sweep of counts and batch of market caps, each fund’s etf/info — each figure dated the session it values — and the SEC covers of the last days’ 10-Q and 10-K filers); an upsert merges each day with what is stored (write_metrics). The ready sensor fires once the ETF and equity daily price tables have landed for the session (a day’s counts and values belong beside its close) and the metrics tables have not; one run a session.

Classes

CryptoMetrics

Bases: DataJob

The coins’ Coin Metrics tables, a UTC day at a time: built in batches; updated each night as the night’s last landing — the closed day and the CRYPTO_REASK_DAYS before it re-asked (so last night’s fills, and the exchange flows still provisional, are replaced by Coin Metrics’ own rows), what Coin Metrics has not told yet for a day filled (DataClient.update_crypto_metrics, our daily bars handed in). Carried state: the base’s, with params this job’s CryptoMetricsParams.

build_crypto_exchange

build_crypto_exchange(cfg: RunConfig) -> None

Every active coin’s exchange-flow history (in, out, held on exchanges, in coins and dollars), as build_crypto_supply.

build_crypto_network

build_crypto_network(cfg: RunConfig) -> None

Every active coin’s network history (addresses, transactions, transfers, blocks, fees, hash rate), as build_crypto_supply.

build_crypto_supply

build_crypto_supply(cfg: RunConfig) -> None

Every active coin’s supply history (Coin Metrics’ current supply, issuance and ten-year expectation), in batches, each written before the next is fetched.

build_crypto_valuation

build_crypto_valuation(cfg: RunConfig) -> None

Every active coin’s valuation history (market cap, the estimated cap, MVRV, price, reference rate, spot volume), as build_crypto_supply.

crypto_metrics_ready

crypto_metrics_ready(context: Any) -> Any

Every 5 minutes from CLOSED after the UTC day ends: the closed day, not yet landed in the four tables — once the coin minutes have landed it (the night’s crypto chain: the daily, then the minutes, then these; the daily bars while the minutes’ job is switched off). It runs as soon as Coin Metrics stamps the day complete for the probes (the first two symbols of the update’s row, else BTC-USD and ETH-USD), else once the row’s wait_minutes have passed since the minutes were first seen landed (the cursor): what Coin Metrics has not told by then is filled, and the next night’s re-ask lands it. One run a day.

update_crypto_exchange

update_crypto_exchange(cfg: RunConfig) -> None

As update_crypto_supply: the coin flows carried, the dollar flows moved with our close; the provisional flows replaced night by night until Coin Metrics finals them.

update_crypto_network

update_crypto_network(cfg: RunConfig) -> None

As update_crypto_supply: the counts carried from the last told day.

update_crypto_supply

update_crypto_supply(cfg: RunConfig) -> None

The closed day and the days before it re-asked, what Coin Metrics has not told yet filled; written as an upsert. Coin Metrics unread: an issue on the table, the fill from the stored rows still landed; the table’s unread issue closed when read.

update_crypto_valuation

update_crypto_valuation(cfg: RunConfig) -> None

As update_crypto_supply: price and reference rate filled with our close, the caps and MVRV moved with it, the spot volume carried.

CryptoMetricsParams

Bases: JobParams

The crypto metrics jobs’ knobs (the row’s params).

Metrics

Bases: DataJob

The ETF and equity metrics: built in batches, updated a session at a time. Carried state: the base’s, with params this job’s MetricsParams.

build_equity_metrics

build_equity_metrics(cfg: RunConfig) -> None

Every active equity’s counts: the SEC’s covers (where the catalog holds a CIK) and FMP’s daily float history, in batches, each written before the next is fetched: one receipt a batch; the fetch failures opened as issues.

build_etf_metrics

build_etf_metrics(cfg: RunConfig) -> None

Every active ETF’s FMP market cap a day, with today’s NAV, net assets, expense ratio and count, as build_equity_metrics.

metrics_ready

metrics_ready(context: Any) -> Any

Every 15 minutes: the newest NYSE session past its close by SETTLED whose ETF and equity daily bars have landed (the day’s counts and values belong beside its close) and whose metrics have not. One run a session; the day rides the run as its session.

update_equity_metrics

update_equity_metrics(cfg: RunConfig) -> None

The last finished session’s row for every active equity: FMP’s sweep of counts and its batch of market caps, and the SEC covers of the last days’ filers. A source unread: an issue on the table and a skip; the fetch failures opened as issues; the table’s unread issue closed.

update_etf_metrics

update_etf_metrics(cfg: RunConfig) -> None

The last finished session’s row for every active ETF: the sweep’s count, the batch’s market cap and each fund’s etf/info, as update_equity_metrics.

MetricsParams

Bases: JobParams

The metrics jobs’ knobs (the row’s params).

ShortMetrics

Bases: DataJob

FINRA’s short interest and daily short volume, the SEC’s fails to deliver and IB’s borrow, for the ETFs and equities. Each source tells a past day (a settlement date, the last session, a published half month) or a snapshot (IB’s file), so the update runs mid-morning New York, after every one has published; each picks up from its table’s last stored day, so a missed morning is taken the next (up to catch_up_days). A build is by hand, a sweep over the world a window at a time (FINRA and the SEC serve every listing in one file or query: the window, not the listings, is what is cut). Carried state: the base’s, with params this job’s ShortMetricsParams.

build_borrow

build_borrow(cfg: RunConfig) -> None

The borrow fee and shares available before our own collection: companiesmarketcap’s republished IB history (from 2023-01-21, companies only), batch listings at a time, each written before the next is fetched.

build_fails

build_fails(cfg: RunConfig) -> None

The SEC’s fails to deliver from its half-month files’ start (2009-07), as build_short_interest: the halves the window covers.

build_short_interest

build_short_interest(cfg: RunConfig) -> None

FINRA’s short interest at every settlement date from its floor (2017-12-29), a window of window_days at a time, each written before the next is fetched.

build_short_volume

build_short_volume(cfg: RunConfig) -> None

FINRA’s daily short-sale volume from its floor (2018-08-01), as build_short_interest: one file a day.

update_borrow

update_borrow(cfg: RunConfig) -> None

IB’s file as read this morning (ftp2.interactivebrokers.com, every lendable US listing), dated the file’s day. A reading is a snapshot: a missed morning is a day without one (companiesmarketcap’s history is the build’s). The FTP unreachable, or a file whose trailer count disagrees with its rows (cut short): an issue on the table and a skip.

update_fails

update_fails(cfg: RunConfig) -> None

The last two ended half months (the SEC publishes a half two to three weeks after it ends: each morning re-asks them, idempotently), reaching back to the last stored day when a missed stretch left more.

update_short_interest

update_short_interest(cfg: RunConfig) -> None

The settlement dates after the last stored one, and the last month’s again (FINRA revises a published date: the re-ask lands the revision; an upsert, idempotent). FINRA posts a date about two weeks after it: most mornings bring nothing new.

update_short_volume

update_short_volume(cfg: RunConfig) -> None

The sessions after the last stored day through the last finished one (the file of a day is out by its evening): one file a day; a holiday’s absence is named, not an error.

ShortMetricsParams

Bases: JobParams

The short-side jobs’ knobs (the row’s params).