Skip to content
data-updater-v2

data_updater_v2.jobs.universe

universe.py — the catalogs and the changes that move them.

universe_build           the five catalogs, each built whole and written, side by side:
                         every listing in today's pull identified afresh, the stored eras
                         carried (a rebuild never re-mints an id), every era no longer
                         listed kept inactive


                             build_etf_universe ───────▶ write_etf_universe
                             build_equity_universe ────▶ write_equity_universe
                             build_index_universe ─────▶ write_index_universe
                             build_crypto_universe ────▶ write_crypto_universe
                             build_commodity_universe ─▶ write_commodity_universe


universe_changes_build   symbol changes, listings and delistings over their whole reach,
                         every US event the vendors tell, our ids where the stored
                         catalogs hold the security (so universe_build runs first)


                             read_catalogs ─┬─ build_symbol_changes ─▶ write_symbol_changes
                                            ├─ build_listings ───────▶ write_listings
                                            └─ build_delistings ─────▶ write_delistings


universe_changes_update  18:00 New York, Monday to Friday: the window since the last
                         landing (``lookback_days`` at least), our ids from the stored
                         catalogs, each table made exact over its window


                             update_symbol_changes ─▶ land_symbol_changes
                             update_listings ───────▶ land_listings
                             update_delistings ─────▶ land_delistings


universe_update          18:00 every day (coins trade on weekends): the catalogs that
                         need no changes


                             update_index_universe ─────▶ land_index_universe
                             update_crypto_universe ────▶ land_crypto_universe
                             update_commodity_universe ─▶ land_commodity_universe


listed_universe_update   once the three changes have landed: ETFs and equities against
                         the stored changes since each catalog's last landing (renames
                         carried, listings and delistings explaining what came and went),
                         then what the updates found that no vendor told into the changes
                         tables, the window's events given today's ids, the issues opened


                             update_etf_universe ────▶ land_etf_universe ────┐
                             update_equity_universe ─▶ land_equity_universe ─┤
                                          land_untold_changes ◀──────────────┘
                                                  │
                                          open_universe_issues


*_repair                 02:00-04:00 New York, only while an unread issue on one of the
                         job's tables is open: the same steps again

Each build or update fetches and writes nothing; each write or land lands one table and files its receipt. A fetch handed to its write through the step store is never redone for a write that failed. A source unread opens an issue on its table and skips the step (its write skips with it); the table landing closes the issue.

Classes

CatalogParams

Bases: JobParams

How a catalog is written (write_universe’s).

ChangesParams

Bases: JobParams

The changes’ windows and how their tables are written (write_listings’ …).

UniverseBuild

Bases: DataJob

The universe builds: the five catalogs, the three changes tables from the stored catalogs, and the delisted eras of the two listed families. Carried state: the base’s, with params this job’s UniverseParams.

build_commodity_universe

build_commodity_universe(cfg: RunConfig) -> dict[str, Any]

FMP’s commodity list, one call. Writes nothing.

build_constituents

build_constituents(cfg: RunConfig) -> dict[str, Any]

Every add, remove and today’s member of the S&P 500, Nasdaq 100 and Dow 30 as FMP tells them, our ids where the catalogs hold the ticker. Writes nothing.

build_crypto_universe

build_crypto_universe(cfg: RunConfig) -> dict[str, Any]

Every USD coin FMP quotes, Bitstamp trades or EODHD prices, one row each, by OpenFIGI’s coin records (the whole list on a build), else code and name. Writes nothing.

build_delistings

build_delistings(cfg: RunConfig, read_catalogs: dict[str, list[Any]]) -> dict[str, Any]

Every US delisting from the start (None: all FMP keeps), announced ones included: FMP’s delisted list and Nasdaq’s suspensions and deletions, combined, our ids where the catalogs hold the ticker. Writes nothing.

build_equity_delisted

build_equity_delisted(cfg: RunConfig) -> dict[str, Any]

Every equity that has left the US venues (EODHD’s delisted list, FMP’s), minted as an inactive era with its ids; the eras the catalog holds carried. Writes nothing.

build_equity_universe

build_equity_universe(cfg: RunConfig) -> dict[str, Any]

Every equity in FMP’s screener identified afresh (profile, sec-profile, OpenFIGI ids, EODHD by ids), the stored eras carried. Writes nothing.

build_etf_delisted

build_etf_delisted(cfg: RunConfig) -> dict[str, Any]

Every ETF that has left the US venues (EODHD’s delisted list, FMP’s), minted as an inactive era with its ids; the eras the catalog holds carried. Writes nothing.

build_etf_universe

build_etf_universe(cfg: RunConfig) -> dict[str, Any]

Every ETF in FMP’s screener identified afresh (profile, etf/info, OpenFIGI ids, EODHD by ids), the stored eras carried. Writes nothing.

build_index_universe

build_index_universe(cfg: RunConfig) -> dict[str, Any]

FMP’s US indices against EODHD’s codes and ids. Writes nothing.

build_listings

build_listings(cfg: RunConfig, read_catalogs: dict[str, list[Any]]) -> dict[str, Any]

Every US listing in the window: FMP’s IPO calendar (90 days a call) and Nasdaq’s additions, combined, our ids where the catalogs hold the ticker. Writes nothing.

build_symbol_changes

build_symbol_changes(cfg: RunConfig, read_catalogs: dict[str, list[Any]]) -> dict[str, Any]

Every rename FMP and EODHD tell (one call each, whole lists), combined, our ids where the catalogs hold the security. Writes nothing.

read_catalogs

read_catalogs(cfg: RunConfig) -> dict[str, list[Any]]

The ETF and equity catalogs, every era: what gives each change its ids. Empty catalogs refuse: the ids would be null for good.

write_commodity_universe

write_commodity_universe(cfg: RunConfig, build_commodity_universe: dict[str, Any]) -> None

The commodity catalog’s new and changed eras landed.

write_constituents

write_constituents(cfg: RunConfig, build_constituents: dict[str, Any]) -> None

The membership lines landed, each index’s story made exact.

write_crypto_universe

write_crypto_universe(cfg: RunConfig, build_crypto_universe: dict[str, Any]) -> None

The crypto catalog’s new and changed eras landed.

write_delistings

write_delistings(cfg: RunConfig, build_delistings: dict[str, Any]) -> None

The delistings landed, the window made exact.

write_equity_delisted

write_equity_delisted(cfg: RunConfig, build_equity_delisted: dict[str, Any]) -> None

The dead equity eras landed in the catalog, FMP’s dated delistings in theirs.

write_equity_universe

write_equity_universe(cfg: RunConfig, build_equity_universe: dict[str, Any]) -> None

The equity catalog’s new and changed eras landed.

write_etf_delisted

write_etf_delisted(cfg: RunConfig, build_etf_delisted: dict[str, Any]) -> None

The dead ETF eras landed in the catalog, FMP’s dated delistings in theirs.

write_etf_universe

write_etf_universe(cfg: RunConfig, build_etf_universe: dict[str, Any]) -> None

The ETF catalog’s new and changed eras landed.

write_index_universe

write_index_universe(cfg: RunConfig, build_index_universe: dict[str, Any]) -> None

The index catalog’s new and changed eras landed.

write_listings

write_listings(cfg: RunConfig, build_listings: dict[str, Any]) -> None

The listings landed, the window made exact.

write_symbol_changes

write_symbol_changes(cfg: RunConfig, build_symbol_changes: dict[str, Any]) -> None

The renames landed, the whole table.

UniverseParams

Bases: JobParams

A universe job’s row params: the catalog part and the changes part, each job’s row filling its own.

UniverseUpdate

Bases: DataJob

The daily universe: the three changes tables, the three catalogs that need no changes, and the two listed catalogs from the changes. Carried state: the base’s, with params this job’s UniverseParams.

land_commodity_universe

land_commodity_universe(cfg: RunConfig, update_commodity_universe: dict[str, Any]) -> None

The commodity catalog’s new and changed eras landed; the table’s unread issue closed.

land_constituents

land_constituents(cfg: RunConfig, update_constituents: dict[str, Any]) -> None

The membership lines landed, each index’s story made exact; the table’s unread issue closed.

land_crypto_universe

land_crypto_universe(cfg: RunConfig, update_crypto_universe: dict[str, Any]) -> None

The crypto catalog’s new and changed eras landed; the table’s unread issue closed.

land_delistings

land_delistings(cfg: RunConfig, update_delistings: dict[str, Any]) -> None

The delistings landed, the window made exact; the table’s unread issue closed.

land_equity_universe

land_equity_universe(cfg: RunConfig, update_equity_universe: dict[str, Any]) -> None

The equity catalog’s new and changed eras landed (the ended first); the table’s unread issue closed.

land_etf_universe

land_etf_universe(cfg: RunConfig, update_etf_universe: dict[str, Any]) -> None

The ETF catalog’s new and changed eras landed (the ended first); the table’s unread issue closed.

land_index_universe

land_index_universe(cfg: RunConfig, update_index_universe: dict[str, Any]) -> None

The index catalog’s new and changed eras landed; the table’s unread issue closed.

land_listings

land_listings(cfg: RunConfig, update_listings: dict[str, Any]) -> None

The listings landed, the window made exact; the table’s unread issue closed.

land_symbol_changes

land_symbol_changes(cfg: RunConfig, update_symbol_changes: dict[str, Any]) -> None

The renames landed, the whole table made exact; the table’s unread issue closed.

land_untold_changes

land_untold_changes(cfg: RunConfig, update_etf_universe: dict[str, Any], update_equity_universe: dict[str, Any]) -> None

What the two updates found that no vendor told — an era opened that no listing names, a rename found by the ids alone, an era ended that no delisting explains — into the changes tables (in_universe, dated today); then every event in the window given its ids against the catalogs as they now stand, so a new era’s rows carry its id. Runs after both catalogs have landed.

listed_universe_repair_due

listed_universe_repair_due(context: Any) -> Any

As universe_changes_repair_due, for the ETF or equity catalog.

open_universe_issues

open_universe_issues(cfg: RunConfig, update_etf_universe: dict[str, Any], update_equity_universe: dict[str, Any]) -> None

What the two catalogs now show since their last landing, as issues: new_listing (build_history: the prices backfill builds the era), rename, delisting. One per era, never twice. Runs after the window’s events carry their ids.

universe_changes_repair_due

universe_changes_repair_due(context: Any) -> Any

Once a night, 02:00-04:00 New York (two hours: the night the clocks go forward has no 02:00), only while a changes table’s unread issue is open.

universe_repair_due

universe_repair_due(context: Any) -> Any

As universe_changes_repair_due, for an index, crypto or commodity catalog.

update_commodity_universe

update_commodity_universe(cfg: RunConfig) -> dict[str, Any]

Today’s commodities against the stored eras. Writes nothing; a source unread opens an issue on the table and skips.

update_constituents

update_constituents(cfg: RunConfig) -> dict[str, Any]

The three indices’ stories read whole again (a late line lands anywhere), today’s members as today’s anchor, our ids from the stored catalogs. Writes nothing; a source unread opens an issue on the table and skips.

update_crypto_universe

update_crypto_universe(cfg: RunConfig) -> dict[str, Any]

Today’s coins against the stored eras: the new identified (OpenFIGI asked for those alone), the gone ended, the rest carried with their FIGI and EODHD code. Writes nothing; a source unread opens an issue on the table and skips.

update_delistings

update_delistings(cfg: RunConfig) -> dict[str, Any]

The delistings from lookback_days back (further, to the table’s last landing, when a run was missed; Nasdaq asked 90 days further back inside), announced ones included: FMP’s delisted list and Nasdaq’s suspensions and deletions, with today’s deletes, combined, our ids from the stored catalogs. Writes nothing; a source unread opens an issue on the table and skips.

update_equity_universe

update_equity_universe(cfg: RunConfig) -> dict[str, Any]

Today’s equities against the stored eras, as update_etf_universe.

update_etf_universe

update_etf_universe(cfg: RunConfig) -> dict[str, Any]

Today’s ETFs against the stored eras, the stored changes since the catalog’s last landing explaining what came and went (renames carried onto their new ticker, listings and delistings matched): only the new, renamed and never-profiled identified. Writes nothing; a source unread opens an issue on the table and skips. Returns the changed rows, and what it found that no vendor told.

update_index_universe

update_index_universe(cfg: RunConfig) -> dict[str, Any]

Today’s indices against the stored eras: the new identified, the gone ended, the rest carried. Writes nothing; a source unread opens an issue on the table and skips.

update_listings

update_listings(cfg: RunConfig) -> dict[str, Any]

The window’s listings — FMP’s calendar and Nasdaq’s additions, with today’s adds — from lookback_days back (further, to the table’s last landing, when a run was missed) through ahead_days ahead, combined, our ids from the stored catalogs. Writes nothing; a source unread opens an issue on the table and skips.

update_symbol_changes

update_symbol_changes(cfg: RunConfig) -> dict[str, Any]

Both vendors’ whole lists again (what catches a late or corrected line), combined, our ids from the stored catalogs. Writes nothing; a source unread opens an issue on the table and skips.