Databases

Storage and binary records

Daily databases memory-map headerless, fixed-size records for one instrument and date. They return the same record classes as the CSV readers: field names, types, conditions, and packing are identical. The native extension is required.

The first eight bytes of each record encode its primary timestamp as an unsigned 64-bit big-endian integer. Remaining numeric fields use the record's packed layout; other multibyte numeric fields are little-endian. Identity comes from the file path. Refer to the record sizes, offsets, and restoration methods.

Database open never downloads or backfills data. Stock, option, and index readers consult the NYSE calendar: a non-session opens as an empty database before checking the filesystem. On a session date, a missing instrument file raises RuntimeError. Crypto, currency, and futures readers require the file directly. Incorrect record-size multiples raise ValueError.

Use the same package format to build and read databases. Rebuild older-format files with --force after a binary layout change. Native daily objects have no public close() or context-manager API; mappings live with their objects and dependent iterators.

Paths, keys, and download options

Set writable absolute paths or pass explicit path overrides. Opening local databases does not use the REST or S3 credentials. The quick start shows the exports.

massive-speedup-download [OPTIONS]

Database conversion CLI

massive-speedup-build-database --processes 4

With no positional input, the builder discovers the download root. Otherwise pass one or more files or directories. Input files must be named exactly YYYY-MM-DD.csv.gz. The CSV header determines the record type; recognized parent directory names select futures exchange variants and enable family filtering.

Each ticker is written to an .incomplete file and renamed to its final name only after it is complete. A date directory carries .massive-speedup.incomplete during construction and .massive-speedup.complete after success. A nonempty destination with no incomplete entries can be skipped before decompressing its source. Readers do not open incomplete files.

Normal reruns preserve existing completed ticker files. --force publishes replacements atomically. Local external-sort staging is cleaned after success, failure, or interruption; remaining destination markers tell the next build to resume/rebuild incomplete work.

Python conversion API

ms.build_database_file(input_path, record_type, *, database_path=None,
    force=False, write_lock=None, block_size=1048576,
    reader_parallelization=1, sort_records=True) -> int

ms.build_database_file_inferred(input_path, *, database_path=None,
    force=False, write_lock=None, block_size=1048576,
    reader_parallelization=1, sort_records=True) -> tuple[str, int]

ms.build_database_file_inferred_with_stats(input_path, *, database_path=None,
    force=False, write_lock=None, block_size=1048576,
    reader_parallelization=1, sort_records=True) -> tuple[str, int, int]

The Python module massive_speedup.build_database exposes infer_record_type(input_path: Path) -> str and matching wrappers write_database_file, write_database_file_inferred, write_database_file_inferred_with_stats with the same parameters and return values. Header inference rejects unsupported schemas; futures exchange identity is inferred from recognized directory names. CLI discovery, date-directory completion checks, worker orchestration, and external bsort are managed by the CLI layer.

import os
from pathlib import Path
import massive_speedup as ms

path = Path(os.environ["MASSIVE_SPEEDUP_DOWNLOAD_PATH"])
kind, written, processed = ms.build_database_file_inferred_with_stats(
    path / "stock_trade" / "2026-09-11.csv.gz"
)
print(kind, written, processed)

StockTradeDatabase

ms.StockTradeDatabase(date: object, ticker: str, *, database_path: str | os.PathLike | None = None)

Returns a daily database of StockTrade records. The full read-only record schema is shared with the CSV parser. Time order is sip_timestamp.

Opens stock_trade/YYYY-MM-DD/TICKER below the database root. ticker must exactly match the stored filename, including any instrument prefix.

Alongside the common daily API: find_after_participant_timestamp, find_before_participant_timestamp, market_close, market_open, ticker.

StockQuoteDatabase

ms.StockQuoteDatabase(date: object, ticker: str, *, database_path: str | os.PathLike | None = None)

Returns a daily database of StockQuote records. The full read-only record schema is shared with the CSV parser. Time order is sip_timestamp.

Opens stock_quote/YYYY-MM-DD/TICKER below the database root. ticker must exactly match the stored filename, including any instrument prefix.

Alongside the common daily API: find_after_participant_timestamp, find_before_participant_timestamp, market_close, market_open, ticker.

CryptoTradeDatabase

ms.CryptoTradeDatabase(date: object, ticker: str, *, database_path: str | os.PathLike | None = None)

Returns a daily database of CryptoTrade records. The full read-only record schema is shared with the CSV parser. Time order is participant_timestamp.

Opens crypto_trade/YYYY-MM-DD/TICKER below the database root. ticker must exactly match the stored filename, including any instrument prefix.

Alongside the common daily API: find_after_participant_timestamp, find_before_participant_timestamp, ticker.

CurrencyQuoteDatabase

ms.CurrencyQuoteDatabase(date: object, ticker: str, *, database_path: str | os.PathLike | None = None)

Returns a daily database of CurrencyQuote records. The full read-only record schema is shared with the CSV parser. Time order is participant_timestamp.

Opens currency_quote/YYYY-MM-DD/TICKER below the database root. ticker must exactly match the stored filename, including any instrument prefix.

Alongside the common daily API: find_after_participant_timestamp, find_before_participant_timestamp, ticker.

IndexValueDatabase

ms.IndexValueDatabase(date: object, ticker: str, *, database_path: str | os.PathLike | None = None)

Returns a daily database of IndexValue records. The full read-only record schema is shared with the CSV parser. Time order is timestamp.

Opens index_value/YYYY-MM-DD/TICKER below the database root. ticker must exactly match the stored filename, including any instrument prefix.

Alongside the common daily API: find_after_participant_timestamp, find_before_participant_timestamp, ticker.

FuturesTradeDatabase

ms.FuturesTradeDatabase(date: object, ticker: str, *, exchange: str = '', database_path: str | os.PathLike | None = None)

Returns a daily database of FuturesTrade records. The full read-only record schema is shared with the CSV parser. Time order is timestamp.

exchange accepts "", "cbot", "cme", "comex", or "nymex". Empty selects the generic future_trade directory; it does not auto-detect an exchange. Nonempty selects future_EXCHANGE_trade. date is the session end date.

Alongside the common daily API: ticker.

FuturesQuoteDatabase

ms.FuturesQuoteDatabase(date: object, ticker: str, *, exchange: str = '', database_path: str | os.PathLike | None = None)

Returns a daily database of FuturesQuote records. The full read-only record schema is shared with the CSV parser. Time order is timestamp.

exchange accepts "", "cbot", "cme", "comex", or "nymex". Empty selects the generic future_quote directory; it does not auto-detect an exchange. Nonempty selects future_EXCHANGE_quote. date is the session end date.

Alongside the common daily API: ticker.

OptionTradeDatabase

ms.OptionTradeDatabase(date: object, root: str, expiration: str, right: str, strike: float, *, database_path: str | os.PathLike | None = None)

Returns a daily database of OptionTrade records. The full read-only record schema is shared with the CSV parser. Time order is sip_timestamp.

root is the underlying symbol; expiration is a YYYY-MM-DD string; right must be "C" or "P"; strike is a finite, nonnegative price rounded to thousandths (maximum encoded strike 99,999.999). The path key is ROOT/EXPIRATION/RIGHT/SSSSSSSS. For example, ("AAPL", "2026-09-18", "C", 150.0) produces AAPL/2026-09-18/C/00150000.

Alongside the common daily API: contract_key, expiration, right, root, strike, strike_millis.

OptionQuoteDatabase

ms.OptionQuoteDatabase(date: object, root: str, expiration: str, right: str, strike: float, *, database_path: str | os.PathLike | None = None)

Returns a daily database of OptionQuote records. The full read-only record schema is shared with the CSV parser. Time order is sip_timestamp.

root is the underlying symbol; expiration is a YYYY-MM-DD string; right must be "C" or "P"; strike is a finite, nonnegative price rounded to thousandths (maximum encoded strike 99,999.999). The path key is ROOT/EXPIRATION/RIGHT/SSSSSSSS. For example, ("AAPL", "2026-09-18", "C", 150.0) produces AAPL/2026-09-18/C/00150000.

Alongside the common daily API: contract_key, expiration, right, root, strike, strike_millis.

Daily iteration, metadata, and timestamp search

db.index_before_timestamp(timestamp, *, galloping=None) -> int
db.index_after_timestamp(timestamp, *, galloping=None) -> int
db.iterate_bounded(start_timestamp) -> Iterator[Record]
db.iterate_bounded(start_timestamp, stop_timestamp) -> Iterator[Record]

galloping=None uses binary search. Supply an integer index from a nearby earlier lookup to accelerate clustered searches; an out-of-range hint is clamped and does not change the result. Hints are row indexes, not timestamps.

Check the -1 sentinel before indexing. db[-1] is a valid request for the last record of the day and can leak future information when mistaken for 'no earlier record'. For time windows, use iterate_bounded; db[i:] is not supported.

import massive_speedup as ms

trades = ms.StockTradeDatabase("2026-09-11", "AAPL")
start = trades.market_open
stop = start + 60 * 1_000_000_000 - 1
for trade in trades.iterate_bounded(start, stop):
    print(trade.sip_timestamp, trade.price)

Timestamp arguments and participant searches

Daily timestamp arguments accept nonnegative integer or finite float nanoseconds, or datetime.time. Prefer integer nanoseconds to preserve precision. A time is combined with the database date; a naive time means UTC, not New York. For exchange-local times supply tzinfo=ZoneInfo(...).

Futures time-of-day lookup combines the session end date and time, checks its hour in America/Chicago, and shifts to the previous calendar date when that hour is 17 or later. Use an absolute integer timestamp when you need to avoid this session-date conversion.

db.find_before_participant_timestamp(timestamp, fuzz=1000000000,
    *, galloping=None, on=True) -> Record
db.find_after_participant_timestamp(timestamp, fuzz=1000000000,
    *, galloping=None, on=True) -> Record

Available on stock trade/quote, crypto trade, currency quote, and index value databases. For index values, the compatibility participant search uses the record's timestamp. Futures and options do not expose these methods.

A participant-time match can have a later SIP receipt time. It is not automatically safe for a decision keyed to SIP time. Use the primary SIP search for 'what was known then', or check the returned SIP timestamp explicitly. A small fuzz window can also miss events whose participant/SIP delay is larger.

from datetime import time
from zoneinfo import ZoneInfo
import massive_speedup as ms

quotes = ms.StockQuoteDatabase("2026-09-11", "AAPL")
i = quotes.index_before_timestamp(time(9, 30, tzinfo=ZoneInfo("America/New_York")))
if i >= 0:
    print(quotes[i].bid_price, quotes[i].ask_price)

MultiDayDatabase

ms.MultiDayDatabase(record_type: str, key: str, *, database_path=None,
    start_date=None, end_date=None, max_open_days=1)

Discovers daily files for one key and presents them as one sequence, in date order. start_date and end_date are inclusive ISO strings or datetime.date values. Both default to no bound. max_open_days must be a positive integer; it bounds the wrapper's least-recently-used daily-object cache, not additional references your code retains.

Accepts stock_trade, stock_quote, crypto_trade, currency_quote, index_value, option_trade, option_quote, future_trade, future_quote, and the exchange-specific future_EXCHANGE_trade/quote names. key is the ticker filename or the option key ROOT/YYYY-MM-DD/C|P/SSSSSSSS. Absolute keys, '..', and malformed contract keys are rejected.

db.iterate_bounded(start_timestamp, stop_timestamp=None)
db.locate_before_timestamp(timestamp, *, on=True, galloping=None)
db.locate_after_timestamp(timestamp, *, on=True, galloping=None)
db.find_before_timestamp(timestamp, *, on=True, galloping=None)
db.find_after_timestamp(timestamp, *, on=True, galloping=None)

Multi-day timestamp arguments accept integer/float nanoseconds, datetime.date (UTC midnight), or datetime.datetime (naive means UTC). They do not accept datetime.time. iterate_bounded uses inclusive bounds and raises ValueError for a reversed range. locate_* returns DatabaseLocation; find_* returns its record. Both directions raise IndexError if there is no match. on=False makes the comparison strict. galloping is a previous DatabaseLocation; its local index is reused only for the matching date.

ms.DatabaseLocation(date: str, index: int, record: object)

A frozen dataclass with date (daily ISO date), index (local row index), and record (the typed record). It can be retained as a reusable search hint.

from datetime import datetime, timezone
import massive_speedup as ms

with ms.MultiDayDatabase(
    "stock_trade", "AAPL",
    start_date="2026-09-07", end_date="2026-09-11", max_open_days=2,
) as trades:
    target = datetime(2026, 9, 11, 14, 0, tzinfo=timezone.utc)
    try:
        hit = trades.locate_before_timestamp(target)
        print(hit.date, hit.index, hit.record.price)
    except IndexError:
        print("No trade at or before that time.")

Stock trade/quote timeline

ms.StockTradeQuoteTimeline(date, ticker, *, database_path=None)
ms.stock_trade_quote_timeline(date, ticker, *, database_path=None)

Both return a forward iterator merging one stock's trade and quote databases in SIP order. This helper is stock-only and has no broker. A quote yields (None, current_quote); a trade yields (current_trade, most_recent_quote_or_None). At an equal SIP timestamp, the quote comes first. The quote attached to a trade is the latest quote already observed, not a quote fetched from a future row.

import massive_speedup as ms

for trade, quote in ms.stock_trade_quote_timeline("2026-09-11", "AAPL"):
    if trade is not None and quote is not None:
        print(trade.price, quote.bid_price, quote.ask_price)

For execution simulation, see SimpleMarket, FuturesMarket, and OptionMarket.