CSV.gz readers

Typed readers, raw fields, and lines

Pass a local gzip path to the parser for the file's dataset. parse skips the CSV header and empty rows and yields read-only record objects. parse_raw yields tuples of bytes in original CSV column order. Neither requires an extracted CSV or a database.

import os
from pathlib import Path
import massive_speedup as ms

path = Path(os.environ["MASSIVE_SPEEDUP_DOWNLOAD_PATH"])
path = path / "stock_trade" / "2026-09-11.csv.gz"
rows = ms.FlatFiles.Stock.Trade.parse(path, sort_by_sip_timestamp=True)
for trade in rows:
    if trade.ticker == "AAPL":
        print(trade.sip_timestamp, trade.price, trade.size)
        break

Unsorted readers preserve file order. A sorting flag loads the file's rows into memory before iteration and orders them by the selected timestamp. Stock flags are mutually exclusive. Currency's sort_by_sip_timestamp parameter must remain False; setting it raises ValueError. Futures readers accept only the path; use the database builder to sort futures tapes.

Paths must contain the expected dataset schema. Malformed fields or incorrect field counts raise errors. Integer timestamps preserve nanosecond precision; use SIP order for stock/option observations that model when the consolidated feed became available.

Gzip line API

ms.gzip_lines(path, parallelization=0, chunk_size=1048576)
ms.read_gzip_lines(path, parallelization=0, chunk_size=1048576)
ms.read_gzip_lines_bytes(path, parallelization=0, chunk_size=1048576)

parallelization=0 selects the decoder's automatic parallelism; a positive integer supplies a decoder thread count. chunk_size is the read chunk size in bytes (default 1 MiB). Only the currency typed/raw field readers expose a parallelization keyword directly; other typed parsers use their internal defaults.

The asset-level convenience names FlatFiles.Stock.parse_trades, FlatFiles.Stock.parse_quotes, FlatFiles.Crypto.parse_trades, FlatFiles.currency.parse_quotes, FlatFiles.Indices.parse_values, and the corresponding option/futures parse_trades / parse_quotes call the same typed readers. Where supported, parse_raw_trades, parse_raw_quotes, and parse_raw_values mirror parse_raw.

StockTrade

ms.FlatFiles.Stock.Trade yields ms.StockTrade. Primary database/aggregation time key: sip_timestamp.

ms.FlatFiles.Stock.Trade.parse(path: str | os.PathLike, *, sort_by_participant_timestamp: bool = False, sort_by_sip_timestamp: bool = False) -> Iterator[StockTrade]

ms.FlatFiles.Stock.Trade.parse_raw(path: str | os.PathLike, *, sort_by_participant_timestamp: bool = False, sort_by_sip_timestamp: bool = False) -> Iterator[tuple[bytes, ...]]

ms.FlatFiles.Stock.Trade.raw_lines(path: str | os.PathLike) -> Iterator[bytes]

CSV column order:

ticker,conditions,correction,exchange,id,participant_timestamp,price,sequence_number,sip_timestamp,size,tape,trf_id,trf_timestamp

decimal_size, size_coefficient, and size_scale are derived accessors, not additional CSV columns. Use them when exact fractional-share quantities matter.

Open StockTrade records from a database →

StockQuote

ms.FlatFiles.Stock.Quote yields ms.StockQuote. Primary database/aggregation time key: sip_timestamp.

ms.FlatFiles.Stock.Quote.parse(path: str | os.PathLike, *, sort_by_participant_timestamp: bool = False, sort_by_sip_timestamp: bool = False) -> Iterator[StockQuote]

ms.FlatFiles.Stock.Quote.parse_raw(path: str | os.PathLike, *, sort_by_participant_timestamp: bool = False, sort_by_sip_timestamp: bool = False) -> Iterator[tuple[bytes, ...]]

ms.FlatFiles.Stock.Quote.raw_lines(path: str | os.PathLike) -> Iterator[bytes]

CSV column order:

ticker,ask_exchange,ask_price,ask_size,bid_exchange,bid_price,bid_size,conditions,indicators,participant_timestamp,sequence_number,sip_timestamp,tape,trf_timestamp

Open StockQuote records from a database →

CryptoTrade

ms.FlatFiles.Crypto.Trade yields ms.CryptoTrade. Primary database/aggregation time key: participant_timestamp.

ms.FlatFiles.Crypto.Trade.parse(path: str | os.PathLike, *, sort_by_participant_timestamp: bool = False) -> Iterator[CryptoTrade]

ms.FlatFiles.Crypto.Trade.parse_raw(path: str | os.PathLike, *, sort_by_participant_timestamp: bool = False) -> Iterator[tuple[bytes, ...]]

ms.FlatFiles.Crypto.Trade.raw_lines(path: str | os.PathLike) -> Iterator[bytes]

CSV column order:

ticker,conditions,exchange,id,participant_timestamp,price,size

Open CryptoTrade records from a database →

CurrencyQuote

ms.FlatFiles.currency.Quote yields ms.CurrencyQuote. Primary database/aggregation time key: participant_timestamp.

ms.FlatFiles.currency.Quote.parse(path: str | os.PathLike, *, sort_by_participant_timestamp: bool = False, sort_by_sip_timestamp: bool = False, parallelization: int = 0) -> Iterator[CurrencyQuote]

ms.FlatFiles.currency.Quote.parse_raw(path: str | os.PathLike, *, sort_by_participant_timestamp: bool = False, sort_by_sip_timestamp: bool = False, parallelization: int = 0) -> Iterator[tuple[bytes, ...]]

ms.FlatFiles.currency.Quote.raw_lines(path: str | os.PathLike) -> Iterator[bytes]

CSV column order:

ticker,ask_exchange,ask_price,bid_exchange,bid_price,participant_timestamp

tickers is derived and does not add a CSV column. parallelization=0 uses automatic decoder parallelism; supply a positive count to cap it.

Open CurrencyQuote records from a database →

IndexValue

ms.FlatFiles.Indices.Value yields ms.IndexValue. Primary database/aggregation time key: timestamp.

ms.FlatFiles.Indices.Value.parse(path: str | os.PathLike, *, sort_by_timestamp: bool = False) -> Iterator[IndexValue]

ms.FlatFiles.Indices.Value.parse_raw(path: str | os.PathLike, *, sort_by_timestamp: bool = False) -> Iterator[tuple[bytes, ...]]

ms.FlatFiles.Indices.Value.raw_lines(path: str | os.PathLike) -> Iterator[bytes]

CSV column order:

ticker,value,timestamp

Open IndexValue records from a database →

FuturesTrade

ms.FlatFiles.Futures.Trade yields ms.FuturesTrade. Primary database/aggregation time key: timestamp.

ms.FlatFiles.Futures.Trade.parse(path: str | os.PathLike) -> Iterator[FuturesTrade]

CSV column order:

ticker,timestamp,sequence_number,report_sequence,price,size,correction,exchange,session_end_date

Open FuturesTrade records from a database →

FuturesQuote

ms.FlatFiles.Futures.Quote yields ms.FuturesQuote. Primary database/aggregation time key: timestamp.

ms.FlatFiles.Futures.Quote.parse(path: str | os.PathLike) -> Iterator[FuturesQuote]

CSV column order:

ticker,timestamp,sequence_number,report_sequence,ask_timestamp,ask_price,ask_size,bid_timestamp,bid_price,bid_size,exchange,session_end_date

Open FuturesQuote records from a database →

OptionTrade

ms.FlatFiles.Options.Trade yields ms.OptionTrade. Primary database/aggregation time key: sip_timestamp.

ms.FlatFiles.Options.Trade.parse(path: str | os.PathLike, *, sort_by_sip_timestamp: bool = False) -> Iterator[OptionTrade]

ms.FlatFiles.Options.Trade.parse_raw(path: str | os.PathLike, *, sort_by_sip_timestamp: bool = False) -> Iterator[tuple[bytes, ...]]

ms.FlatFiles.Options.Trade.raw_lines(path: str | os.PathLike) -> Iterator[bytes]

CSV column order:

ticker,conditions,correction,exchange,price,sip_timestamp,size

The raw CSV ticker is decoded into root, expiration, right, and strike. Typed option records have no ticker attribute. list(record) starts with those four fields, followed by the remaining CSV columns.

Open OptionTrade records from a database →

OptionQuote

ms.FlatFiles.Options.Quote yields ms.OptionQuote. Primary database/aggregation time key: sip_timestamp.

ms.FlatFiles.Options.Quote.parse(path: str | os.PathLike, *, sort_by_sip_timestamp: bool = False) -> Iterator[OptionQuote]

ms.FlatFiles.Options.Quote.parse_raw(path: str | os.PathLike, *, sort_by_sip_timestamp: bool = False) -> Iterator[tuple[bytes, ...]]

ms.FlatFiles.Options.Quote.raw_lines(path: str | os.PathLike) -> Iterator[bytes]

CSV column order:

ticker,ask_exchange,ask_price,ask_size,bid_exchange,bid_price,bid_size,sequence_number,sip_timestamp

The raw CSV ticker is decoded into root, expiration, right, and strike. Typed option records have no ticker attribute. list(record) starts with those four fields, followed by the remaining CSV columns.

Open OptionQuote records from a database →

Record construction, packing, and comparison

record = ms.StockTrade(fields)  # sequence of CSV field strings
packed = record.pack()
restored = ms.StockTrade.from_packed(packed, record.ticker)

Every record class accepts fields: Sequence[str] in its CSV schema order, supports pack() -> bytes, from_packed(...), list(record), equality, hashing, and text representation. Fields are read-only. For options, iteration replaces the source ticker with the four decoded contract fields. All other record iteration follows CSV field order; derived accessors are not appended.

Row.from_packed(packed: bytes, ticker: str) -> Row
# OptionTrade / OptionQuote use contract identity instead:
Row.from_packed(packed: bytes, root: str, expiration: str,
                right: str, strike: float) -> Row

These packed arguments also work as constructor overloads on every record class except CryptoTrade, which exposes only the from_packed method for restoration. Pass exactly Row.packed_size bytes. Ticker/contract identity is supplied separately because it is not in the packed bytes.

Stock, crypto, and currency record ordering operators compare participant timestamps; options compare SIP timestamps; index values compare their timestamp. Futures records do not define ordering operators: sort explicitly with key=lambda row: row.timestamp. Equality compares the record fields, not merely the time.

StockQuotes is an alias of StockQuote. Record classes are also exposed under FlatFiles for stock, crypto, currency, option, and index records; the reader APIs remain the nested Trade, Quote, and Value namespaces. Asset serialize() returns parser metadata as a string and processor_name() returns its processor label. FlatFiles.Forex is a legacy payload-summary parser, not an alias for FlatFiles.currency.Quote.

Stock condition enums and update rules

StockTrade.conditions and StockQuote.conditions are frozensets of the corresponding IntEnum members when a code is known, and integers otherwise. Stock quote indicators remain integers. Enum members compare to integers; members from different condition enum classes compare unequal and cannot be ordered against each other.

For quotes these helpers evaluate the union of conditions and indicators. With no exclusion rule present, the corresponding helper returns true. StockTradeAggregator uses these eligibility rules for price and size calculations. Quote aggregators consume the supplied quote values directly; use the quote helper methods to filter first if your calculation requires that policy.

Window aggregators for every record type

Aggregator(rows, interval_seconds: float, *, start_timestamp=None)
# Stock trades can also attach a broker:
ms.StockTradeAggregator(rows, interval_seconds, *, start_timestamp=None,
                        quotes=None, trade_emulator=None)

rows can be the matching native daily database or an iterable of matching record objects. interval_seconds must be positive and finite and resolve to at least one nanosecond; fractional seconds are supported. start_timestamp excludes earlier rows and anchors window boundaries; None anchors at epoch zero. A matching daily database also accepts datetime.time with the database date; other iterables require numeric nanoseconds.

Aggregation combines consecutive (instrument, window) groups. Use per-instrument, time-ordered input; globally interleaved symbols can produce multiple partial bars for the same window. Windows are [start, start + interval), empty windows are not emitted, and the last nonempty partial window is emitted at EOF. A completed bar belongs at the end of its window, never at its opening timestamp.

Crypto, futures, and option trade result classes alias StockTradeAggregation. Futures and option quote results alias StockQuoteAggregation. IndexValueAggregation aliases ValueAggregation. Only StockTradeAggregator accepts quotes and trade_emulator; provide both together. Its broker and trade_emulator accessors are documented in the bar broker section.

import massive_speedup as ms

trades = ms.StockTradeDatabase("2026-09-11", "AAPL")
bars = ms.StockTradeAggregator(
    trades, interval_seconds=60, start_timestamp=trades.market_open
)
for bar in bars:
    print(bar.window_start, bar.close, bar.volume_weighted_avg)

Result attributes are read-only and lazily converted to Python values. Non-finite observations are excluded from price statistics; undefined means, returns, or weighted means can be NaN. Standard deviation is population standard deviation. Quote volume fields sum displayed sizes across updates, not executed share volume.

StockTradeAggregation

Trade-result fields shared by stock, crypto, futures, and options. Stock condition rules can select different rows for OHLC and volume.

StockQuoteAggregation

Quote-result fields. Spread = ask − bid; mid = (ask + bid) / 2. Currency quotes have no size, volume, or size-weighted fields.

CurrencyQuoteAggregation

Quote-result fields. Spread = ask − bid; mid = (ask + bid) / 2. Currency quotes have no size, volume, or size-weighted fields.

ValueAggregation

Index-value result fields; no trade size or volume is present.