massive-speedup flatfile tick pipeline
An open-source Goblin Reactor project flatfiles only · Apache-2.0

Massive.com flatfiles / Python + C++23

Market ticks,
in the order
they happened.

massive-speedup downloads, streams, indexes, and causally replays Massive.com flatfile data—so incremental models learn from the market as it arrived, not from a finished file pretending to be the present.

Get the source See the strategy template
No socket transport. No live feed client. Just durable market history.
stock_quote + stock_trade / SPY
SIP time kind price book
09:30:00.101 Q — 621.20 × .22
09:30:00.118 T 621.21 621.20 × .22
09:30:00.143 T 621.22 621.20 × .22
known now
future tape stays closed
09:30:00.144 → end of session
decision desired +1
latency +150 ms
later book fill @ ask
≈5× faster direct .csv.gz iteration than Python gzip + csv
1 tape per symbol, stored in timestamp order
0 peeks at a fill price before simulated latency expires
mmap fixed records with efficient timestamp search
01 Causal by construction

Make “crystal balling” harder.

A finished market file contains the whole day. Your algorithm should not. The iterator advances a single event at a time, updates only the state visible at that timestamp, then lets your strategy decide.

Incremental model causal

The clock moves one way.

Each observation updates persistent model state. An order intent enters the simulator now; the outcome arrives only after the tape advances through its configured delay.

09:30:00.101 observe
→
same instant update
→
decision intent
→
later fill
Batch hindsight lookahead risk

The whole day looks like “now.”

Vectorized results, session extrema, and final resamples can leak future context into an earlier decision. Even fill prices can sneak backward if simulation and signal time are not separated.

full file load all
→
future calculate
→
past decide
←
future price
DESIGN RULE

If a value did not exist at the current event’s timestamp, it does not belong in the model’s decision.

02 Orderly from source to strategy

One flatfile pipeline.
Four deliberate stages.

Keep the original archives, stream them directly when that is enough, or reorganize them into compact per-symbol databases for repeated research runs.

01 / acquire

Download in order

Fetch selected Massive.com trades and quotes into predictable dataset directories, with a human date boundary and resumable local organization.

massive-speedup-download
02 / stream

Read .csv.gz directly

Native readers decode gzip and CSV rows into typed records without routing the hot path through Python’s gzip and csv modules.

FlatFiles.Stock.Trade.parse(...)
03 / organize

Build tick databases

Convert archives into fixed-length binary records partitioned by dataset, date, and symbol—published atomically only when each build is complete.

massive-speedup-build-database
04 / replay

Seek, merge, simulate

Memory-map a symbol, search by timestamp, merge trades with quotes, and advance a market simulator through the same chronological tape.

SimpleMarket(...)
complete monthly build

Download one month.
Build every database.

Explicitly request every supported product family for the rolling month through yesterday. The builder then discovers every downloaded archive and creates its per-symbol database files.

refresh-flatfiles.sh 
#!/bin/sh
set -eu

# Choose durable, writable absolute paths.
export MASSIVE_SPEEDUP_DOWNLOAD_PATH="/home/you/market-data/massive-flatfiles"
export MASSIVE_SPEEDUP_DB_PATH="/home/you/market-data/massive-databases"

# Make AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and
# MASSIVE_API_KEY available in this script's protected environment.
# . /home/you/.config/massive-speedup/credentials

# Download every available file for every product in the last month.
massive-speedup-download \
  --products stocks currencies futures crypto options indices \
  --end-date "one month ago"

# Discover every downloaded .csv.gz and build all databases.
massive-speedup-build-database --processes 8
cron-ready

Let the rolling window refresh itself.

Save the workflow above as an executable script, then schedule it. The relative date is recalculated on every run. Existing downloads and completed database sessions are skipped, so recurring runs focus on newly available work. Cron does not inherit your interactive shell environment; provide credentials through a protected file or the job’s configured environment.

crontab 
# Save the workflow as /home/you/bin/refresh-flatfiles.sh
mkdir -p /home/you/log
chmod 700 /home/you/bin/refresh-flatfiles.sh
crontab -e

# Run every morning at 04:15 and retain an append-only log.
15 4 * * * /home/you/bin/refresh-flatfiles.sh >> /home/you/log/massive-speedup.log 2>&1

≈5× is the project’s direct-reader comparison against Python gzip + csv; actual throughput depends on data, storage, CPU, and workload.

03 Two useful representations

Stream the archive.
Or seek the tape.

Use the original compressed files for one-pass jobs. Use the native store when an experiment will revisit the same symbol and time range again and again.

Direct archive reader no extraction

Typed records from .csv.gz

Iterate trades, quotes, currencies, crypto, futures, options, and index values as read-only Python objects backed by native parsing.

read_flatfile.py 
import massive_speedup as ms

trades = ms.FlatFiles.Stock.Trade.parse(
    "stock_trade/2026-01-23.csv.gz"
)

for trade in trades:
    print(
        trade.ticker,
        trade.sip_timestamp,
        trade.price,
        trade.size,
    )
Per-symbol tick store mmap + search

Chronological and addressable

Select a symbol when opening the database, iterate in timestamp order, or jump near a nanosecond timestamp with native search.

seek_ticks.py 
import massive_speedup as ms

trades = ms.StockTradeDatabase(
    "2026-01-23", "AAPL"
)

i = trades.index_before_timestamp(
    1769178600000000000
)
trade = trades[i]

for later_trade in trades[i:]:
    consume(later_trade)

One symbol. One ordered tape.

Dataset and session select the directory. Symbol selects the file. Every fixed-size record begins with its chronological timestamp key, making the store compact, iterable, and efficiently searchable.

ticks/ ├── stock_trade/ │   └── 2026-01-23/ │      ├── AAPL ← timestamp-ordered records │      ├── MSFT │      └── SPY └── stock_quote/     └── 2026-01-23/ …
04 Honest execution timing

The broker knows what time it is.

Every market iteration supplies a broker bound to the current symbol and timestamp. Your strategy requests a desired position. The market simulator owns quote-aware pricing, order state, and delayed processing.

01 Bid/ask, not a free midpoint. Market buys cross at the ask; market sells cross at the bid.
02 Decision time is not fill time. A configurable delay separates intent from execution eligibility.
03 The fill stays unknowable. The strategy cannot inspect a later fill price while making the earlier decision.
simulated order / unit position +1 causal tape
.143 s
observation trade 621.22 · NBBO 621.20 × 621.22 The current event and latest book are now visible.
.143 s
strategy intent broker.set_desired(+1) The strategy asks to be long. No future execution price is returned.
+150 ms
processing delay order pending · later tape still locked Quotes continue to evolve while the decision waits.
.293 s
eligible fill buy at the then-current ask The execution becomes history only when simulated time reaches it.
Buy market order cross at ask
Sell market order cross at bid
causal_market.py 
import massive_speedup as ms

market = ms.SimpleMarket(
    "2026-01-23",
    ["AAPL", "MSFT"],
    trade_latency_ns=150_000_000,
    quotes=True,
    execution="market",
    unit_shares=1.0,
)

for symbol, ts, trade, quote, trades, quotes, broker in market:
    desired = algorithm.update(symbol, trade, quote)
    broker.set_desired(desired, symbol)

print(market.summary())

Open-source complement

Data causality meets model causality.

massive-speedup advances the historical market one event at a time. rtta supplies stateful models that update one observation at a time. Together they encourage backtests whose data, signal, decision, and execution clocks all point forward.

Explore the rtta project
05 Backtest → live

Keep the algorithm.
Swap the market.

The examples in rtta use the same event-driven strategy shape for a massive-speedup replay and a jinghong.LiveMarket session. The seven-part iteration contract keeps market plumbing out of the model.

strategy.py · fill in the marked algorithm block 
#!/usr/bin/env python3
"""One strategy loop: massive-speedup replay or jinghong.LiveMarket."""

import argparse
import math
import os

import massive_speedup
import rtta


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--symbol", default="SPY")
    parser.add_argument("--date", help="historical YYYY-MM-DD")
    parser.add_argument("--live", action="store_true")
    parser.add_argument("--execute", action="store_true",
                        help="send orders; live mode is dry-run by default")
    parser.add_argument("--live-account", action="store_true",
                        help="use the live endpoint instead of paper")
    parser.add_argument("--redis-socket",
                        default="/tmp/jinghong-redis.sock")
    parser.add_argument("--channel", default="polygon.ws")
    args = parser.parse_args()
    args.symbol = args.symbol.strip().upper()

    if args.live:
        # Live transport is separate; massive-speedup stays flatfile-only.
        import jinghong

        market = jinghong.LiveMarket(
            symbols=[args.symbol],
            redis_path=args.redis_socket,
            channel=args.channel,
            quotes=True,
            execution="market",
            passivity=0.0,
            unit_shares=1.0,
            dry_run=not args.execute,
            alpaca_key=os.environ.get("APCA_API_KEY_ID", ""),
            alpaca_secret=os.environ.get("APCA_API_SECRET_KEY", ""),
            paper=not args.live_account,
        )
    else:
        if not args.date:
            parser.error("--date is required unless --live is set")
        market = massive_speedup.SimpleMarket(
            args.date,
            [args.symbol],
            trade_latency_ns=150_000_000,
            quotes=True,
            execution="market",
            unit_shares=1.0,
        )

    # Replace this example with your own incremental rtta model and state.
    model = rtta.FourierResidueIdentity(
        max_lag=8,
        horizon=2,
        test_lag=1,
        span=2048,
        median_window=512,
        entry_z=2.5,
        exit_z=1.0,
        fillna=False,
    )
    chained = 100.0
    last_price = None
    desired = 0.0

    try:
        for event in market:
            symbol, timestamp, trade, quote, trades, quotes, broker = event

            # Your algorithm goes here. Only use this event and prior state.
            if trade is not None:
                price = float(trade.price)
                if last_price is not None:
                    chained *= price / last_price
                last_price = price

                output = model.update(chained)
                if math.isfinite(output.z_sign):
                    if output.z_sign >= 2.5:
                        desired = 1.0
                    elif output.z_sign <= 1.0:
                        desired = 0.0

            broker.set_desired(desired, symbol)
    finally:
        if args.live:
            market.stop()

    if not args.live:
        print(market.summary())


if __name__ == "__main__":
    main()
Backtest python strategy.py --symbol SPY --date 2026-01-23
Live dry-run python strategy.py --symbol SPY --live
Paper orders python strategy.py --symbol SPY --live --execute

Keep history fast.
Keep the future out.

Download the flatfiles once, turn them into durable ordered tapes, and build incremental models against the market that was actually knowable.

View on GitHub