Operations¶

中文页面

This page is the operations index. Detailed procedures are maintained by topic so credentials, A-share ingestion, backups, and local governance remain easy to find.

Task	Guide
Configure the shared data root and provider credentials	Credentials
Run A-share / TuShare raw, clean, universe, current-data refresh, and raw-to-PIT workflows	A-share TuShare
Fetch, build, publish, and inspect China macro and industrial context data	Context data
Build consistent PIT features for mutual-fund top-10 holdings	Fund top-10 ownership features
Fuse Guan and TuShare A-share minute data	A-share minutes
Archive public-source ETF minute data	ETF minutes
Restore the retired Hong Kong archive	Hong Kong archive recovery
Create local snapshot backups and run development/governance checks	Backup and development
Build a publishable Python package	Package publishing
Audit current/latest paths and plan a retention dry run	Data governance
Prepare unpublished suspension and dividend research inputs	Execution input readiness
Understand test scripts and coverage	Testing operations

Common command groups¶

marketdata paths
marketdata contract build / inspect
marketdata registry build
marketdata data catalog / materialize / query
marketdata context fetch / build / publish / inspect
marketdata data build-guan-annual-minutes
marketdata data build-guan-deal-minutes
marketdata data finalize-a-share-minute-coverage
marketdata data mirror-public-etf-minute
marketdata tushare backfill-etf-history
marketdata tushare build-etf-daily-forward-adjusted
marketdata tushare validate-etf-daily-pair
marketdata governance audit-current-paths / plan-retention
marketdata tushare plan-a-share-minute-backfill
marketdata tushare run-a-share-minute-backfill
marketdata tushare build-a-share-fund-top10-portfolio-features
marketdata tushare validate-a-share-fund-top10-portfolio-features
marketdata backup-data

Public CLI documentation tests derive reachable commands from the parser. When adding a command, document it in the relevant topic page and update the tests. Historical minute backfill coverage and resume limits are described in the A-share minute guide.

Capture daily-clean raw inputs¶

marketdata governance snapshot-clean-inputs --artifacts-root "$DATA_PLATFORM_ROOT" --start-date YYYYMMDD --end-date YYYYMMDD --out-dir NEW_PATH creates an independent, date-filtered snapshot of daily, adjustment, daily-basic and limit-status inputs. Serialize raw writers first. The output must be a new path under the data root. A completed receipt pins source manifests and each captured file's SHA-256. Failed attempts are preserved without a completed receipt.

Add --reuse-from PRIOR_SNAPSHOT to reuse byte-identical read-only shards from a completed, non-retired canonical timestamp snapshot under the same inputs root. The complete donor inventory and receipt hashes/sizes are verified first. Only immutable snapshot shards share hardlinks; raw inputs retain separate inodes and new manifests/receipts remain independent. Changed/new shards are copied. Only cross-device hardlink errors fall back to copying; other errors fail closed.

Publishers holding their existing publication lock can use the mutually exclusive --reuse-latest-completed option. Output must be a direct child of assets/tushare/a_share/daily_clean_inputs. Selection uses capture timestamp and numeric suffix. It skips historical archive symlinks (including dangling links) without resolving or reading their targets, retired directories and unreceipted attempts, and rejects invalid real completed candidates. Explicit --reuse-from still rejects symlinks and requires a canonical real snapshot directory. With no candidate, capture copies every shard. No old directories or latest publication representation are changed.

Receipt storage records donor receipt fingerprint, reused/copied file counts and logical byte counts; per-file storage_method identifies hardlink or copy. These counts do not measure physical disk savings. Published snapshot shards must remain read-only: corrections must write new files, never overwrite shared inodes. Linking changes donor link count and ctime without changing bytes, permissions or mtime. Removing one linked filename leaves other snapshots intact; retirement receipt restoration dependencies remain protected. Existing readers and producer rollbacks can continue reading the same layout.

Scheduled retention apply is disabled; use the reviewed lifecycle retirement conditions in data governance before any data removal.

Private JSON configuration¶

Use one private configuration copied from config/config.example.json. Keep credentials outside Git and set mode 0600. DATA_PLATFORM_CONFIG selects the file; when unset, an existing ${XDG_CONFIG_HOME:-$HOME/.config}/quant-market-data-platform/config.json is selected. An invalid selected file stops the command. Existing process variables take precedence, including empty values. Null entries are unconfigured. Only DATA_PLATFORM_ROOT expands ${HOME} or ~; secret strings remain opaque. Legacy env files are read only when no JSON is selected.

marketdata config check --config "$DATA_PLATFORM_CONFIG"
marketdata config run --config "$DATA_PLATFORM_CONFIG" -- python /path/to/job.py

config check reports names and configured booleans, without setting values or network requests. config run directly replaces the process using argv and preserves child exit status and signals. Services use an immutable installed release and a non-secret configuration path.

QuantZone research factors¶

See QuantZone operations.

marketdata quantzone check --config "$DATA_PLATFORM_CONFIG"
marketdata quantzone download-factors --config "$DATA_PLATFORM_CONFIG" --dry-run
marketdata quantzone download-factors --config "$DATA_PLATFORM_CONFIG"

For multi-vintage research inputs, use read_statement_observations(..., dataset="income", columns=[...]) to project required fields. Include identity, disclosure, observation, availability and source-hash fields when downstream code selects revisions. The reader verifies all partition checksums before applying filters.

Resume THS constituent mirrors¶

marketdata tushare mirror-a-share-ths-member --out-dir <output> --skip-existing explicitly reuses validated concept cache partitions and fetches remaining concepts. Nonempty output directories are rejected by default. Resume with the same fields and provider configuration; constituents represent the acquisition snapshot.

Single-file reference manifests¶

Reference publication writes a sibling .manifest.yml bound to the owner receipt and payload hash, with counts, dates and source semantics. Partial constraint provenance remains partial. Refresh the receipt and manifest whenever the reference payload changes.

Reference version_date records the version label date. When a source end_date is present, manifest freshness and query end use that actual source coverage date.

Institutional trading amount units¶

TuShare top_inst reports buy, sell, and net_buy in yuan. top_inst_events converts these amounts to ten-thousand yuan before aggregation and rolling calculations, matching daily.amount / 10 from its thousand-yuan source unit. Field names remain stable; assets built before this correction must be rebuilt before using their amount features or turnover ratios. Counts and provider rates are unchanged.

The quality ratchet budget increases by one test file and 58 Python lines for this conversion and its independent arithmetic regression. Existing complexity thresholds remain unchanged. Remove this budget only when replacing the test with equivalent coverage without adding source or test lines.

Institutional survey pagination¶

marketdata tushare mirror-a-share-stk-surv fetches each calendar date using limit=400 and increasing offset. A date is written only after a short or empty terminal page. The request interval applies to every page, and retries retain the existing provider policy. Repeated pages, oversized responses, inconsistent survey dates and the 1,000-page bound fail with a partial manifest. Other event mirrors retain their existing behavior.

--skip-existing requires a matching pagination.json receipt bound to the payload hash, exact query and page counts. Existing survey payloads without that evidence must be preserved and refetched into a new directory. Completed pagination establishes observed retrieval coverage, not a revision-safe historical observation. The official survey endpoint documentation states the 400-row cap and supports cyclic or paginated extraction. A proxy ignoring pagination is rejected when pages repeat; no provider endpoint stability beyond the recorded responses is asserted.

The statement observation ledger builder also accepts completed non-VIP per-security query manifests. Preserve original manifests and date claims when recovering old caches, verify archive integrity, and record actual recovery timestamps and hashes. If original partition content seals are absent, do not combine newly computed hashes with historical date claims to imply verified earlier observations.

Use --no-proxy on either QuantZone command to connect directly when an inherited HTTP or SOCKS proxy is unsuitable. The option suppresses proxy environment variables only while constructing the SDK client and restores their original values even if construction fails. Other commands keep their inherited proxy policy.

Historical minute backfill capacity¶

The reverse runner accepts --batch-size (default 20, maximum 33) and --cooldown-seconds (default 1.2). Explicit --minute-quota-allow-burst enables the shared ledger burst pool while retaining provider limits, holds, and stop conditions. The default remains the guaranteed pool. A complete 241-bar grid for 33 securities produces 7,953 rows, below the 8,000-row response cap. Longer schedules must preserve single-instance execution, resumable partitions, and independent batch acceptance.

The quality budget adds 110 counted Python lines for explicit burst settings, command arguments, and real shared-ledger regressions covering default refusal and explicit admission. File-count and complexity limits do not increase. Retire this budget only when equivalent coverage reduces source and test lines.

Overseas research forward receipts¶

Append-only collection and timing rules describe the independent module entry point and owner archive.

Minute history monitoring and gap repair¶

Run scripts/operations/tushare_minute_history.py status --artifacts-root <data-root> --runtime-root <runtime-root> for read-only progress, quota, storage and audit-block status. Counts distinguish metadata completion, independently audited dates and Beijing omissions. Expired audit bindings are reported separately. Download completion still requires publication acceptance.

After the first download pass ends, monitoring reports download_complete_with_gaps only when the completion receipt agrees with the current date inventory, deferred queue and quarantine records, and the quota ledger remains readable. The stopped acquisition may retain an old preflight snapshot; missing, corrupt or future-dated evidence still fails closed. Audit blocks, low-storage protection and semantic-quarantine alerts remain active. Completion does not set full_market_ready to true.

watch adds --state-root <monitor-root> and writes status.json plus deduplicated local alerts.jsonl. Stalling requires at least three consecutive active, runnable samples without progress across 15 minutes. Quota waits, idle writers and audit blocks have separate states. Monitoring needs no provider credentials.

repair handles only source-bound stock-day gaps in deferred_missing.json, requesting at most one stock-day per invocation. Defaults cap daily requests at 500 and attempts per item at three, with exponential backoff starting at 30 minutes. It shares the main writer lock and quota ledger. An empty queue creates no provider client. Use --dry-run to inspect candidates.

Repair preserves original partitions and receipts, then atomically installs a new complete candidate after validation. Semantic failures retain an audit block for separate investigation. Candidates require later acceptance and never switch published aliases. Generic timers live under scripts/systemd/tushare-minute-history-watch.* and tushare-minute-gap-repair.*; rendered units must use stable production releases.

The code-size baseline adds the bounded repair worker, evidence helpers, read-only inventory, shared lock, CLI and failure-path tests. Complexity and large-file limits stay fixed. Future edits must fit this measured baseline or reduce it when these components are consolidated.

Explicit semantic quarantine¶

Use scripts/operations/tushare_minute_history.py quarantine --artifacts-root <data-root> --runtime-root <runtime-root> --acceptance <failed-acceptance.json> --requery <raw-stock-day.parquet> --trade-date YYYYMMDD --symbol <ts-code> --reason '<investigation conclusion>' after independently investigating a reproducible supplier inconsistency. This credential-free command shares the writer lock and revalidates the source binding, complete minute grid and structural semantics. Only hard-notional failures confined to the named stock-day are eligible; the entire requery must reproduce the original stock-day.

The registry at metadata/minute_backfill/reverse_scheduler/semantic_quarantine.json binds source partition, sidecar, failed acceptance and requery SHA256 values. Original evidence and the persistent block remain unchanged. Acquisition proceeds only when quarantine fully covers that block. Unknown failures, changed evidence and malformed registries remain blocking. Custom host preflights must use quarantine_status(root, marker)["blocked"] rather than marker existence. Gap repair uses the same gate and retains additional failures under semantic-audit-additional-blocks/.

Partition acceptance, candidate publication, operational assembly and alias promotion reject quarantined dates, including paths through symlinks. Monitoring exposes semantic_quarantine_dates with a deduplicated warning; exhausted first-pass acquisition returns download_complete_with_gaps. Correct minute values still require separate investigation and repair. This command provides no automatic release from quarantine or publication exemption. Monitoring streams the bound evidence hashes to detect changes.

The size baseline admits the quarantine evidence helpers and failure-path tests. Complexity and large-file budgets stay fixed; consolidate or remove these additions when the workflow is retired.

Statement template diagnostics¶

python scripts/operations/audit_fundamentals_templates.py --vintage <vintage> --output <new-directory> [--dataset income] replays normalization filters from complete raw v2 inputs and quarantines observation keys that remain ambiguous after provider revision selection. Diagnostic tables, conflicting rows, and a hash receipt go to a new external data directory. Original retrieval timestamps remain intact and actual generation time is recorded separately. The tool makes no API requests, publishes no normalized asset, and preserves sealed inputs and shared aliases. The strict builder continues to reject conflicting keys. Diagnostic quarantine grants no production exception.

Repeat --dataset to select income, balancesheet, or cashflow; the default checks all three. For a failed vintage, select only datasets with complete raw v2 inputs. A normalized manifest is optional; its missing prior hash is recorded as null. Outputs contain no standard asset manifest, PIT, or vintage root seal, so the vintage remains incomplete.

The shared ratchet python_lines baseline increases by exactly 64 lines for dataset selection and failed-vintage regression coverage. Complexity and all other limits stay unchanged; reclaim the corresponding line budget when the diagnostic entry point is consolidated or fixtures shrink.

Fundamentals archive order¶

The periodic archive saves all selected raw datasets before strict normalization, PIT, and sealing. See a-share-fundamentals.en.md. Normalization failure retains the other completed raw inputs without a root seal. The shared ratchet python_lines increases by exactly 137 lines for the two-phase orchestration and failure/reentry coverage. Complexity limits stay unchanged. Reclaim the corresponding line budget when orchestration entry points are consolidated or regression fixtures shrink.

archive_state.json atomically records the archive phase, dataset, completed child manifest hashes, and failure type. Same-day fundamentals conflicts also record the business key; arbitrary exception text is excluded to avoid exposing request details. This is diagnostic progress evidence: running can also indicate an interrupted process. Completion still requires validated manifest.yml and SEALED.json. Retries reuse completed children only with a proven matching semantic identity. Reentry into a sealed archive only verifies integrity and does not certify newly supplied parameters or rewrite its state receipt.

New archives write retry_identity.json before provider access. It records dates, datasets, run IDs, report types, the normalized endpoint SHA-256, mappings, delay, observation age and cadence, Git revision, implementation hash, actual Python version, and installed numpy/pandas/pyarrow/PyYAML/TuShare versions. Missing required libraries or versions fail as unproven. The optional network SDK TuShare is explicitly not_installed when absent during offline reuse. SDK availability or any runtime version change blocks retry, and an absent SDK cannot make real provider requests. Any semantic change requires a new output vintage. Mapping order is retained because duplicate source entries use the last target in PIT parsing. Operational settings are recorded separately per attempt. page_size and max_pages control request pagination, while raw completeness still rejects truncated units. Changing these controls cannot certify incomplete raw data as completed, and reused completed inputs retain matching query identity and validated manifest/source hashes. Invalid hostnames and endpoints containing whitespace, control characters, credentials, query strings, or fragments are rejected. Retry evidence retains neither endpoint text nor token values. Derived reuse also requires an explicit matching identity receipt bound to the sealed manifest hash, source lineage, and payload integrity.

Every execution creates attempts/<id>/ with sanitized parameters.json, exact previous receipt bytes when present, attempt state, and append-only stage events. The root state file is only the latest progress index. Nonempty legacy output without a complete recorded identity fails with identity is unproven before writes or provider calls. Current defaults must never fill unknown legacy parameters. Preserve such output for offline investigation and plan an isolated destination separately. Run one writer per vintage. Revision discovery requires a source release checkout containing Git metadata and both src/ and scripts/, and any revision change conservatively blocks retry.

This change increases only the shared inventory budget by one Python file and 499 lines, leaving complexity limits unchanged. Remove that allowance when a shared asset transaction interface retains legacy rejection, semantic comparison, and attempt history coverage.

Delayed margin recovery and receipt states: A-share operations.

Operational Asian index capture¶

Use the installed owner environment and existing private provider configuration:

marketdata config run --config "$DATA_PLATFORM_CONFIG" -- python -m quant_market_data_platform.overseas_index_capture --root "$DATA_PLATFORM_ROOT/reports/index-recovery/capture" --date 20261021 --symbols HSI,N225,KS11

The command emits a JSON receipt path and exits 0 for complete capture or 1 for partial/failed capture. It uses the configured TUSHARE_TOKEN_2 route and one logical index_global request per code, with a 21-day lookback and a 12-second transport timeout. Transport failover can visit configured routes; the deployment caller must enforce a 45-second process-group deadline. The module does not enforce that total deadline itself. Keep all outputs outside the source repository. Interrupted runs retain started.json for inspection. This endpoint provides index closes, not restored Hong Kong equity production.

For a same-day daily-clean rebuild, generate a new plan and use its new clean output directory. Do not build into a completed version or latest. Preserve failed-run evidence and choose a new directory when needed. See daily-clean build versions.

Independent historical minute coverage¶

See the historical coverage pilot for an offline scan of at most ten dates. Its diagnostic output cannot authorize acceptance or publication.

Data operations status¶

The read-only overview summarizes asset dates, gaps and service incidents.

Read-only minute partition validation¶

marketdata tushare validate-minute-partition \
  --data-root /path/to/minute-overlay --trade-date 20260706 --exchange SZ

The installed owner CLI validates the trade_date=YYYYMMDD partition, its completeness sidecar, file hashes, minute grid and exact exchange universe. It neither fetches data nor modifies inputs. Success writes one JSON receipt to stdout; invalid evidence exits nonzero without a success receipt. Consumers use a pinned installed CLI rather than importing provider internals.

Weekly raw fundamentals observations¶

The scheduled archive captures all four raw datasets each Saturday at 02:30 Asia/Shanghai with --archive-mode raw --observation-frequency weekly. It seals raw content only, without normalization, PIT construction or publication. The CLI default remains --archive-mode full, preserving the existing full pipeline. Existing data is retained; this reduces future acquisition and storage growth and reclaims no existing bytes. At unchanged per-capture size, weekly acquisition produces about one seventh as many captures as daily acquisition; actual growth depends on provider payloads.

Raw observations use fundamentals_observations/observation=YYYYMMDD and OBSERVATION_SEALED.json. Reentry and --verify-only use the same mode, query, endpoint and cadence arguments. A changed scope fails before supplier access. Partial captures have progress/attempt evidence but no completion marker. Weekly gaps remain visible; the default three-day maximum observation age is unchanged.

Use explicit offline builds when derivatives are needed. The example selects four fields; extend mappings for the research requirement. Derivatives go to separate directories and do not publish latest or retroactively certify historical revisions.

python scripts/operations/archive_tushare_fundamentals_vintage.py \
  --artifacts-root "$DATA_PLATFORM_ROOT" --snapshot-date 20261024 \
  --archive-mode raw --observation-frequency weekly --api-url https://api.tushare.pro

OBS="$DATA_PLATFORM_ROOT/assets/tushare/a_share/fundamentals_observations/observation=20261024"
DERIVED="$DATA_PLATFORM_ROOT/research/fundamentals-20261024"
for dataset in income balancesheet cashflow fina_indicator; do
  marketdata tushare normalize-a-share-fundamentals --dataset "$dataset" \
    --raw-dir "$OBS/raw/$dataset" --out-dir "$DERIVED/normalized/$dataset"
done
marketdata tushare build-a-share-fundamentals-pit \
  --normalized-dir "$DERIVED/normalized/income" \
  --normalized-dir "$DERIVED/normalized/balancesheet" \
  --normalized-dir "$DERIVED/normalized/cashflow" \
  --normalized-dir "$DERIVED/normalized/fina_indicator" \
  --field-map revenue=revenue --field-map total_assets=total_assets \
  --field-map n_cashflow_act=n_cashflow_act --field-map roe=roe \
  --out-dir "$DERIVED/pit"

Minute daily run evidence and fallback policy¶

The daily operational entry preserves resumable plan/backfill paths and creates a unique invocation journal under metadata/minute_operational/history/ in the external data root. It retains attempt statuses, safe failure categories, receipt hashes and explicit rate-limit numbers before a later resume can replace mutable checkpoints. Raw responses, endpoint URLs and credentials are not retained. A force-killed process can leave an unfinished running record; that is not success.

Run the installed operational script with --artifacts-root "$DATA_PLATFORM_ROOT" --status-only to read the formal asset, calendar and retained invocations. It makes no provider calls, catalog repairs or history writes. States distinguish current, behind, current_with_failed_run and recovered_after_failure. Recovery requires a later successful invocation and coverage of the failed date. Missing historical evidence is not fabricated.

--no-fallback is mutually exclusive with --fallback-token-env. With no configuration the existing TUSHARE_TOKEN fallback remains the default. The non-secret project environment selector TUSHARE_OPERATIONAL_FALLBACK_TOKEN_ENV chooses another existing credential variable; an explicit empty value disables fallback. Explicit CLI options take precedence. This changes neither credentials nor provider entitlements. A local request budget does not establish supplier permission; an account limited to one request per hour cannot provide rapid full-market failover. Shared transient retry, quota accounting and publication checks remain.

Offline US equity minute layouts¶

An optional standalone module mirrors audited by-day Parquet into by-security ZSTD6 files. See offline layouts. It does not register platform markets or switch current assets.

Offline mirror recompression¶

Use python -m quant_market_data_platform.offline_equities.repack --source <mirror> --output <new-version> to create a separate lossless delta/ZSTD12 candidate from a completed security mirror. See the offline guide for verification and resume.

L2 pilot output safety¶

marketdata quality pilot --file <raw.parquet> --output-dir <task-output>

Before creating directories or writing, the pilot checks every input Parquet footer and the complete output plan. Inputs must not collide on canonical or labels filenames. Repeated source identities, existing output files (including symlinks or hardlinks), and symlinked output subdirectories are rejected. An existing output directory is allowed only when every planned leaf and manifest is new. Exclusive file creation also refuses a destination that appears after preflight.

Keep inputs stable while the pilot runs. Later I/O failures may leave partial derived files; this is not atomic publication or automatic resume. Inspect that output and select a fresh directory instead of deleting or replacing raw inputs.

marketdata contract capture: capture and recheck selected publication metadata; see pinned input references.