Shared Data Contracts¶

SH/SZ historical minute data uses the separate contract metadata/current_assets/a_share_sh_sz_minute_current.json. Select it explicitly with PublishedAssetContract.load(path).asset("minute_1m_tushare_sh_sz_historical"). The published scope is SH_SZ, with full_market_publication_allowed: false. It does not replace minute_1m, minute_1m_tushare, or a_share_current.json. Each immutable version retains a _publication_receipt.json with coverage, per-date hashes and exclusions; coverage_exclusions lists excluded dates. Coverage bounds do not imply a contiguous history: consumers must check the actual partitions and gaps. The default full-market load_current is unchanged.

中文页面

status: active

owner: market-data-platform

audience: human and agent

last_verified: 2026-09-06

source_of_truth: yes

superseded_by: n/a

Dataset semantic contracts¶

market_data_platform.dataset_contract.v1 uses a list of unique, nonempty strings for primary_key; omission and an empty list are allowed. Column types, names and order are not coerced. Public validation and writing APIs reject non-string members before replacing an existing contract. See L2 ingestion gates for the full semantics.

Artifact root¶

The shared artifact root defines the storage boundary between data tooling and strategy repositories. Configure it with:

export DATA_PLATFORM_ROOT=/data/market-data-platform

DATA_PLATFORM_ROOT identifies shared market-data inputs and platform artifacts. Downstream run records, caches, and reports belong to their respective repositories and are not configured through this variable.

Research code can match an input to the current contract with market_data_platform.contract.match_current_contract_entry, load the contract with load_current_contract, classify path existence with path_kind, and persist resolved path/manifest/current-contract associations with describe_input_path. These APIs read contract state only; they do not refresh data or publish assets.

Historical Hong Kong runs can still locate metadata/current_assets/hk_current.json through current_contract_path(..., market="hk"). This is path compatibility only; the platform has not resumed Hong Kong data production.

Current data contract¶

The contract is stored at:

<artifacts_root>/metadata/current_assets/<market>_current.json

Its top-level JSON includes contract metadata and an assets mapping. The contract metadata records name, market, provider, version, artifacts_root, and target_date. Each asset entry can carry alias_path, resolved_path, manifest_path, and as_of.

Build the contract from standard asset aliases:

marketdata contract build \
  --market a_share \
  --provider tushare \
  --artifacts-root "$DATA_PLATFORM_ROOT" \
  --target-date 20260109

By default, the command merges the existing current contract and generates metadata/dataset_registry.csv. Add --no-registry to write only the JSON contract.

Inspect whether the contract exists, whether aliases are missing, and whether each asset's as_of is behind the target date:

marketdata contract inspect \
  --market a_share \
  --provider tushare \
  --artifacts-root "$DATA_PLATFORM_ROOT" \
  --target-date 20260109 \
  --fail-on-severity error

The inspect output includes manifest coverage bounds and row counts. A-share research candidates should pass --require-start-date YYYYMMDD; a later asset start produces a warning and can block promotion with --fail-on-severity warning.

Minute asset entry points¶

The two stable A-share minute aliases are:

<artifacts_root>/assets/derived/a_share/minute_1m
<artifacts_root>/assets/derived/a_share/minute_1m_tushare

minute_1m is the Guan legacy canonical and currently points to minute_1m_v3_20260714. minute_1m_tushare is the TuShare-native operational canonical. Both have the same eight-column Parquet schema, but there is no promise of cross-provider feature equivalence. After acceptance, minute assets enter a_share_current.json and dataset_registry.csv; the TuShare asset uses key minute_1m_tushare, while Guan remains the minute_1m rollback/comparison asset. Downstream consumers record the selected alias, resolved version, and receipt hash. Guan daily source and market scope come from its coverage receipt; TuShare integrity and lineage come from its operational receipt. See A-share minute data.

Hong Kong market support was retired with RQData on 2026-07-26. freeze-hk and hydrate-hk have been removed. Historical reproduction uses tag hk-freeze-20260613 or the private archive repository.

Asset keys¶

For mainland China, --provider tushare explicitly selects the TuShare raw-asset layout. The contract.provider in a_share_current.json is then tushare. Common keys include:

Key	Canonical relative path
`instruments`	`assets/tushare/a_share/instruments/a_share_all_instruments_latest.parquet`
`trade_cal`	`assets/tushare/a_share/trade_cal/a_share_trade_cal_latest.parquet`
`daily`	`assets/tushare/a_share/daily/a_share_all_daily_latest`
`adj_factor`	`assets/tushare/a_share/adj_factor/a_share_all_adj_factor_latest`
`daily_basic`	`assets/tushare/a_share/daily_basic/a_share_all_daily_basic_latest`
`limit_status`	`assets/tushare/a_share/limit_status/a_share_limit_status_latest`
`moneyflow`	`assets/tushare/a_share/moneyflow/a_share_all_moneyflow_latest`
`moneyflow_dc`	`assets/tushare/a_share/moneyflow_dc/a_share_all_moneyflow_dc_latest`
`moneyflow_hsgt`	`assets/tushare/a_share/moneyflow_hsgt/a_share_all_moneyflow_hsgt_latest`
`top_inst`	`assets/tushare/a_share/top_inst/a_share_all_top_inst_latest`
`ths_hot`	`assets/tushare/a_share/ths_hot/a_share_all_ths_hot_latest`
`dc_concept`	`assets/tushare/a_share/dc_concept/a_share_all_dc_concept_latest`
`dc_concept_cons`	`assets/tushare/a_share/dc_concept_cons/a_share_all_dc_concept_cons_latest`
`kpl_list`	`assets/tushare/a_share/kpl_list/a_share_all_kpl_list_latest`
`kpl_concept_cons`	`assets/tushare/a_share/kpl_concept_cons/a_share_all_kpl_concept_cons_latest`
`limit_step`	`assets/tushare/a_share/limit_step/a_share_all_limit_step_latest`
`limit_cpt_list`	`assets/tushare/a_share/limit_cpt_list/a_share_all_limit_cpt_list_latest`
`report_rc`	`assets/tushare/a_share/report_rc/a_share_all_report_rc_latest`
`stk_surv`	`assets/tushare/a_share/stk_surv/a_share_all_stk_surv_latest`
`broker_recommend`	`assets/tushare/a_share/broker_recommend/a_share_all_broker_recommend_latest`
`fund_portfolio`	`assets/tushare/a_share/fund_portfolio/a_share_all_fund_portfolio_latest`
`top10_holders`	`assets/tushare/a_share/top10_holders/a_share_all_top10_holders_latest`
`top10_floatholders`	`assets/tushare/a_share/top10_floatholders/a_share_all_top10_floatholders_latest`
`stk_holdertrade`	`assets/tushare/a_share/stk_holdertrade/a_share_all_stk_holdertrade_latest`
`moneyflow_ths`	`assets/tushare/a_share/moneyflow_ths/a_share_all_moneyflow_ths_latest`
`limit_list_ths`	`assets/tushare/a_share/limit_list_ths/a_share_all_limit_list_ths_latest`
`margin_detail`	`assets/tushare/a_share/margin_detail/a_share_all_margin_detail_latest`
`margin`	`assets/tushare/a_share/margin/a_share_all_margin_latest`
`namechange`	`assets/tushare/a_share/namechange/a_share_all_namechange_latest.parquet`
`margin_secs`	`assets/tushare/a_share/margin_secs/a_share_all_margin_secs_latest.parquet`
`st_history_reconstructed`	`assets/tushare/a_share/st_history_reconstructed/a_share_all_st_history_reconstructed_latest.parquet`
`st_intervals_reconstructed`	`assets/tushare/a_share/st_intervals_reconstructed/a_share_all_st_intervals_reconstructed_latest.parquet`
`hsgt_top10`	`assets/tushare/a_share/hsgt_top10/a_share_all_hsgt_top10_latest`
`ths_index`	`assets/tushare/a_share/ths_index/a_share_all_ths_index_latest`
`ths_member`	`assets/tushare/a_share/ths_member/a_share_all_ths_member_latest`
`daily_clean`	`assets/tushare/a_share/daily/a_share_all_daily_clean_latest`
`flow_ownership_features`	`assets/tushare/a_share/flow_ownership_features/a_share_all_flow_ownership_features_latest`
`hotspot_features`	`assets/tushare/a_share/hotspot_features/a_share_all_hotspot_features_latest`
`fund_portfolio_features`	`assets/tushare/a_share/fund_portfolio_features/a_share_all_fund_portfolio_features_latest`
`holder_structure_features`	`assets/tushare/a_share/holder_structure_features/a_share_all_holder_structure_features_latest`
`top_inst_events`	`assets/tushare/a_share/top_inst_events/a_share_all_top_inst_events_latest`
`holdertrade_events`	`assets/tushare/a_share/holdertrade_events/a_share_all_holdertrade_events_latest`
`hsgt_market_features`	`assets/tushare/a_share/hsgt_market_features/a_share_all_hsgt_market_features_latest`
`normalized_fundamentals`	`assets/tushare/a_share/normalized_fundamentals/a_share_all_normalized_fundamentals_latest`
`pit_fundamentals`	`assets/tushare/a_share/pit_fundamentals/a_share_all_pit_fundamentals_latest`
`industry_changes`	`assets/tushare/a_share/industry_changes/a_share_all_industry_changes_latest`
`universe_by_date`	`assets/universe/a_share_all_full_by_date.csv`
`universe_symbols`	`assets/universe/a_share_all_full_symbols.txt`
`universe_meta`	`assets/universe/a_share_all_full_by_date.meta.yml`

The complete key inventory, including reconstructed ST history, THS index assets, and all provider fields, remains available in the Chinese companion. Do not infer that a key is present in a published contract merely because it is supported by the layout.

daily_clean.is_st has a value only when the build receives a validated, full-range st_history_reconstructed asset. Otherwise it is unknown; do not backfill historical ST state from current instrument names. The build manifest records st_history_file. Research-quality validation checks its receipt, hash, date coverage, and row-level is_st consistency. revision_safe=false means the historical source does not prove announcement-time availability.

Reconstructed ST history records effective dates separately from information availability. Its available_from field is the first exchange session on or after the effective start when ann_date predates that start; when ann_date is on or after the effective start, availability begins on the next exchange session after ann_date. A missing or malformed ann_date, or a missing next session in the supplied calendar, leaves availability unknown. Receipt schema market-data-platform.reconstructed-st-history.v2 identifies this contract. This conservative date rule does not establish an exact intraday publication time, so revision_safe remains false.

daily_clean schema tushare.a_share.daily_clean.v2 carries the matching st_available_from value beside is_st. It is null when the symbol has no active ST interval or when the source cannot establish availability. A consumer may use a positive ST value for a decision only when st_available_from is present and no later than that decision date; unknown availability must fail closed.

The public load_a_share_research_daily and load_daily_watch20_daily views expose is_st and st_available_from from each dated daily row. Both require a completed tushare.a_share.daily_clean.v2 manifest with the daily_clean.st_available_from.v1 contract and reject older assets. The generic A-share view also exposes is_suspended and list_date. Consumers must use the dated ST fields for historical decisions and must not substitute current instrument metadata.

audit-a-share-st-event-timing reads receipt-backed ST events and reconstructed ST history, auditing rows where ann_date=trade_date. It emits st_event_timing_audit.parquet and a hash receipt. Status values include prior_dated_st_event, same_day_time_unknown, later_event_date_conflict, and no_active_prior_event. Conflict candidates are limited to the same symbol and ST events effective within the following 10 calendar days and require manual review. The audit compares dates only, does not modify published is_st, and does not treat the st event endpoint as a complete daily state history.

Raw reference downloads from download-a-share-reference create a same-directory *.receipt.json with query range, row count, quality status, and file SHA-256. This proves download-file integrity, not intraday announcement availability. Publication must revalidate the source hash.

For DailyWatch20, ths_hot_strict_v2 and ths_hot_strict_v3 preserve TuShare's raw ranks without rank imputation. Compatibility strategy v2 requires all top 20 ranks. Production strategy v3 requires rank 1 and permits at most two missing other ranks. A degraded receipt records rank_coverage_status=degraded, missing_ranks, and max_missing_ranks=2; a missing rank 1 or more than two missing ranks blocks publication.

Revision-safe fundamentals start from raw v2. Every query unit records request start/completion, content SHA-256, byte size, and exact retrieval time. Completed raw, normalized v2, and PIT v2 snapshots include integrity.files, integrity.aggregate_sha256, and manifest.seal.json, with immutable_snapshot=true. Strict reads and publication recompute file hashes and the manifest seal. Completed directories are immutable; new observations use a new dated snapshot. Periods before revision_safety.revision_safe_from are reconstructed_pit. Vintage archives live under assets/tushare/a_share/fundamentals_vintages/vintage=YYYYMMDD; root SEALED.json binds all child manifests. These archives do not participate in current-alias selection.

Statement normalization accepts company templates comp_type=1,2,3,4,7 (general, bank, insurance, securities, diversified financial). These describe reporting templates, not consolidation scope; only report_type selects the standard consolidated report. No template takes priority over another for the same observation. After provider update/announcement selection, conflicting same-key records fail the ambiguity check. Rebuild corrections into a new directory, preserving raw retrieval timestamps and old seals; a normalization repair does not certify historical PIT availability. See TuShare income, balance sheet, and cash flow.

dc_concept_cons retains schema tushare.dc_concept_cons.v1 and adds daily completeness evidence. Consumers must check completeness.trade_dates[<trade_date>].complete, not only top-level status: completed or the presence of Parquet. Daily receipts include row/page/request counts, distinct theme count, terminal-page status, and coverage.row_coverage_ratio. Top-level complete fields summarize the batch. Empty results are marked incomplete; the previous Parquet remains as last-known-good but cannot pass the target-date production gate.

Without --provider, mainland contracts default to the TuShare layout. One a_share_current.json identifies the currently selected provider; it does not aggregate raw snapshots from multiple providers. For a partial migration, downstream consumers should still use canonical a_share_current.json, while health reports explicitly identify assets not yet produced, such as adj_factor, limit_status, or daily_clean.

Dataset registry and manifests¶

The human-oriented registry is derived from current contracts and asset manifests:

<artifacts_root>/metadata/dataset_registry.csv

Rebuild it independently with:

marketdata registry build --artifacts-root "$DATA_PLATFORM_ROOT"

Every published directory asset must contain manifest.yml; a single-file asset uses a neighboring *.manifest.yml. A manifest records enough dataset context for consumers to read status, row and symbol counts, date coverage, and lineage. The current contract remains authoritative for downstream path selection.

Read-only Python contract API¶

Downstream Python projects can load the current contract through PublishedAssetContract and retrieve the complete on-disk manifest by asset key:

from market_data_platform import PublishedAssetContract

contract = PublishedAssetContract.load_current(
    "/data/market-data-platform",
    market="a_share",
)
pit = contract.asset("pit_fundamentals")
print(pit.resolved_path)
print(pit.manifest["semantics"])
print(pit.provenance_dict())

The current contract selects a published version; the reader reloads the full manifest at manifest_path rather than treating a contract summary as the complete schema. alias_path, resolved_path, manifest_path, and explicit relative data paths must stay within artifacts_root. External absolute paths, .. traversal, and escaping symlinks are rejected. manifest_sha256 hashes exact file bytes; content_fingerprint hashes canonicalized manifest content and is stable across YAML formatting changes. provenance_dict() provides serializable contract/manifest hashes, schema version, lineage, and asset paths for research records.

Constructing an asset reference does not traverse large Parquet directories. If publication needs per-file verification, it must put the relevant checksums in the manifest; the reader preserves them in the loaded manifest and lineage.

Daily-clean input snapshot receipt¶

a_share.daily_clean_input_snapshot.v1 records a completed, independently copied input attempt. It contains status, created_at, start_date, end_date and datasets. Each of daily, adj_factor, daily_basic and limit_status records its root-relative source_path, source_manifest_sha256, observed rows, and a files list of dataset-relative path, bytes and sha256. Per-dataset manifests refer to ../snapshot_receipt.json and describe captured file/row counts. Consumers must require a completed receipt before using a new snapshot. The receipt is published only after copies and source stability checks pass. Raw writers must be serialized by the caller; unrelated mutable latest directories are not an immutable lineage contract.

For TuShare trade-date mirrors, totals.rows, totals.symbols, and totals.files describe all stored Parquet partitions under the asset data directory, including retained partitions outside the current request. run_totals records rows, symbols, files, and date counters for this invocation; skipped partitions contribute no newly written rows. Query dates describe the request and are not inferred from this inventory. Physical inventory does not certify source completeness: per-date receipts remain authoritative, including incomplete receipts beside retained last-known-good files.

Institutional trading amount units¶

TuShare top_inst reports buy, sell, and net_buy in yuan. top_inst_events converts these amounts to ten-thousand yuan before aggregation and rolling calculations, matching daily.amount / 10 from its thousand-yuan source unit. Field names remain stable; assets built before this correction must be rebuilt before using their amount features or turnover ratios. Counts and provider rates are unchanged.

Institutional survey completeness receipts¶

The tushare.stk_surv.v1 manifest retains its asset key and event-date partition layout. Its additive pagination section records page_size, max_pages, complete and per-date page row counts, observed rows and payload hashes. Per-file pagination.json uses tushare.stk_surv.pagination.v1, recording the exact query, terminal page counts and SHA256. A short terminal page is required, including an empty follow-up after an exact 400-row page. Resume validates receipt query, payload hash, row count and survey dates; asset totals independently count all physical retained partitions, while run_totals counts current writes. Unverified retained partitions keep the asset partial; unreadable inventory records unknown row/symbol totals and fails closed. On failure, completed dates remain preserved and the manifest is partial with the failed date and exception type; provider error text is not persisted. No failed date receives a completeness receipt. Receipts do not certify historical point-in-time availability or provider immutability.

Industry interval manifests bound observed membership dates: query.start_date is the minimum effective date, and query.end_date is the maximum of all effective dates and nonempty interval end dates. A later open membership start cannot be excluded by an earlier closed interval end. These bounds are derived from payload values, not requested refresh dates or the retrieval vintage.

Statement observation ledgers accept per-security income, balancesheet and cashflow sources alongside their _vip batch counterparts, mapped to the same canonical datasets. Both require content checksums, timezone-aware actual observation timestamps and statement identities; visibility remains bounded by the later observation/disclosure date plus the delay. A hash computed during recovery does not certify content at the original download date and cannot justify backdating visibility.

Overseas research forward receipts¶

overseas-receipt-v1 is an independent append-only owner archive. Fresh request CSV files, source dates, assumed closes and actual receipt times are linked by SHA256. See collection and timing rules. QMR owns derived forecasts. Existing published assets and the Hong Kong production contract are unchanged.

Statement normalized v2 canonicalizes whitespace in comp_type and includes it in semantics.dedupe_keys before provider revision selection. semantics.statement_template_identity identifies this dimension; updates and announcements are ordered only within a template, and conflicting rows within one template still fail. All distinct templates survive. No numeric, non-null-count, or retrieval-time precedence is implied. PIT v2 adds optional provenance _source_comp_type, with <field>__source_comp_type in as-of field provenance. Flat PIT building, streaming validation, and queries reject multiple templates for the same symbol, report period, availability date, and source, including equal values, nulls, and distinct retrieval times. A separate explicit selection contract is required to produce one value. Legacy assets without this column remain readable with unknown template identity; no company type is inferred and existing assets are not rewritten. Schema versions remain v2; manifest semantics and actual columns identify template support.

_source_comp_type stores a canonical JSON object mapping each source dataset to its template, for example {"balancesheet":"7","income":"2"}. Coalescing preserves these associations; separate dataset and template sets are insufficient. A legacy scalar template is accepted only for a single source, and empty provenance means unknown identity. Coalesced scalar provenance with known templates and multiple sources is rejected because its associations cannot be recovered.

Operational Asian index receipts¶

overseas-index-operational-v1 is an operational recovery protocol, separate from research capture windows and published market assets. Each exclusive run contains started.json, one raw CSV per requested code, and a final receipt.json. Only the final receipt is consumable. Supported codes are HSI, N225 and KS11. Source entries bind CSV basename, SHA256, row count, source dates, provider (tushare-compatible), route hostname and aware UTC request times. The receipt includes requested date, start/completion times and code hash. Overall status is complete, partial or failed; source status is ok, empty or failed. Failures expose exception class and a sanitized category (permission, transport, provider), never the raw exception message. Unrecognized provider errors remain provider; classification is best effort.

Consumers validate exact exchange sessions, finite positive closes, byte hashes, size limits and observation cutoff. A later receipt cannot support an earlier report. No research qualification or current asset alias is changed.

Daily-clean build versions¶

Each plan-a-share-current-refresh plan assigns a unique clean output ID and carries that path through building, validation, universe construction and promotion. A second same-day plan uses a new directory. Save and execute commands from the same plan. Standalone promotion without --daily-clean-dir retains the legacy dated-directory fallback.

The clean builder rejects symlink outputs, completed manifests and manifests whose incomplete state cannot be established, before deleting existing data or staging. A failed output without a manifest, or one explicitly marked failed, running, building or pending, remains rebuildable. This protects the build entry point, not filesystem immutability or concurrent writers. Latest remains a materialized copy and raw publication is unchanged.

Exchange minute validation CLI receipt¶

marketdata tushare validate-minute-partition returns validation_schema_version=marketdata.minute_exchange_overlay.v1, exchange=SH|SZ and the existing schema_version=tushare.a_share.minute_partition.promotion.v1. Fields are trade_date, partition_path, sidecar_path, partition_sha256, sidecar_sha256, rows, symbols, market_symbol_counts, expected_bars_per_symbol, universe_hash, universe_rule, universe_source and mirror_generated_at. The owner validates the full partition and its exact exchange rule. Consumers reject unsupported schemas, mismatched dates or exchanges, missing fields and failed commands; freshness claims do not replace this validation.

Raw fundamentals observation contract¶

Raw-only archives have schema tushare.a_share.fundamentals.raw_observation.v1, archive_mode: raw, a sealed request identity and exactly four content-verified raw child receipts. The namespace is assets/tushare/a_share/fundamentals_observations/observation=YYYYMMDD; OBSERVATION_SEALED.json binds the manifest hash. It is distinct from a fundamentals_vintages full vintage and its SEALED.json and cannot certify PIT readiness or revision safety. No normalized or PIT child is required or claimed. Completion verifies child manifest seals and content hashes; reentry verifies identity before accepting receipts. Partial output is never complete. Weekly observations describe actual capture cadence, without widening freshness limits or filling gaps. The original full archive schema and retry identity remain unchanged.

Minute daily invocation evidence¶

metadata/minute_operational/history/<timestamp>-<uuid>.json uses a_share.minute_operational_run.v1: unique run_id, start/finish times, target date, run status, fallback policy, safe attempt snapshots and result summary. Subsequent invocations do not replace completed records. Resumable checkpoints and the daily function return value remain compatible. Snapshots bind the receipt bytes read at that time; only failure types, categories, message hashes and explicit rate-limit numbers are retained, not raw responses.

The script's read-only --status-only returns a_share.minute_operational_status.v1 with status, target_date, date_max, last_run and last_failed_run. behind exits 75. Current data does not certify a natural scheduler run or message delivery. Malformed history or paths escaping the owner root fail explicitly instead of hiding evidence.

Research input metadata capture¶

marketdata.asset-snapshot.v1 is a read-only receipt of captured publication references, not a new market-data asset. It reuses PublishedAssetRef identities and rejects observed metadata drift. integrity_scope=manifest_identity, data_content_verified=false, and atomic_transaction=false are substantive limits. The canonical manifest fingerprint is not a Parquet content hash. See the contract capture command for the API and reproduction.