Web Dashboard Data-Source Research Note¶
This is an early research note about the data required to reproduce an external market overview terminal, current gaps, and possible provider costs. Prices and recommendations below are a dated snapshot, not current purchasing advice. The project dashboard command is
dm dashboard; it writesout/web_dashboard.htmlandout/web_dashboard_payload.json.
Executive summary¶
- The largest gaps identified were next-twelve-month price/earnings (NTM P/E) and forward earnings per share (EPS). The value-oriented starting option in this 2026-07-02 assessment was FMP Premium at USD 79/month, approximately CNY 536/month, or about USD 59/month (CNY 401/month) on an annualized plan. Higher-quality alternatives such as FactSet Estimates and S&P Capital IQ Estimates required a quote.
- The next priority was point-in-time S&P 500 constituents, so breadth could be calculated from the actual constituents on each historical date. The note listed EODHD Fundamentals Data Feed at USD 59.99/month (about CNY 407) or All-in-One at USD 99.99/month (about CNY 679) as lower-cost options, and S&P Global Index Data as a quote-based option.
- A daily job can gradually populate
market_state_panel.csv. For useful backtests at launch, the note recommended backfilling at least three to five years of VIX, breadth, valuation, and forward-return data.
Findings¶
This note inventories the data needed by the market overview terminal at https://www.lucasgou.cloud/, what this project had reproduced at the time, the remaining gaps, and indicative provider costs.
The project had a static dashboard generated by dm dashboard, with outputs out/web_dashboard.html and out/web_dashboard_payload.json. At that time the page read this repository's ETL outputs, snapshots, and an optional state panel. It did not call the target website's API.
The assessment grouped the findings into three points:
- Market sentiment and cross-market prices had a display-ready foundation: VIX, VVIX, Cboe Put/Call, AAII, SPY/QQQ, gold, DXY, and the 10-year Treasury yield were represented.
- Valuation and constituent breadth were the principal gaps. SPX/NDX NTM P/E, forward EPS, valuation percentiles, and point-in-time constituent breadth generally required commercial data or a manually maintained panel.
- For a course reproduction, start with proxy indicators, a fixed-constituent CSV, low-cost price APIs, and a manually maintained valuation panel. A formal research workflow could later evaluate FactSet, Bloomberg, LSEG, S&P Capital IQ, or Cboe DataShop.
The cost estimates were dated 2026-07-02 and used an approximate USD/CNY rate of 6.79. Recheck provider prices, academic and personal/commercial licensing, exchange permissions, content storage, and redistribution terms before purchasing.
Data required by the target page¶
The target homepage is a market overview terminal centered on sentiment, valuation, drawdowns, regime backtests, and an event stream.
| Page module | Required data | Purpose |
|---|---|---|
| Market sentiment | Fear & Greed, VIX, VVIX/VIX, SPX/NDX 20/50/200-day breadth, 30-day trends | Assess risk appetite, fear, and market participation |
| Core valuation | SPX/NDX NTM P/E, forward EPS, 1-, 5-, and 10-year percentiles | Place index valuations in historical context |
| EPS and valuation attribution | Index price, forward EPS, forward P/E, valuation change | Attribute index returns to earnings or multiple expansion |
| Drawdown levels | Current SPY/QQQ price, prior high, drawdown, implied P/E, valuation percentile | Show prices and valuations at drawdown thresholds |
| Regime backtest | FNG, VIX, 50-day breadth, NTM P/E percentile, subsequent 5/30/90/252-day returns | Find similar historical states and summarize subsequent returns |
| Strategy lab | Natural-language strategy, rule parsing, instrument identification, backtest result, data coverage | Let users specify and backtest a strategy |
| Open Hub event stream | Headline, source, URL, entity labels, deduplication ID, saved state, media assets | Aggregate news, filings, AI, and financial events |
Observed fields in the target site's API samples:
| API sample | Key fields | Module |
|---|---|---|
/api/v1/sentiment/overview |
fear_greed, vix, vol_structure, breadth.spx, breadth.ndx |
Market sentiment |
/api/v1/sentiment/fear-greed/trend |
series[].trade_date, series[].value |
Fear & Greed trend |
/api/v1/sentiment/volatility/trend |
vix_series, vol_structure_series |
Volatility trend |
/api/v1/sentiment/breadth/trend |
above_20d_series, above_50d_series, above_200d_series |
Constituent breadth |
/api/v1/valuation/timeline |
current_value, percentile, series, valuation_source |
Valuation timeline |
/api/v1/valuation/price-attribution |
total_return, eps_contribution, valuation_contribution, pe_start, pe_end |
Return attribution |
/api/v1/valuation/drawdown-scenarios |
current_price, high_price, current_drawdown_pct, scenarios[].implied_pe |
Drawdown levels |
/api/v1/market-regime/overview |
conditions, metrics[].window_days, win_rate, avg_return |
Regime backtest |
/api/v1/strategy-lab/runs |
strategy_spec, charts, data_coverage |
Strategy lab |
Data already connected in this project¶
The dashboard records source, proxy methodology, and known gaps in sourceAnalysis. Its primary input files are:
| Input file | Purpose |
|---|---|
out/scores.json |
Topic scores, factor decomposition, and topic-valuation proxies |
out/actions.json |
Threshold-triggered action suggestions |
out/raw_market.json |
Raw market ETL results, including indexes, topics, BTC, and sentiment |
out/raw_events.json |
Event headlines, sources, URLs, and source chains |
out/etl_status.json |
Fetcher health status |
data-snapshots/latest/cross_market_snapshot.json |
U.S. equities, macro, commodities, and cross-market mappings |
data-snapshots/latest/tushare_snapshot.json |
A-share indexes, advancing/declining counts, flows, and limit-up/down data |
out/market_state_panel.csv |
Optional state panel for historical backtests, breadth, valuation, and drawdown fields |
| Source | Status in this note | Use |
|---|---|---|
| Alpha Vantage | Available | Primary index and topic-price path |
| FMP | Available | Topic valuation, price fallback, and financial proxies |
| SEC EDGAR | Available | Filings and basic XBRL data |
| Cboe VIX/VVIX historical CSV | Available | VIX, VVIX, and VVIX / VIX / 3.5 |
| Cboe Daily Market Statistics | Available | Put/Call sentiment |
FRED VIXCLS and 10Y |
Available | VIX fallback and 10-year Treasury yield |
| AAII Sentiment | Available | Retail bullish/bearish sentiment |
| yfinance-style snapshots | Available | SPY, QQQ, SMH, U.S. indexes, gold, DXY, and related series |
| Coinbase and OKX | Available | BTC spot, funding rates, and perpetual-futures basis |
| SoSoValue | Available | BTC ETF net flows |
| TuShare | Available | A-share extension module |
| RSS, SEC, and public webpages | Partially available | Event headlines, source chains, and public links |
At the time, the dashboard modules had this implementation status:
| Module | Existing implementation | Reproduction status |
|---|---|---|
| Market sentiment | VIX, VVIX, Put/Call, AAII, fear_greed_proxy, SPY/QQQ participation proxies |
Partial |
| Core valuation | Topic valuation factors, valuation_rate_gap_proxy, optional valuation-panel fields |
Proxy |
| Drawdown levels | Current SPY/QQQ prices; optional prior highs and valuation fields | Partial on prices |
| Regime backtest | Similar-state framework using historical state-panel columns | Framework available |
| Strategy lab | Page entry point and gap description | Not implemented |
| Event stream | Headline, source, URL, sourceChain |
Partial |
Remaining gaps¶
| Gap | Impact | Suggested direction in the original note |
|---|---|---|
| Raw CNN Fear & Greed values | Only fear_greed_proxy can be shown |
Keep the local proxy and add SPX momentum, actual breadth, credit spreads, and safe-haven demand |
| SPX/NDX constituent breadth | Cannot reproduce 20/50/200-day constituent-breadth trends | Start with a fixed-constituent CSV and low-cost daily prices; later obtain point-in-time constituents |
| SPX/NDX NTM P/E and forward EPS | Core valuation, return attribution, and implied P/E remain proxies | Start with a manually maintained valuation_panel.csv; obtain consensus estimates for a formal version |
| Valuation percentiles and historical returns | Too few state-backtest observations to reproduce nearest-state statistics | Build a history panel and add 5/30/90/252-day forward returns |
| Macro-calendar license | Trading Economics returned HTTP 401; the calendar needed a fallback | Consider FMP or EODHD for lower cost; evaluate Trading Economics for wider coverage |
| News body and media assets | Open Hub can show only headlines and source chains | Add deduplication IDs and entity labels first; connect bodies/media only after licensing review |
| HSI prices | Stooq returned insufficient data, leaving the Hong Kong index unavailable | Consider FMP, Twelve Data, a Hong Kong ETF, or exchange-authorized data |
| Strategy lab | No natural-language rule parsing or backtest result | Build rule templates first, then consider LLM parsing and a backtest sandbox |
Provider options and indicative costs¶
The following prices were copied from public pricing pages for the 2026-07-02 assessment and converted approximately to CNY. Institutional datasets commonly require a quote.
| Data gap | Free option | Value option | Higher-quality option | Suggested first step |
|---|---|---|---|---|
| VIX, VVIX, Put/Call | Already connected through Cboe, FRED, and Cboe Daily; CNY 0, with scheduling and storage costs | Alpha Vantage Premium from USD 49.99/month (~CNY 339); EODHD options/index add-on from USD 29.99/month (~CNY 204) | Cboe DataShop, LiveVol, Bloomberg, LSEG; typically quote-based | Persist the existing daily snapshots as historical series |
| Fear & Greed | Local fear_greed_proxy; CNY 0, but its composite methodology and history need maintenance |
Reuse FMP, Tiingo, Twelve Data, or EODHD, around USD 22–59/month (~CNY 149–401) | Build within a Bloomberg, LSEG, or FactSet workflow, or license a sentiment dataset | Keep the local proxy; do not use the target site's API |
| SPX/NDX constituents and breadth | Public constituent lists with yfinance or Alpha Vantage free tier; historical membership is incomplete | FMP Starter USD 22/month (~CNY 149); Tiingo Power USD 30/month (~CNY 204); Twelve Data Grow from USD 29/month, commonly USD 79 monthly (~CNY 197–536); EODHD Fundamentals USD 59.99/month (~CNY 407), with a historical S&P 500 constituents endpoint | S&P Global, Nasdaq official data, Bloomberg, or LSEG; typically quote-based | Start with fixed constituents, calculate 20/50/200-day breadth, and label it proxy |
| Index/ETF daily prices and highs | yfinance, Alpha Vantage compact daily prices, Nasdaq Data Link free data; CNY 0 | FMP, Tiingo, Twelve Data, Massive, or EODHD, from about USD 22–30/month (~CNY 149–204) | Bloomberg, LSEG, S&P Global, or Nasdaq Data Link premium; typically quote-based | Add price-based drawdown levels first |
| SPX/NDX NTM P/E and forward EPS | SEC XBRL plus ETF holdings for trailing/proxy measures, or a manually maintained CSV; CNY 0 | FMP Premium USD 79/month (~CNY 536), or annualized about USD 59/month (~CNY 401); Twelve Data Grow USD 29–79/month (~CNY 197–536); EODHD Fundamentals USD 59.99/month (~CNY 407) or All-in-One USD 99.99/month (~CNY 679) | FactSet Estimates, Bloomberg Data License, LSEG Datastream, I/B/E/S, or S&P Capital IQ Estimates; typically quote-based | Start with valuation_panel.csv; purchase consensus estimates for a formal version |
| EPS and valuation attribution | Demonstration attribution using trailing EPS or earnings yield; CNY 0 | FMP Premium USD 79/month (~CNY 536); or aggregate index weights with EODHD Fundamentals, Tiingo Power, or Twelve Data Grow, from about USD 30–79/month (~CNY 204–536) | Consensus and index-aggregation data from FactSet, Bloomberg, LSEG, or S&P; typically quote-based | Wait for stable NTM P/E and forward EPS inputs |
| Macro calendar and events | FRED, SEC EDGAR, company IR RSS, exchange notices, public news RSS; CNY 0 | Trading Economics Standard USD 149/month (~CNY 1,012); FMP Starter/Premium USD 22–59/month (~CNY 149–401); EODHD Calendar & News USD 19.99/month (~CNY 136) | Bloomberg News, LSEG News, Reuters, Dow Jones, Factiva, or S&P Global; typically quote-based | Store only headline, source, and URL when content is not licensed |
| A-share prices and flows | Public exchange data and AkShare; CNY 0 | TuShare Pro, JoinQuant, or Ricequant; verify current plan and account permissions | Wind, Choice, or iFinD; generally quoted by year, seat, and module | Retain this as an extension module |
| Historical state panel | Local CSV appended daily from yfinance, FRED, Cboe, and local breadth scripts; CNY 0 | Local DuckDB or Parquet factor store plus FMP, EODHD, Tiingo, Twelve Data, or Massive from about USD 22–79/month (~CNY 149–536) | Point-in-time data lake from Bloomberg, LSEG, S&P, or FactSet; typically quote-based | Stabilize the market_state_panel.csv schema first |
Suggested sequence from the original note¶
- Persist VIX, VVIX, Put/Call, AAII, SPY/QQQ, 10Y yield, and topic valuations to the state panel.
- Add SPX/NDX 20/50/200-day breadth using fixed-constituent CSVs and low-cost daily prices. Label the results
proxy. - Add
valuation_panel.csvfor manual or licensed SPX/NDX NTM P/E, forward EPS, and valuation percentiles. - Add SPY/QQQ historical prices and 5/30/90/252-day forward returns for nearest-state statistics.
- Replace the HSI source to reduce dependence on Stooq for the Hong Kong module.
- Use FMP, EODHD, or public RSS for macro events until Trading Economics credentials and plan access are confirmed.
- Add
event_id, entity labels, and deduplication rules before storing licensed news bodies or media. - Defer the strategy lab to phase two. Start with fixed rule templates, then consider natural-language parsing.
Dashboard generation¶
Generate the static dashboard with:
uv run dm dashboard
| Output file | Purpose |
|---|---|
out/web_dashboard.html |
Static page for local viewing or static-site publication |
out/web_dashboard_payload.json |
Structured data for tests, diagnosis, and downstream use |
Optional state-panel commands:
uv run dm state-panel --period 10y --timeout 30
uv run dm dashboard --state-panel path/to/market_state_panel.csv
dm state-panel attempts to build out/market_state_panel.csv from FRED, Cboe VIX/VVIX, Cboe Put/Call, and yfinance. To calculate SPX/NDX constituent breadth, pass --include-breadth --breadth-mode sample. This is a slow operation; write to a separate CSV first and review its quality.
Without --state-panel, dm dashboard checks MARKET_STATE_PANEL and then out/market_state_panel.csv. If neither provides an explicit panel, the page derives one proxy state from Cboe/FRED VIX, Cboe Put/Call, AAII, SPY/QQQ, topic valuation, and the 10-year Treasury yield. It is labeled derived:free-source-proxy, represents one day only, and does not replace a historical state panel.
Suggested state-panel fields:
| Field | Purpose |
|---|---|
date |
Observation date |
own_risk_appetite_score |
Locally constructed risk-appetite score |
rsp_spy_participation_proxy |
Market-participation proxy |
valuation_rate_gap_proxy |
Valuation/rate-gap proxy |
spx_above_50d_pct, ndx_above_50d_pct |
SPX/NDX 50-day breadth |
spy_high, qqq_high, spy_52w_high, qqq_52w_high |
SPY/QQQ highs |
spx_ntm_pe, ndx_ntm_pe, spy_ntm_pe, qqq_ntm_pe |
Index or ETF valuation proxies |
spx_ntm_pe_percentile, ndx_ntm_pe_percentile |
Valuation percentiles |
vix, spy_close, sp500_close, ten_year_yield, hy_oas |
Latest state indicators |
spy_forward_return_fwd_90d, spy_forward_return_fwd_252d, spy_forward_return_fwd_1260d, spy_forward_return_fwd_2520d |
Subsequent returns |
Coverage labels¶
The page uses four status labels:
| Status | Meaning |
|---|---|
available |
Native project data is available for direct display |
proxy |
A local or constructed proxy is used; the page must explain its methodology |
simulated |
Example, simulated, or local fallback data is used |
missing |
An input file, required field, valid credential, or permission is missing |
When news content is not licensed, store only its headline, source, and URL. Mark valuation, breadth, Fear & Greed, and other proxy-derived fields as proxy.
References¶
| Data type | Reference |
|---|---|
| VIX and macro series | FRED API, FRED VIXCLS |
| VIX, VVIX, Put/Call | Cboe VIX Historical Data, Cboe Daily Market Statistics |
| Investor sentiment | AAII Investor Sentiment Survey |
| Filings and XBRL | SEC EDGAR APIs |
| Equity daily prices and basic market data | Alpha Vantage Documentation, Nasdaq Data Link API, Nasdaq Data Link Docs |
| Lower-cost market data | Financial Modeling Prep, EODHD, Tiingo, Twelve Data, Massive |
| Macro calendar and news | Trading Economics Calendar API, FMP Calendar & News |
| Consensus estimates and valuation | FactSet Consensus Estimates, Bloomberg Data License, LSEG Datastream, S&P Capital IQ Estimates |
| A-share extension | TuShare |