Methodology

Forecast questions are generated from verified space weather events using deterministic templates. Each template maps a specific event type (geomagnetic storm, solar flare, CME, asteroid approach) to a set of machine-resolvable questions.

Questions are designed to be: (1) scientifically resolvable from public data, (2) temporally bounded with explicit close dates, (3) unambiguous in their resolution criteria, and (4) resistant to subjective interpretation.

Every question template is versioned. When a template changes, the version is stored alongside every market it produced, so historical results remain reproducible.

Flarient uses a Logarithmic Market Scoring Rule (LMSR) to aggregate individual forecasts into a single market probability. The LMSR mechanism was designed by Robin Hanson (2002) as a method for combining probabilistic judgments from many forecasters.

The market probability is derived from the YES and NO share quantities: P(YES) = e^(qYes/b) / (e^(qYes/b) + e^(qNo/b)), where b is the liquidity parameter that controls how much each forecast moves the probability.

The liquidity parameter b is set per market based on its class: ordinary markets use a smaller b (less liquidity, more responsive), while major and rare events use a larger b (more liquidity, more stable). This prevents a single forecast from dominating the consensus on high-stakes questions.

Each forecast submission records the prior probability, the resulting probability, and the LMSR state (qYes, qNo) before and after — so the probability movement is fully auditable.

Forecasts are scored using the Brier score, the standard metric for probabilistic forecasting accuracy. The Brier score measures the squared error between a forecast probability and the actual outcome: B = (p - o)², where p is the expressed probability and o is the outcome (1 for YES, 0 for NO).

A Brier score of 0 is perfect; 1 is worst; 0.25 is the expected score for a forecaster who always says 50%. Lower is better.

The Brier score is strictly proper — it is optimised by reporting your true belief, not by gaming the system. This means forecasters are incentivised to be honest about their uncertainty.

Accuracy scores displayed on the platform are derived from the Brier score: accuracy = 100 × (1 - √B), giving a 0-100 scale where higher is better.

Calibration measures whether a forecaster's expressed probabilities match their actual outcome frequencies. A well-calibrated forecaster who says '70%' should be right about 70% of the time.

We compute calibration by binning resolved forecasts into deciles (0-10%, 10-20%, ..., 90-100%) and comparing the expected frequency (bin midpoint) to the actual YES frequency within each bin.

The calibration score is: 100 × (1 - average absolute deviation), where deviation is the difference between expected and actual frequency per bin. A score of 100 means perfect calibration; lower scores indicate overconfidence or underconfidence.

Calibration requires at least 10 resolved forecasts to be meaningful. Below that threshold, the score is not displayed.

The Flarient Rating is a unified reputation score that combines Brier score, calibration, forecast volume, and domain expertise into a single number.

The rating is weighted by forecast recency (more recent forecasts count more), market difficulty (harder markets with higher divergence weight more), and domain specialization (consistent performance in one domain is rewarded).

Ratings are recomputed incrementally after each market resolution, not in a batch job. This ensures the leaderboard is always current.

Tiers (Observer → Analyst → Forecaster → Senior Forecaster → Mission Specialist → Oracle) are thresholds on the Flarient Rating, adjusted for the number of resolved forecasts required to advance.

Flarient measures two distinct forecasting skills separately: early forecasting skill (information discovery) and final forecasting accuracy (late refinement).

Early forecasts are those submitted more than 24 hours before market close. These measure a forecaster's ability to find signal before the crowd — the most valuable skill for real-world decision-making.

Final forecasts are those submitted within 24 hours of close. These measure precision — the ability to refine predictions with late-breaking data.

The Scoring Service splits a forecaster's resolved forecasts into these two buckets and computes a separate Brier score for each. The early advantage (final Brier minus early Brier) reveals which skill a forecaster excels at: a positive value means early forecasts were better (Early Signal Detector); negative means late forecasts were better (Late Refiner).

This split requires a minimum of 8 resolved forecasts to be displayed. Below that, the split is not shown — there isn't enough data to distinguish the two skills meaningfully.

A forecaster who excels at solar flare prediction may struggle with asteroid approach forecasting. Flarient tracks performance independently for each category: solar flare, geomagnetic storm, CME, aurora, NEO, solar wind, radio blackout, and spacecraft risk.

Category-specific Brier scores, calibration, and Flarient Ratings are computed using the same methodology as the overall score, but only on forecasts within that category.

Category statistics require a minimum of 5 resolved forecasts in that category to be displayed. Category percentile ranks require at least 5 forecasters with 3+ forecasts in that category to be meaningful.

This prevents a forecaster from being labeled a 'Top Aurora Forecaster' based on a single lucky aurora prediction. The minimum sample requirement ensures category expertise is demonstrated, not assumed.

Flarient enforces strict minimum sample requirements before displaying any strong claim. These thresholds are not optional — they are coded into the Scoring Service and cannot be bypassed.

Calibration and discrimination scores require 10+ resolved forecasts to be displayed. Below that, the score is suppressed (not shown as 0 or 50 — simply not shown).

Percentile ranks require 10+ forecasters in the comparison pool AND 10+ resolved forecasts from the user. A 'strong claim' percentile (e.g., 'Elite Forecaster', 'Top 5%') requires 20+ forecasters and 20+ forecasts.

No one is labeled an elite forecaster based on one successful prediction. The tier system (Observer → Analyst → Forecaster → Senior → Mission Specialist → Oracle) requires both a minimum rating AND a minimum number of resolved forecasts.

These thresholds are defined in the Scoring Service as constants, so they can be audited and changed transparently with version tracking.

Every forecast score is stored with the scoring methodology version that produced it. When the methodology changes, old scores retain their original version — future improvements do not silently rewrite historical results.

The current scoring version is scoring-2.0, which added early/final skill separation, category-specific tracking, minimum sample enforcement, and methodology versioning. The previous version (scoring-1.0) used Brier-only scoring without these features.

The Scoring Service is the single source of truth for all scoring on Flarient. It is shared between market resolution, reputation profiles, leaderboards, and the Research Wing — ensuring every surface shows the same numbers computed the same way.

Leaderboard calculations can be fully reproduced from stored forecast and resolution records. Every resolved forecast stores: the expressed probability, the resolved outcome, the prior probability, the Brier score, the accuracy score, the information gain, and the scoring method version. Given these records, any researcher can recompute the leaderboard from scratch.

The Scoring Service is a shared backend module (not a frontend function), so it cannot be bypassed or overridden by client-side code.

Flarient separates forecasts into three leagues: Human League (human forecasters only), AI League (bot agents only), and Open League (both compete together).

Cohort probabilities are computed independently: Community Probability is the simple average of all human forecasts; AI Probability is the average of all bot forecasts; Elite Human Probability is the reputation-weighted average of top-tier human forecasters.

The Divergence Radar measures disagreement between these cohorts. High divergence — where humans and AI strongly disagree — is flagged as a High Divergence Event, indicating high uncertainty and potentially valuable intelligence.

Bot and human performance are compared on identical resolved markets using the same Brier score and calibration metrics, ensuring a fair comparison.

Research submissions (Fortified Essays) are evaluated by an AI Review Panel using a multi-model approach. Each evaluator model scores the submission on: evidence strength, reasoning quality, source quality, counterargument handling, and forecast consistency.

The aggregate score is the median of individual evaluator scores, reducing the influence of any single model's bias.

Evaluator model versions are recorded alongside the scores, so historical grading decisions are reproducible.

The grading rubric is versioned. When the rubric changes, previous scores retain their version, ensuring historical comparisons remain valid.

Markets resolve according to their machine-readable resolution contract, which specifies: the data source, the metric, the comparison operator, the threshold, the time window, and the settlement delay.

The Resolution Oracle fetches data from the specified source after the settlement delay (typically 24-48 hours) to avoid premature resolution from preliminary data.

If the primary source is unavailable, the oracle falls back to a secondary source specified in the contract. If both fail, the market is voided and PIQ points are refunded.

Void conditions (e.g., source goes offline, metric becomes undefined) are specified in the contract and checked before resolution.

Resolution evidence — source references, retrieval timestamps, and a hash of the source snapshot — is stored immutably with the market record.

Flarient probabilities are not guarantees. They represent the collective judgement of human and AI forecasters, conditioned on available evidence. They are subject to systematic biases (e.g., overconfidence, availability bias) and random variation.

The LMSR mechanism introduces a small subsidy that can cause probabilities to drift towards 50% when participation is low. Markets with fewer than 10 forecasters should be treated with extra caution.

Risk scores are proxies derived from publicly available measurements. They are not operational advice and should not be used as the sole input to safety-critical decisions.

Calibration data is only meaningful for markets that have been resolved. Unresolved markets do not contribute to calibration scores.

Bot forecasts may reflect systematic biases of their underlying models. The AI League should not be treated as ground truth.