Methodology

Continual reinforcement learning with delayed market feedback.

Peaceful Beaver is the public name of the deployed policy. At each decision time it maps the information then available to an action. Market outcomes are labeled later and used in controlled batch updates. Published decisions are not revised.

State and observations

Issuer state is only partially observed, and market dynamics change over time. We therefore use POMDP notation. The observation $$o_t$$ contains news available by the decision time, issuer evidence, price response, liquidity and market regime. The memory state $\mathcal{M}_t$ contains only records available at or before $$t$$ .

b_t=q\!\left(s_t\mid o_{\le t},\mathcal{M}_t\right)

The belief state is computed on the original decision clock. Later prices and outcomes are excluded.

Policy and risk constraint

The Scout Policy, $\pi_{\mathrm{S}}(a\mid b_t)$ , proposes candidate actions. The Risk Critic estimates downside cost and constraint violations for those actions. It may reject a proposal but does not add candidates. The Portfolio Arbiter applies the policy and risk thresholds and records the final action.

a_t^{*}=\arg\max_{a\in\mathcal{A}}Q_R(b_t,a)\quad\text{subject to}\quad Q_C(b_t,a)\le\varepsilon

\(Q_R\)

is reward value;

\(Q_C\)

is constraint cost;

\varepsilon

is the admission threshold.

Reward definition

Reward is not fully observed when an action is published. It is labeled over fixed later horizons. The reward includes benchmark-relative return, adverse excursion, thesis persistence, false-positive cost and action cost. Coefficients are fixed for an evaluation cycle so that challenger policies are compared on the same objective.

r_t=\alpha R_t^{\mathrm{excess}}-\beta\,\mathrm{MDD}_t+\eta D_t^{\mathrm{thesis}}-\kappa C_t^{\mathrm{false}}-\lambda C_t^{\mathrm{action}}

The objective is risk-adjusted policy value rather than directional accuracy alone.

Trajectory and credit assignment

Each decision has a stable identifier linking its observation state, action, source timestamps and later outcome labels. Opportunity selection, risk review and final arbitration are stored separately. This permits component-level error analysis without changing the original record.

\tau_t=\left(b_t,a_t,r_{t:t+H},b_{t+H}\right)

The outcome window is attached to the action recorded at

\(t\)

Replay memory

The replay set stores point-in-time trajectories. Hot, warm and cold are retention tiers: active cases, comparable prior regimes and consolidated long-horizon cases. Sampling weights may use recency, state similarity, policy disagreement and temporal-difference error. Retrieval is bounded and respects the decision timestamp.

\delta_t=r_t+\gamma V(b_{t+1})-V(b_t)

\delta_t

is used for critic calibration and replay priority.

Batch policy update and evaluation

Production data are generated by behavior policy $\mu$ . A challenger $\pi_{\theta'}$ is fitted on discovery windows, tuned on separate calibration windows and evaluated once on a frozen holdout. Promotion requires higher estimated policy value and no deterioration in downside, evidence or stability constraints. The production policy does not update after an individual request or outcome.

\pi_{\mathrm{new}}\leftarrow\operatorname{SafeImprove}\!\left(\pi_{\mathrm{behavior}},\mathcal{D}_{\mathrm{replay}}\right)

A promoted policy is versioned, deployed atomically and can be rolled back.

Computation policy

Research depth is included in the action. Routine states may stop after the standard pass. Weak evidence, unusual states or disagreement between policy and critic can request additional research. Compute cost is included in the objective.

a_t=\left(a_t^{\mathrm{market}},a_t^{\mathrm{compute}}\right),\qquad \widetilde{r}_t=r_t-\lambda C\!\left(a_t^{\mathrm{compute}}\right)

Additional computation is used only when its estimated information value exceeds its cost.

Relation to True Intrinsics

The True Intrinsics research program starts from the Big World Hypothesis: the environment is larger and less stationary than a fixed model. Here that assumption is implemented through point-in-time state estimation, replay across market regimes, delayed reward labels and repeated policy evaluation. Market outcomes provide the external feedback signal.

Publication record

A dated page is generated only after the decision and memory snapshot pass publication checks. It contains the action, evidence state and decision time recorded for that date. Later outcomes may affect a future batch update, but they do not alter the published page. Public evidence may identify news publishers, company releases and regulatory filings; internal services and licensed-data implementation details are omitted.

Public name: Peaceful Beaver. Internal service and release identifiers are omitted. The equations describe the policy design; they do not imply that an action or value estimate is correct.