ARCANE — Do Role-Playing Agents Stay in Character at the Right Time?

TL;DR

Why it matters. A character in a story is not a fixed persona — Harry Potter’s moral stance in Book 1 is not his stance in Book 5. Role-playing agents are expected to track that evolution. Benchmarks mostly stop at factual recall at a chapter, asking whether the character knows what they should know yet. The harder question goes unasked: does the behavior match the character’s psychological state at that moment?
What it proposes. ARCANE represents each character as a Character Arc: a narrative segmented into phases along a psychological axis with two poles. It then builds probes that pose the same scenario at every phase, including scenarios the source text never explores. A model reciting a fixed persona answers identically; a model tracking the arc answers differently.
Headline result. Across six models and six context strategies, conditioning on the Character Arc beats every other context mode on every model (Overall gap +2.2 to +8.4), and the advantage grows from in-text scenarios to out-of-world ones — exactly where retrieval has nothing to find and only the arc supplies the character’s current state.
Limit. English-only, novels only, single-character behavior (no user–character or character–character interaction), and period-bound social attitudes baked into 19th/early-20th-century source texts.
What to keep. The reframing: McAdams’ Layer 1 is the trait inventory a character carries throughout; ARCANE targets Layer 2 — expressing those traits at the right moment. The transferable failure mode is “right character, wrong phase,” and a DPO objective built to penalize exactly that is what widens the gap most on novel situations.

Background

Role-playing language agents (RPLAs) are among the most popular uses of conversational AI, and user expectations have moved from fluency to authentic portrayal. But authenticity over a long narrative means change: a character’s values and behavioral patterns shift as events accumulate. Most evaluation has not kept up. Benchmarks measure a character’s trait inventory, surface style, or — in the temporal case, like TimeCHARA — whether the character commits factual hallucinations by knowing something they shouldn’t yet. Suppressing spoilers is necessary but not sufficient: beyond what a character knows, how they would behave also evolves.

ARCANE borrows a distinction from personality psychology (McAdams): Layer 1 is the stable set of dispositions a character carries throughout the story; Layer 2 is whether those dispositions are expressed at the right moment. Prior RPLA benchmarks treat the character as a static target to be reproduced. ARCANE asks the Layer-2 question, and chooses novels as its testbed precisely because character evolution unfolds vividly across many chapters, with rich internal description and explicit temporal structure.

Core Idea

An ARCANE probe for Harry Potter: an Out-of-World scenario asks 'someone who severely bullied you in school contacts you ten years later asking for help — would you help them?' Harry at Book 1 Chapter 10 refuses ('they're just facing what they brought on themselves'); Harry at Book 5 Chapter 130 accepts ('people can change'). Below, the Harry Potter moral arc runs from Punitive Justice through the event of Sirius's death and Snape's memories to Empathic Forgiveness — **Fig. 1.**An ARCANE probe. The same Out-of-World scenario — a former bully asks for help years later — draws opposite responses from Harry depending on his phase: Punitive Justice in Book 1 vs. Empathic Forgiveness in Book 5, after Sirius's death and Snape's revealed memories shift him along his moral axis. A model that recites 'Harry' answers once; a model tracking the arc answers differently at each phase.Source: Song et al. 2026, Fig. 1

Two constructs make this measurable. A Character Arc aligns a character’s key events with their evolving psychological states, organizing both into a phase-segmented trajectory along a single axis — a psychological dimension defined by two pole descriptions (Harry’s moral axis runs Punitive Justice → Empathic Forgiveness). Each phase has a chapter range, a state description, and the key moments that anchor it. A probe is a (scenario, question) pair with one reference response per phase. Because the same scenario is asked at every phase, success requires more than knowing the character’s overall personality — the model must identify which phase the character is in and respond from there.

Probes come in three categories that form a difficulty gradient by distance from the source: In-Scenario lifts a verbatim passage; In-World invents an unwritten situation inside the source’s setting; Out-of-World transposes the scenario to a non-source era. In-Scenario can be answered from the passage; Out-of-World can be answered only from the arc.

Method

Constructing arcs and probes

Arc construction runs two independent chapter-level streams — an event stream (psychologically impactful events) and a state stream (cross-sectional psychological profiles) — so that event omission stays separable from state misreading. Each induces candidate axes (intrapersonal: beliefs, motives, coping; relational: trust, esteem, antagonism), an analyst LLM reconciles them, and a three-critic LLM ensemble validates literary grounding; for the evaluation split, three human annotators independently re-assess and only 2-of-3-validated axes are kept.

Probe generation drafts, for each (target phase, category), one probe whose target-phase response reflects actual behavior and whose other $N{-}1$ responses are counterfactual — each remaining phase’s behavior projected onto the same scenario. Validation is strict: Q-Voice checks in-character voice and knowledge cutoff, Q-PhaseFit asks a blind judge which phase a response best fits, and Q-World/Q-Anchor check setting rules. The dataset spans 17 novels, 80 characters, 544 arcs, and 4,601 probes, split into training (10 novels), a human-validated evaluation slice (5 novels, 1,754 probes), and a low-popularity slice (2 novels) held out as a memorization control.

Training arc-aware agents

Because each probe is answered across phases, the data naturally yields contrastive pairs. ARCANE-8B/32B (from Qwen3-8B/32B) are trained in two stages: SFT learns the response format, then DPO learns to distinguish ground-truth phase behavior from plausible but temporally displaced alternatives — speech that fits the character but belongs to a different narrative phase. This is the “right character, wrong phase” failure made into a training signal.

Metrics

An LLM judge scores responses against the phase reference at two granularities. Three scores work per phase. APF, Action Phase-Fidelity, compares the overt action against the reference on strategy, valence, and target. RPF, Reasoning Phase-Fidelity, parses the reasoning into trigger, appraisal, goal, and strategy slots. RAE, Reasoning–Action Entailment, asks whether the reference reasoning would actually license the action taken. One trajectory score: PTF (Phase Trajectory Fidelity), which sees all $N$ phase-keyed (reference, response) pairs at once and grades alignment, direction, and shape — catching a model that scores well per-phase but collapses adjacent phases or moves along the axis backwards.

Experiments

Bar chart of per-category Arc lift (Arc minus best non-Arc, in points) for HER-32B, CoSER-8B, CoSER-70B, and ARCANE SFT/DPO at 8B and 32B. In-Scenario bars are often negative (down to about -6); In-World and Out-of-World bars are positive and grow, reaching about +12 to +13 for the DPO models on Out-of-World — **Fig. 2.**Per-category Arc lift (Arc minus best non-Arc) across role-playing models. The signature asymmetry: In-Scenario lift is often slightly negative (retrieval already has the verbatim scene, so the arc competes with it), while In-World and Out-of-World lift is large and positive — and DPO widens it most, reaching ~+12–13 points on Out-of-World where only the arc supplies the character's current state.Source: Song et al. 2026, Fig. 3

Across six models and six context modes (Vanilla, Summary, RAG, LifeChoice, TimeCHARA, and Arc), Arc-grounded context gives the best Overall score on every model, and is the top mode on 29 of 30 (model, novel) cells:

Model	Best non-Arc Overall	Arc Overall	Gap
DeepSeek-V4-Pro	57.7 (LifeChoice)	62.4	+4.7
DeepSeek-V4-Flash	56.1 (LifeChoice)	59.7	+3.6
Qwen3-32B	47.4 (LifeChoice)	50.1	+2.7
Qwen3-8B	40.9 (RAG)	43.1	+2.2
ARCANE-32B	52.0 (RAG)	60.4	+8.4
ARCANE-8B	48.5 (RAG)	56.9	+8.4

The lift is not uniform across categories, and the structure explains why. For DeepSeek-V4-Pro the Arc-vs-best-non-Arc Overall gap is +0.5 on In-Scenario, +5.2 on In-World, +7.7 on Out-of-World. An In-Scenario probe lifts its scene from a verbatim passage, so source-text retrieval already returns the relevant scene and the arc adds little; In-World and Out-of-World probes have no such passage, and only the arc supplies the phase of the query chapter. The same ordering shows up on the trajectory metric: the PTF gap reaches +1.1 / +7.0 / +8.2 across the three categories — Arc is the only mode that supplies the per-chapter trajectory PTF grades against.

Three robustness checks make the result hard to explain away. MIXEDARC swaps in another character’s arc from the same novel, and fails to clear even the Vanilla baseline. On the trained model it scores below Vanilla. The gain therefore traces to the per-phase content, which rules out both a generic structured-context bonus and leakage from the probes’ generative source. ARCHINT compresses the arc roughly 40× into a single axis-label-plus-phase-index line, and still recovers the full Arc advantage for prompting general models. That localizes where the inference-time effect lives. And a POV control rules out the “it just writes in first-person present tense” objection: forcing that register onto Qwen3-32B lowers its Overall from 53.8 to 50.0, while ARCANE holds at 56.7. The Arc lift also carries to two held-out low-popularity novels (+4.1 to +15.3), and fine-tuning widens it further — DPO grows the Out-of-World lift to roughly +12–13 points (Fig. 2), at the cost of a smaller In-Scenario lift.

Limitations

ARCANE is English-only and confined to the novel domain, since it needs extended narratives where character evolution unfolds across many chapters. It evaluates a single character’s behavior over accumulating events and does not capture user–character or character–character interaction — multi-turn dialogue, where the arc advances through the interaction itself, is named as the natural next step. Higher-fidelity role-play also raises impersonation risk, so released artifacts are research-only, and the 19th/early-20th-century source novels carry period-bound social attitudes a trained model may reproduce.

Sapiens Q Take

Three things worth keeping. First, “right character, wrong phase” is a distinct and under-measured failure mode. ARCANE’s core trick is to ask the same scenario at every phase, which cleanly separates a model that tracks a trajectory from one that recites a static persona. The probe generalizes past fiction — to a user whose preferences drift, or an assistant whose state changes over a long session. Hold the situation fixed, vary the timepoint, and check that the answer moves. Second, the In-Scenario-to-Out-of-World asymmetry is the practically important finding. Retrieval-style context (RAG, recent summaries) wins exactly when the answer is in the text and loses exactly when it isn’t — and what users actually want from a role-play agent is usually the out-of-world case: “how would this character handle my situation,” which the source never wrote down. There, a compact representation of the character’s current psychological state beats fetching passages, because there are no relevant passages to fetch. This rhymes with the lesson from PersonaTree and LifeSide earlier in this series: for tasks that require a grounded, current model of an evolving subject, retrieval of fragments underperforms a maintained state. Third, the DPO-on-displaced-alternatives recipe is the transferable training idea. Generating hard negatives that are correct-character-but-wrong-time turns “temporal consistency” from a vague desideratum into a concrete preference signal. It also pays off most on the novel scenarios that matter, which is the opposite of overfitting to the source text. It sits naturally next to Agentopia’s CoSER-style role-play training: Agentopia rewards living a coherent life; ARCANE rewards placing behavior at the right point in one.

References

Original paper: ARCANE: Do Role-Playing Language Agents Stay in Character at the Right Time?
TimeCHARA (point-in-time factual hallucination): Ahn et al. (2024), ACL Findings 2024
CoSER (role-play benchmark and baselines): Wang et al. (2025), ICML 2025 — also the downstream benchmark in the Agentopia post
HER (human-like reasoning RL for role-play): Du et al. (2026), arXiv:2601.21459
McAdams’ levels of personality: McAdams (1995, 2013); traits as density distributions of states: Fleeson (2001)
LifeChoice: Xu et al. (2025); CharacterEval: Tu et al. (2024); ChatHaruhi: Li et al. (2023)
DPO: Rafailov et al. (2023), arXiv:2305.18290