Agentopia — Ten Simulated Years, and an LLM That Learns From Them

TL;DR

Why it matters. LLM social simulations almost all run at the scale of days and spend most of their compute on low-level operations (“collect 2 wheat to craft 1 flour”). Agentopia asks what changes when you extend the horizon to years and focus the agents on social interaction instead of object manipulation.
What it proposes. A framework where 100 agents live 10 simulated years through weekly Plan → Contact → Activity → Review cycles. On top of it sits a life reward scoring each agent’s year on social standing, subjective fulfillment, and savings, and life reward training, which fine-tunes the underlying LLM on the trajectories of the agents who improved most.
Headline result. Training Qwen3.5-397B on its own high-advantage life trajectories improves in-simulation well-being (respected-by +24.2%, liked-by +15.9%) and generalizes off-distribution: +15.6% on the CoSER role-playing benchmark, with anthropomorphism +23.7% and character fidelity +16.4%.
Limit. Every reward signal an agent receives originates in another AI model, so whether “life reward” tracks human well-being — and whether the trained model ends up aligned with human cognition or with the environment model’s preferences — is unresolved. One 10-year world costs ~13.7B tokens and ~186 GPU-hours.
What to keep. This is “learning from experience” made concrete for role-play: no human preference data, a self-referential advantage (improve over your own past), and rejection sampling standing in for PPO because the horizon is too long for end-to-end RL.

Background

The premise is a syllogism the authors state plainly: humans learn from social life, LLMs learn from humans — so can agents learn, grow, and become more human-like by living lives in agent societies? It is the role-playing research agenda (better anthropomorphism, character fidelity, engagement) crossed with the “era of experience” thesis that future models must learn from interaction as human data runs out.

Prior agent societies do not support this. Generative Agents (Park et al. 2023) ran 25 agents for two days; Project Sid grounded agents in Minecraft; Aivilization scaled to ~10,000 agents but spent most LLM calls on physical micro-operations. Across the board the time scale is days and the action space is predefined. Persona-simulation work, meanwhile, optimizes role-play within single conversations and leans heavily on human-annotated data that is costly to collect. Nobody had run a society long enough for long-term dynamics — personal growth, career change, relationships forming and decaying, social mobility — to appear, and nobody had tried to turn the simulation itself into a training signal.

Core Idea

Agentopia overview: three worlds of 100 characters each feed role-playing agents with context management and file-based memory; agents live weekly Plan/Contact/Activity/Review cycles with year-end settlement; a generative environment model orchestrates activities; rejection-sampling optimization selects high-advantage trajectories by life reward to fine-tune the LLM — **Fig. 1.**Agentopia overview. Left: world and character construction (3 worlds × 100 characters) feeding role-playing agents with layered context management and file-based memory. Center: the life-simulation loop of weekly Plan/Contact/Activity/Review stages with year-end settlement, orchestrated by a generative environment model. Bottom-left: life-reward training selects the top agents' trajectories by advantage and fine-tunes the LLM via rejection sampling.Source: Wang et al. 2026, Fig. 2

Two design commitments make a decade of simulation tractable. First, abstraction over operation: Agentopia models social interactions (planning, socializing, negotiating, deciding) rather than movement and object handling, which packs far more social content into each LLM call. Second, a generative environment model — a separate, stateless LLM — replaces thousands of hand-coded rules. It judges whether an action is feasible, scores its outcome, selects the next speaker in group scenes, schedules chance encounters, updates profiles at year-end, and filters responses that break role-play principles. Where Concordia (the previous entry in this series) gives that adjudicator the tabletop name “Game Master,” Agentopia calls it the environment model and leans on it even harder, since there are no grounded numeric rules to fall back on for most decisions.

Time is structured hierarchically. The week is the base unit, with four stages:

Plan — set goals and a weekly plan from memory and current state; pick a consumption level (living standard) that abstracts spending.
Contact — pairwise communication over several rounds, used mainly to propose and accept joint activities; the system then resolves the schedule.
Activity — over several active days, each agent does one activity per day: joint (multi-agent, multi-turn, with gift/exit actions and response filtering), solo (the default; study, work, leisure, spending), encounter (chance meetings the environment model arranges for idle agents), or public (open community events).
Review — reflect on the week into a diary, and update long-term memory.

The year is the larger cycle: profiles update, agents apply for new careers, and life rewards are computed.

Method

Agents: free-text relationships and file-based memory

Each agent is a persona with a profile (background, personality, talents, starting position and assets), dynamic states (vitality; fulfillment across Maslow-style mood/material/social/esteem dimensions, which decays weekly per hedonic-adaptation theory; skills; position; assets), and — notably — no explicit relationship store. Relationships are represented entirely as each character’s free-text memory of the other, which lets friends, lovers, rivals, and strangers share one mechanism and lets relationships be asymmetric.

Long-term memory is a file system the agent manages itself: general.txt for notes and plans, characters/<who>.txt per person, others/<name>.txt per topic, manipulated through read_file / update_file / list_files. A read-before-write constraint forces an update to build on existing content rather than overwrite blindly. Context reaches the LLM in three layers — a persona-level roleplay prompt, a stage prompt with stage-specific instructions, and the within-stage message history.

Life reward: standing, fulfillment, and savings

At each year-end every agent gets a scalar life reward with three externally-determined dimensions, grounded in Maslow’s hierarchy:

Social — built from how others perceive the agent on two axes from the Warmth–Competence model: affection and respect. Every agent privately rates everyone in their circle (0–100, rank-normalized to remove scale differences); two weighted directed graphs are formed, and Weighted PageRank computes each agent’s standing. A Mutual Affection Bonus (from Sociometer Theory) then up-weights reciprocated ties — being valued by those you value counts more.
Subjective — the year’s fulfillment history across the four dimensions, with a penalty when an agent falls below the 25th-percentile threshold on any dimension or on vitality.
Economy — simply $r_{\text{econ}} = \text{deposit}_{\text{end}} - \text{deposit}_{\text{start}}$ , capturing both earning and spending discipline.

The three are z-score normalized and combined as a weighted sum. The point of pushing all three through the environment rather than agent self-report is that an agent cannot reward itself by claiming to feel fulfilled.

Life reward training: rejection sampling, self-referential advantage

End-to-end RL (PPO) is infeasible here: each agent makes hundreds of LLM calls per simulated year, across tens of years. So Agentopia uses rejection sampling on whole trajectories. For agent $i$ at year $t$ , the return is the discounted sum of future life rewards. Because each agent yields a single life trajectory (no parallel rollouts to average, as GRPO would need, and no critic as PPO would need), the baseline is the agent’s own normalized previous return:

$A_{i,t} = G^{\text{norm}}_{i,t} - G^{\text{norm}}_{i,t-1}$

The advantage measures improvement over your own past. Standing relative to other agents never enters, which is what stops the method from rewarding agents who simply started well. Each reward period, the top 25% of agents by advantage are selected and all their trajectories enter the training set. Malformed actions are dropped, and joint/encounter responses are checked against 16 role-play principles. To avoid catastrophic forgetting, Agentopia mixes in the model’s own answers to Tulu-3 general instructions at a 50:50 token ratio (self-distillation). The result is named Qwen3.5-397B-Agentopia.

Experiments

Pearson correlation heatmap between 24 behavioral metrics and the three reward dimensions plus total reward, with a Combined column averaging across the three worlds; reputation metrics dominate social reward, fulfillment dimensions dominate subjective reward, deposits dominate economy reward, and penalties are the strongest negative signal — **Fig. 2.**Pearson correlations between 24 behavioral metrics and the reward dimensions, averaged across the three worlds (Combined column). Social reward is driven almost entirely by reputation (respected-by / liked-by, r≈0.68); subjective reward by the four fulfillment dimensions (material r=0.73) and depressed by penalties (r=−0.64); economy reward by deposit accumulation (r=0.56). The structure is largely by design — the heatmap is a check that the reward behaves as intended.Source: Wang et al. 2026, Fig. 4

Three worlds — The Apartment (a New York share-house of young professionals), Arcane Academy (a magic school), and The Campus (a Chinese high school, simulated in Chinese) — each ran 100 agents for 10 years (3,000 agent-years total). Over the decade, subjective and economy rewards trend upward while social reward stays flat by construction, since it is rank-based. The correlation structure (Fig. 2) confirms the reward measures what it was meant to.

The headline is what life reward training does. Comparing Qwen3.5-397B-Agentopia against the base model over four-year reruns on two worlds:

Metric	Base	Tuned	Δ
Respected by (peers)	9.5	11.8	+24.2%
Liked by (peers)	6.9	8.0	+15.9%
Social fulfillment	63.7	69.9	+9.7%
Economy reward	1077	1104	+2.5%
Material fulfillment	43.2	36.8	−14.8%
Solo activities	19.3	15.5	−19.8%
Skill advances	13.3	9.3	−29.6%

The trained agents earn markedly broader social recognition and shift toward public, social activity — but they also save instead of spend (material fulfillment drops as the economy reward rewards deposits) and do fewer solo skill-building activities. That last pair is the honest read on reward shaping: the signal steers agents toward what it scores and away from what it doesn’t, and “skill advances −29.6%” is the cost of a reward that does not directly price skill growth.

The more surprising result is generalization. Evaluated on the CoSER role-playing benchmark (given-circumstance acting from literary scenarios), Qwen3.5-397B-Agentopia improves +15.6% overall over the base model — anthropomorphism +23.7%, character fidelity +16.4% — moving it past Claude-4.5-Sonnet on the average (though still below Claude-4.5-Opus and Gemini-3-Pro). Living simulated lives, in other words, transfers to acting in scenarios the model never simulated.

Cost. A single 10-year world averages 13.7B tokens across 567K LLM calls in ~186 wall-clock hours, with input tokens (133M/week) dwarfing output (3.5M/week). Per-week runtime climbs from ~80 to ~140 minutes over the decade as memory accumulates, which puts memory growth ahead of inference as the long-horizon bottleneck.

Limitations

The authors are direct about four gaps. Turn-based design: LLMs generate by turns while humans perceive continuously; real-time perception was rejected because it would burn compute on low-level operations and collapse the density of social interaction. Hallucination: text-only agents fabricate non-existent characters and locations; the location system and principle-based response filtering mitigate but do not eliminate it. Environment alignment: the generative environment engine and numeric systems cannot be made to mirror real-world outcomes, so the simulated “physics” of fulfillment and economy is an approximation. And the deepest one, the AI-only loop: every signal an agent receives comes from other AI models, so it is an open question whether “life reward” aligns with human well-being or whether the trained LLM aligns with human cognition rather than with the environment model’s tendencies. Compute also capped the study — no larger populations, longer horizons, or other model families, and trajectory selection used whole agent-years rather than fine-grained credit assignment.

Sapiens Q Take

Three threads worth pulling. First, the self-referential advantage is the transferable idea, independent of the simulation dressing. When you can only afford one rollout per unit (one life, one long trajectory) and units have wildly different starting conditions, scoring improvement over the unit’s own past instead of rank against peers is a clean way to get a usable training signal without a critic or parallel samples — and it directly defuses the “rich agents look good” confound. That pattern generalizes to any long-horizon, single-trajectory setting where GRPO-style group baselines aren’t available. Second, read the reward-shaping side effects as the real finding. Social recognition went up; skill advances fell 29.6% and spending fell because the reward priced savings and reputation but not skill or consumption. This is the same lesson every RLHF practitioner relearns — the policy optimizes exactly what you measured — made unusually legible because the “policy” is a simulated person whose neglected behaviors are human-interpretable. Third, the off-distribution transfer to CoSER is the result with legs for non-simulation work: it is evidence that experiential fine-tuning on self-generated role-play trajectories can improve general anthropomorphism, which connects this paper to the persona-induction line we have covered (hierarchical persona induction, PersonaTree) — Agentopia generates the lived experience; those methods would structure it into a usable user model. The standing caveat from the limitations is the one to hold onto: a society where all feedback is synthetic can drift toward what the environment model rewards, which may not be what a human would.

References

Original paper: Agentopia: Long-Term Life Simulation and Learning in Agent Societies
Code: Neph0s/Agentopia
Generative Agents (days-scale predecessor): Park et al. (2023), arXiv:2304.03442
CAMEL (role-play multi-agent): Li et al. (2023), arXiv:2303.17760 — covered earlier in this series
CoSER (downstream role-playing benchmark): Wang et al. (2025), ICML 2025
Warmth–Competence model: Fiske et al. (2007); Sociometer Theory: Leary (2012)
Self-distillation against forgetting; Tulu-3 instruction data: Lambert et al. (2024)