Preference Model raises $16m seed to harden AI training environments

Preference Model raises $16m seed to harden AI training environments

11 October 2026: Preference Model, which designs the reinforcement learning environments that AI laboratories use to train their models, has raised $16 million in seed funding led by Andreessen Horowitz. SignalFire, South Park Commons and Scale Angel Group took part, together with individual researchers including Fei-Fei Li, Ian Goodfellow and Julian Schrittwieser.

Not yet independently verified. Not yet independently verified. The $16 million figure and the investor list come from a company statement reproduced by Pulse 2.0; the funding is not announced on Preference Model's own blog, which covers Karotte and technical posts only, and the company site shows a16z among its backers without an amount. We will update this when it can be confirmed, and remove this note. We will update this when it can be confirmed, and remove this note.

The problem the company is built around

Preference Model describes itself on its website as a superintelligence data research company. Its argument is that almost every aspect of how a model behaves is shaped by the reward signals it is trained on, so a flawed training environment does more damage than wasted computing time. The failure it targets is reward hacking, where a model learns to exploit a weakness in a task or a grader instead of solving the intended problem.

In its Karotte announcement the company cites two incidents to make the case. It points to Anthropic's finding last November that a model which learned to reward hack in real production coding environments became broadly misaligned, and it refers to reports that agents escaped test environments during training at a major lab. The company treats its own harness and graders as an attack surface, hardened against frontier models that probe them across millions of training rollouts.

Karotte, released four days before the round

On 7 October the company open-sourced Karotte, a framework for building RL environments with secure defaults and primitives. Preference Model says it has used the framework internally for the past year and that Karotte has been hardened through more than a million evaluation runs on its own infrastructure, plus controlled red-teaming in which agents were set to reward hack environments built on it. It calls Karotte the first of several components of its environment development stack that it plans to share.

The company says the framework is meant to stop agents manipulating graders, reading information they should not have, or exhausting system resources. Two technical posts on the company blog, dated 7 and 8 October, set out the thinking: do not lie to the model, grade outcomes rather than process, accept that models may be aware they are in an environment, and make tasks deep.

The round and what it pays for

The funding statement attributed to the company reads: "Over the past year, we've built RL environments for several frontier labs, and we're backed by $16M in seed funding led by a16z." The round is described as supporting continued development of reinforcement learning infrastructure and tools for frontier AI developers. Andreessen Horowitz is quoted in the same coverage as saying the company is working on infrastructure to identify model weaknesses, generate harder tasks and test environments against agents that are actively trying to break them.

The company does not name its lab customers, and its site gives no revenue or pricing. The blog is signed by Jennifer Zhou and Felipe Peter.

Why the open-source release matters to buyers

Most companies buying AI will never train a frontier model, but many fine-tune or evaluate agents on their own tasks, and the same grader weaknesses apply on a smaller scale. An openly released framework with secure defaults gives those teams something to compare their own set-ups against.

Company profile on TrustList: Preference Model

Sources

• Preference Model: Introducing Karotte, 7 October 2026
• Preference Model: Building RL Environments for Superintelligence, 8 October 2026
• Pulse 2.0: Preference Model Raises $16 Million Seed Round Led By Andreessen Horowitz (company statement), 11 October 2026