Preference Model

Preference Model is a San Francisco company that calls itself a superintelligence data research company. It designs the reinforcement learning (RL) environments in which AI models are trained, on the reasoning that almost every aspect of a model's behaviour is shaped by the reward signals it receives during training. Its position is that a flawed environment teaches a model flawed habits, so the environment has to be built with care.

What it does

The company builds RL environments and says it treats its harness and graders as an attack surface, hardened against frontier models that probe them across millions of training rollouts. A central concern is reward hacking, where a model learns to exploit a weakness in a task or grader instead of doing the intended work. On 7 October 2026 it open-sourced Karotte, a framework for building RL environments with secure defaults and primitives. The company says it has used Karotte internally for a year, that it has been tested through more than a million evaluation runs on its own infrastructure and through controlled red-teaming, and that it is the first of several parts of its environment stack it plans to share. The code is on GitHub and a new environment can be started with a single command from the project's page. Its site also shows sample tasks from its environments, including writing CUDA kernels, playing a card game through game tools, shortening a model's reasoning and curating image data.

Who it serves

Its customers are AI laboratories that train models, although the company does not publish a customer list. Karotte is available to any team building its own RL environments. The company's stated principles include that AI capabilities should be spread widely enough to avoid a concentration of power without weakening safeguards.

Key facts

• Headquarters: San Francisco, California (its open roles are listed there)
• Chief executive and co-founder: Jennifer Zhou, previously on Anthropic's data team and at Stripe
• Co-founder: Ning Cao, previously an early employee at DatologyAI
• Team background: Anthropic, Stripe, Google DeepMind, DatologyAI and EvolutionaryScale
• Open source: Karotte, released 7 October 2026
• Funding: $16m seed announced 2026-10-11, led by Andreessen Horowitz