Research Engineer | Evals

Dyssonance AI; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Dyssonance AI seeks an Evals Engineer to shape how we measure progress toward a fundamentally different AI. You will work at the intersection of research, engineering, and experimental science to translate novel capabilities into falsifiable hypotheses and to design benchmarks when existing ones fall short.

You will establish meaningful baselines, build evaluation infrastructure, and influence research directions by exposing confounders and designing robust experiments that reveal why systems

Qualifications

• Strong software engineering skills, particularly in Python and modern ML tooling.
• Experience designing evaluations for machine learning systems, language models, or related tech.
• Excellent experimental-design instincts, including controls, ablations, confounders, and reproducibility.
• Experience developing datasets, benchmarks, evaluation protocols, or annotation methodologies.
• Strong quantitative reasoning and the ability to distinguish meaningful evidence from misleading metrics.
• Ability to translate ambiguous research questions into concrete experiments and working software.
• Clear technical communication skills to explain experiments and next steps.

Responsibilities

• Invent new ways to measure intelligence through evaluations for stateful reasoning, coherence, and continual reasoning under uncertainty.
• Turn research claims into testable hypotheses with observable behaviors and measurable criteria.
• Build novel benchmarks and datasets to evaluate emerging capabilities.
• Establish rigorous baselines by comparing against frontier models and ablations.
• Design experiments that explain results and identify underlying mechanisms.

Skills

Experiment design

Data analysis

Clear technical communication

Research-to-software translation

Tools

Python

ML tooling

Job description

Research Engineer | Evals

About Dyssonance

Dyssonance AI is an intelligence research lab building a fundamentally different approach to artificial intelligence: one that models thought, not language.

Founded by a particle physicist and former Google DeepMind leader, Dyssonance draws on principles from physics and cognitive science to develop a new architecture for intelligence. At its core is Continual Reasoning Under Uncertainty (CRUU), a framework that enables AI systems to form, maintain, and evolve beliefs as they encounter incomplete, ambiguous, or contradictory information.

Rather than relying on language prediction as the foundation of intelligence, Dyssonance builds stateful reasoning systems designed to identify inconsistencies, seek coherence, exercise judgment, and proactively surface insights that humans may not know to ask for.

We're challenging fundamental assumptions about how AI should work and building the science, architecture, and infrastructure to explore what comes next.

The Challenge

How do you evaluate an AI system when the benchmarks you need don't exist yet?

Most established AI benchmarks assume a model receives a prompt, produces an answer, and can be scored against a known target. But what happens when the system you're evaluating doesn't fit that paradigm?

How do you measure whether an AI recognized an important problem nobody explicitly asked it to solve? Whether it revised its beliefs appropriately when new evidence emerged? Whether it maintained coherence across changing information? Whether it identified what mattered without being told where to look? And how do you establish a meaningful baseline when the systems you're comparing operate on fundamentally different architectural principles?

Those are all very deep open research questions, and we're looking for an Evals Engineer who wants to help answer them.

The Role

You will help define how we measure progress toward a fundamentally different kind of AI.

This is not primarily an eval-harness or leaderboard role. You'll work at the intersection of research, engineering, and experimental science: translating novel capabilities into falsifiable hypotheses, designing experiments that isolate them, inventing benchmarks when existing ones are inadequate, and building the infrastructure to evaluate our systems rigorously and continuously.

A major part of your work will involve establishing meaningful baselines. Dyssonance differs from conventional language models in ways that can make naïve comparisons misleading. You'll need to determine what constitutes a fair comparison, identify what existing benchmarks actually measure, expose confounders, and design experiments that distinguish genuine architectural capabilities from advantages attributable to prompting, compute, context, or other factors.

We want someone who questions metrics rather than accepting them simply because they're widely used.

What You'll Do

• Invent new ways to measure intelligence. Design evaluations for capabilities that are difficult to capture with conventional benchmarks, including stateful reasoning, coherence, epistemic awareness, saliency awareness, proactivity, and continual reasoning under uncertainty.

• Turn research claims into testable hypotheses. Define observable behaviors, measurable criteria, experimental controls, and falsifiable predictions for capabilities that may not yet have established evaluation methodologies.

• Build novel benchmarks and datasets. Identify where existing benchmarks provide meaningful signal, where they fall short, and where entirely new evaluation protocols, datasets, rubrics, or adversarial test cases are needed.

• Establish rigorous baselines. Compare Dyssonance against frontier language models, conventional architectures, ablations, and previous versions of our own systems, accounting for differences in compute, context, state, and interaction paradigms.

• Design experiments that explain results. Go beyond performance scores to isolate why a system behaves differently, distinguish competing hypotheses, and identify the mechanisms behind successes and failures.

• Develop evaluation methodologies. Combine automated evaluation, human assessment, model-based judges, statistical analysis, and qualitative investigation as appropriate, while validating the reliability of the evaluators themselves.

• Build reproducible evaluation infrastructure. Develop pipelines, harnesses, and tooling that make experiments repeatable, results independently verifiable, and important metrics continuously trackable as the system evolves.

• Find the weaknesses in our own evidence. Detect benchmark contamination, evaluator bias, metric gaming, hidden confounders, and other sources of misleading conclusions.

• Influence research direction. Work closely with researchers and engineers to turn evaluation findings into actionable hypotheses, architectural improvements, and research priorities.

What We're Looking For

You're unusually good at turning fuzzy questions into rigorous experiments.

You don't need a benchmark to already exist before you can measure something. You can start with a claim like "this system recognizes important information without being explicitly asked about it" and work backward: What observable behavior would demonstrate that? What alternative explanations need to be ruled out? What would the control be? How could the benchmark accidentally reward the wrong behavior?

You combine a researcher's skepticism with an engineer's instinct to build.

You bring

• Strong software engineering skills, particularly in Python and modern machine learning tooling.

• Experience designing, implementing, or analyzing evaluations for machine learning systems, language models, agents, reasoning systems, or related technologies.

• Excellent experimental-design instincts, including a strong understanding of controls, ablations, confounders, statistical uncertainty, and reproducibility.

• Experience developing datasets, benchmarks, evaluation protocols, or annotation methodologies.

• Strong quantitative reasoning and the ability to distinguish meaningful evidence from misleading metrics.

• The ability to translate ambiguous research questions into concrete experiments and working software.

• Comfort operating in an environment where the methodology, baseline, and success criteria may all need to be invented.

• Clear technical communication skills, including the ability to explain what an experiment establishes, what it doesn't, and what should be investigated next.

Nice to have

Experience with one or more of the following is particularly relevant:

• Cognitive science, experimental psychology, or psychometrics.
• Evaluation of long-horizon, stateful, or autonomous AI systems.
• Causal inference, experimental design, or scientific measurement.
• LLM-as-judge methodologies, human evaluation, and evaluator calibration.
• Research involving reasoning, uncertainty, belief revision, or decision-making.
• Developing novel benchmarks or evaluation methodologies in an academic or industrial research setting.

We care more about your ability to reason rigorously and build effective experiments than about a particular degree, academic background, or previous job title.

How You'll Work

You'll work closely with our research and platform teams, helping establish the experimental foundations for an architecture that challenges many conventional assumptions about AI.

Some weeks, you may be building evaluation pipelines against thousands of reasoning trajectories.

Others, you may be working with a researcher on a much more fundamental question:

"What experiment would convince us that this capability is real?"

We expect you to challenge the premise when necessary.

If a result looks impressive but the control is weak, say so. If our favorite metric isn't measuring what we think it is, demonstrate it. If a benchmark rewards a shortcut, break it. If the evidence doesn't support a claim we want to make, tell us before anyone outside the company does.

Part of your job is obviously to make Dyssonance shine, but the biggest part of it is to enable the Dyssonance team to always improve the technology and push its boundaries.