Member of Technical Staff - Evals Research Engineer

NeoSigma

San Francisco (CA)

On-site

USD 140,000 - 210,000

Full time

3 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Competitive salary plus meaningful equ
Comprehensive medical benefits and PTO
Direct impact on company direction
High ownership and career-defining

Job summary

NeoSigma is seeking a Backend Engineer to own ingestion, evaluation, and serving of agent behavior data at production scale. You’ll work across telemetry ingestion, platform reliability, sandbox infrastructure, and the API surface used by customers and integrations.

You will design evaluation frameworks, research reward modeling, and build infrastructure to ship features end-to-end, including open-source contributions and technical blogs.

Qualifications

  • Strong foundations in machine learning, deep learning, and computer science.
  • Hands-on experience with LLM evaluation, agentic systems, reinforcement learning, or post-training.
  • Strong programming skills in Python and experience building reliable ML research infrastructure.
  • Comfortable navigating open-ended research problems and translating ideas into working implementations.

Responsibilities

  • Design and build evaluation frameworks for agentic systems.
  • Research methods for agent evaluation, reward modeling, reward-hacking, post-training, and self-improving systems.
  • Develop rubrics, verifiers, reward signals, and benchmarks to measure performance and reliability.
  • Own research streams end-to-end and build infrastructure to ship working systems.
  • Publish research findings through technical blogs, papers, and open-source contributions.

Skills

Machine learning
Deep learning
Computer science
Python
Reinforcement learning

Tools

Python

Job description

Member of Technical Staff - Evals Research Engineer
The Role

We are looking for a Backend Engineer to own the systems that ingest, evaluate, and serve agent behavior data at production scale. You will work across telemetry ingestion, platform reliability, sandbox infrastructure, and the API surface that customers and integrations build on. Some weeks you will be deep in distributed systems and query performance. Other weeks you will ship a customer-facing feature across backend, APIs, and the data layer.

What you'll do:
  • Design and build evaluation frameworks for agentic systems
  • Research methods for agent evaluation, reward modeling, reward-hacking, post-training, and self-improving systems
  • Develop rubrics, verifiers, reward signals, and benchmarks to measure agent performance, reliability, and generalisation
  • Own research streams end-to-end and building infrastructure to ship working systems
  • Publish research findings through technical blogs, papers, and open-source contributions
  • Moving fast with high agency
  • Strong foundations in machine learning, deep learning, and computer science
  • Hands-on experience with LLM evaluation, agentic systems, reinforcement learning, or post-training
  • Strong programming skills in Python and experience building reliable ML research infrastructure
  • Strong research taste and the ability to identify and solve complex and meaningful problems
  • Comfortable navigating open-ended research problems and quickly translating ideas into working implementations
  • High standards for experimental rigor, engineering quality, and clear technical communication
Our Core Values:
  • Customer Obsession - We start with the customer and work backwards. We aim to earn trust through consistent delivery, thoughtful listening, and by obsessing over customers.
  • Intellectual Honesty - We operate with high trust and low ego. Ideas matter more than titles, and we communicate openly and directly while assuming good intent, even in strong disagreement.
  • Bias for Action - We set high standards and move quickly to meet them. We prefer building and learning with customers over debating in the abstract, and we iterate based on real feedback.
  • Extreme Ownership - We take responsibility for outcomes, not just tasks. Ownership means seeing problems through to completion and ensuring solutions truly work in practice.
Benefits and perks
  • Competitive salary plus meaningful equity package
  • Comprehensive medical benefits and generous PTO
  • Direct impact on company direction and technical decisions
  • High ownership and the opportunity to make a career-defining impact

As a founding member, you’ll help define the technical foundation of NeoSigma. Your scope will grow with the company, from owning core systems end-to-end to shaping architecture, hiring, and engineering culture. This role has a natural path toward technical leadership or engineering management as the team scales.

About Us

NeoSigma is a product-driven research lab building the intelligence layer that helps close the feedback loop between your customers, products, and AI systems.

We are a small, intensely technical team of researchers and engineers who have trained frontier-scale models and widely used AI products and agents at Parallel Web, Essential AI, MIT, Apple, and Amazon.

We are backed by world-class investors and leaders from Google DeepMind, OpenAI, Decagon and others.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Evals Research Engineer — Backend for Agentic AI Systems
Evals Research Engineer — Backend for Agentic AI Systems

NeoSigma • San Francisco (CA)

On-site
USD 140,000 - 210,000
Competitive salary plus meaningful equ
Comprehensive medical benefits and PTO
Direct impact on company direction
+1
Research Engineer - Evals
Research Engineer - Evals

AGI, Inc. • San Francisco (CA)

On-site
USD 120,000 - 160,000
Competitive cash and equity
Top-tier relocation support
In-person work environment
Member of Technical Staff [Agent]
Member of Technical Staff [Agent]

NeoCognition • Palo Alto (CA)

On-site
USD 120,000 - 140,000
Head of Evals
Head of Evals

Trajectory Labs • Berkeley (CA)

On-site
USD 250,000 - 400,000
Health coverage
401(k)
Catered meals
+1
Member of Technical Staff, Evals
Member of Technical Staff, Evals

We Love AI Jobs • San Francisco (CA)

Remote
USD 200,000 - 350,000
Member of Technical Staff, Evals
Member of Technical Staff, Evals

Handshake • Mountain View (CA)

On-site
USD 190,000 - 260,000
Health Insurance
401k match
Sabbatical
+4
Head of Evals, AI Red Teaming
Head of Evals, AI Red Teaming

Trajectory Labs • Berkeley (CA)

Remote
USD 200,000 - 400,000
Health coverage
401(k)
Food
+1
Member of Technical Staff (Evals & Post-Training)
Member of Technical Staff (Evals & Post-Training)

Ambral • San Francisco (CA)

On-site
USD 150,000 - 230,000
Equity ownership
Equinox membership
Free meals, coffee, snacks
+2
Member of Technical Staff, AI Evaluation
Member of Technical Staff, AI Evaluation

Harrison Clarke • San Francisco (CA)

On-site
USD 150,000 - 230,000
Meaningful equity
Competitive salary
Member of Technical Staff [Research]
Member of Technical Staff [Research]

NeoCognition • Palo Alto (CA)

On-site
USD 120,000 - 150,000