Signal, Data & Evaluation Engineer: Benchmarks & Datasets

Matter Intelligence; San Francisco (CA); On-site; USD 140,000 - 210,000; Full time

Benefits offered by this job

Equity package

Health coverage

Real-world deployment paths

Job summary

Matter Intelligence is hiring a Signal, Data, and Evaluation Engineer to build datasets, benchmarks, environments, and evidence for defensible models and AI systems. You will work across data curation, simulation and learning environments, model and agent evaluation, and release-quality evidence.

This role reports to a designating leader and requires building versioned datasets, testing, and clear communication of evidence supporting releases in an onsite San Francisco environment.

Qualifications

• Experience in ML evaluation, dataset engineering, reinforcement-learning environments, simulation, scientific ML, test infrastructure, or production AI systems.
• Strong foundations in statistics, experimental design, benchmark validity, distribution shift, calibration, reward design, and evidence-based acceptance.
• Fluency in Python and modern machine-learning frameworks, with the software engineering discipline to build tested modules, data contracts, or services.
• Experience converting qualitative model or agent failures into reproducible datasets, tests, or environment scenarios.
• Ability to reason across the complete data path and identify how acquisition, transformation, indexing, orchestration, and presentation affect correctness.

Responsibilities

• Build versioned dataset workflows for collection, curation, filtering, labeling, ground truth, synthetic data, hard-negative mining, contamination controls, and train-evaluation isolation.
• Design benchmark suites across perception, multimodal reasoning, physics-informed prediction, retrieval, planning, tool use, world modeling, and customer tasks.
• Create simulation and environment infrastructure for reinforcement learning, imitation learning, offline learning, model-based learning, and agent training.
• Define repeatable evaluation methods across accuracy, calibration, generalization, robustness, safety, latency, cost, and operational outcomes.
• Turn field, mission, and customer failures into versioned datasets, benchmark cases, adversarial tests, and environment scenarios.
• Build automated promotion gates and communicate clearly what has been tested, what remains unproven, and what evidence supports release.

Skills

ML evaluation

Dataset engineering

RL environments

Simulation

Scientific ML

Test infrastructure

Production AI

Python

PyTorch

JAX

Tools

PyTorch

JAX

Job description

Matter Intelligence is hiring a Signal, Data, and Evaluation Engineer to build datasets, benchmarks, environments, and evidence for defensible models and AI systems. You will work across data curation, simulation and learning environments, model and agent evaluation, and release-quality evidence.

This role reports to a designating leader and requires building versioned datasets, testing, and clear communication of evidence supporting releases in an onsite San Francisco environment.