Research Engineer, AI Evaluations & Benchmarks

gNucleus AI; Sunnyvale (CA); On-site; USD 140,000 - 190,000; Full time

Job summary

gNucleus AI is seeking an AI Engineer to own our bench project and lead domain-specific AI evaluation harnesses for physical/spatial AI and 3D parametric CAD models and agents. You will create domain-specific evaluation tasks and benchmark datasets to build our comprehensive evaluation suite.

You will be the primary owner for our evaluation benchmarks, collaborating with Product, Research, and Safety teams to translate vague capability goals into automated benchmark pipelines and releases.

Qualifications

• PhD or Master’s in CS/ML or related field.
• 1–3 years hands-on with RLHF, preference modeling, or instruction tuning.
• Experience with LLM evaluation or benchmark development.
• Ability to create quantitative evaluation rubrics and reports.
• Strong communication and cross-functional collaboration.

Responsibilities

• Design domain-specific AI evaluation tasks and datasets.
• Build reproducible benchmark pipelines and releases.
• Develop robust evaluation harnesses and automated grading suites.
• Analyze model failure modes and translate insights into data requirements.
• Communicate results via benchmark reports and papers.

Skills

Deep learning

Reinforcement learning

LLM evaluation

Benchmark development

Data analysis

Education

PhD or Master’s in CS/ML

Tools

PyTorch

TensorFlow

Python

Job description

We are seeking an AI Engineer to own our bench project and lead domain-specific AI evaluation harnesses for physical/spatial AI and 3D parametric CAD models and agents. In this role, you will focus on creating domain-specific evaluation tasks and benchmark datasets to construct our comprehensive evaluation suite.

You will act as the primary owner for our evaluation benchmarks, working closely with Product, Research, and Safety teams to turn vague capability goals into actionable, automated benchmark pipelines and releases.

About the Role

We are seeking an AI Engineer to own our bench project and lead domain-specific AI evaluation harnesses for physical/spatial AI and 3D parametric CAD models and agents. In this role, you will focus on creating domain-specific evaluation tasks and benchmark datasets to construct our comprehensive evaluation suite.

You will act as the primary owner for our evaluation benchmarks, working closely with Product, Research, and Safety teams to turn vague capability goals into actionable, automated benchmark pipelines and releases.

Key Responsibilities

• Conduct ongoing research on AI models and perform analysis to track industry progress and capabilities.


• Design and curate domain-specific AI evaluation tasks, dataset suites, and standardized testing scenarios.


• Build reproducible evaluation systems, including harnesses, graders, and benchmark infrastructure.(e.g., containers, repos, test harnesses, sandboxes, code execution).


• Develop robust evaluation harnesses and automated grading suites.


• Analyze model failure modes, categorize root causes, and translate insights into prioritized data requirements.


• Establish release cycles, manage project timelines, resource allocation, deliverables for evaluation cycles and create benchmark reports.


• Communicate results through benchmark reports, technical articles, and research papers.

Requirements

• Ph.D. or Master's degree in Computer Science, Machine Learning AI, STEM, or a related field.


• 1-3 years of hands‑on experience with post-training techniques such as RLHF, preference modeling, or instruction tuning, and with LLM evaluation or benchmark development.


• Proven technical expertise in data analysis tools; Deep understanding of deep learning, reinforcement learning, and large-scale model fine-tuning.


• Exceptional ability with analytical mindset to break complex model outputs into structured, quantitative evaluation rubrics and error taxonomies, distill evaluation metrics and dataset coverage into clear reports and release readiness assessments.


• Strong problem‑solving abilities, adaptability in fast‑paced environments, collaborative teamwork across cross‑functional departments, and proactive project ownership. High "taste" for good engineering: you care about correctness, code quality, and how real engineering teams actually work.


• Excellent written and verbal communication skills, with the ability to speak credibly with senior engineers and researchers.

Preferred Qualifications

• Experience or familiarity with at least one CAD software (e.g., SolidWorks, CATIA, NX, Creo, Onshape, FreeCAD)


• Publish research findings in top-tier AI conferences.


• Work samples or code (e.g., eval frameworks, benchmark suites, failure‑analysis reports or tooling) that demonstrate relevant skills.