Senior AI Evaluation Engineer for Frontier Models

NVIDIA; Warszawa; On-site; PLN 390,000 - 650,000; Full time

Job summary

NVIDIA is seeking senior engineers to pioneer evaluation methodologies for state-of-the-art AI models, including LLMs, vision, and agentic systems. You will build and scale experiments on enterprise-grade GPU clusters to influence model releases and roadmap decisions.

Collaborating with model research, training, and product teams, you will design evaluation environments, pipelines, and metrics, translate signals into go/no-go decisions, and publish findings that advance benchmarking standards

Qualifications

• 6+ years of hands-on experience with LLMs, designing and running evaluations for large language models or multimodal AI systems.
• Strong statistical foundations: experimental design, significance testing, regression analysis, and the ability to distinguish signal from noise in benchmark results at scale.
• Proven experience building evaluation infrastructure (pipelines, benchmark harnesses, reproducible CI systems) not just consuming existing benchmarks.
• Clear, precise communicator who can translate quantitative evaluation results into decisions for researchers, product teams, and senior leadership.

Responsibilities

• Design and build decision-grade evaluation environments for frontier models spanning reasoning, multimodal, long-context, and agentic systems.
• Research and develop novel evaluation methodologies for emerging model families and capability domains where benchmarks don't exist or generalize.
• Build and operate evaluation infrastructure and pipelines including benchmark environments, regression CI systems, and statistical analysis tooling.
• Partner with model research, training, and customer teams to translate evaluation signals into concrete decisions: release go/no-go, training direction, and competitive positioning.

Skills

LLMs experience

Evaluation design

Statistical analysis

CI pipelines

Communication

Education

BS/MS/PhD in CS/ML/Statistics

Tools

Benchmark harnesses

Reproducible CI systems

MLflow/W&B

Job description

NVIDIA is seeking senior engineers to pioneer evaluation methodologies for state-of-the-art AI models, including LLMs, vision, and agentic systems. You will build and scale experiments on enterprise-grade GPU clusters to influence model releases and roadmap decisions.

Collaborating with model research, training, and product teams, you will design evaluation environments, pipelines, and metrics, translate signals into go/no-go decisions, and publish findings that advance benchmarking standards