Performance Instrumentation Engineer — ML Telemetry

NVIDIA Corporation; United States; On-site; USD 168,000 - 270,000; Full time

Benefits offered by this job

Equity

Benefits

Job summary

NVIDIA is seeking a Software Engineer, Performance Instrumentation, to join our team in Santa Clara, CA. You will define data requirements, identify visibility gaps, and ensure quality of instrumentation across production ML and systems software.

The role emphasizes cross-team collaboration, low-overhead design, and rigorous validation. Applicants should have a bachelor’s degree in CS/CE (or related), 8+ years of experience, and strong Python/C++ skills along with CUDA and ML framework

Qualifications

• Bachelor’s degree or equivalent experience in CS/CE or related field.
• 8+ years of software engineering experience with instrumentation or tracing in ML or systems software.
• Proficiency in CUDA and knowledge of PyTorch or JAX; NVTX annotation APIs.
• Experience maintaining data quality in pipelines with validation and provenance.
• Ability to measure instrumentation overhead and trade-offs.
• Ability to implement cross-team changes from requirements to delivery.
• Strong debugging and validation across components; experience with automated tools.
• Excellent communication with engineers and data users; proficient in Python and C++.

Responsibilities

• Define and maintain performance data requirements from profiling libraries, frameworks, and telemetry.
• Identify visibility gaps and collaborate to address them or build solutions.
• Inform build of low-overhead instrumentation and annotation APIs for consumers.
• Ensure data quality at ingestion with validation, sanitization, checks, and alerts.
• Measure overhead and observer effects on real workloads; keep within limits.
• Collaborate with multiple partner teams; manage commitments and progress.

Skills

Instrumentation

Tracing

Python

C++

CUDA

PyTorch

JAX

Debugging

Communication

Education

Bachelor’s degree in Computer Science/Computer Engineering or related field

Tools

NVTX

NCCL

Job description

NVIDIA is seeking a Software Engineer, Performance Instrumentation, to join our team in Santa Clara, CA. You will define data requirements, identify visibility gaps, and ensure quality of instrumentation across production ML and systems software.

The role emphasizes cross-team collaboration, low-overhead design, and rigorous validation. Applicants should have a bachelor’s degree in CS/CE (or related), 8+ years of experience, and strong Python/C++ skills along with CUDA and ML framework