Senior Performance Instrumentation Engineer for ML Systems

Salesonaut; Santa Clara (CA); On-site; USD 168,000 - 270,000; Full time

Benefits offered by this job

Equity

Benefits

Job summary

NVIDIA is seeking a Software Engineer, Performance Instrumentation, to join our team in Santa Clara, CA. You will define data requirements, identify visibility gaps, and implement low-overhead instrumentation across ML and systems software.

Responsibilities include ensuring data quality, measuring overhead, and collaborating with partner teams to deliver reliable telemetry and insights. Strong Python/C++ and CUDA skills are essential.

Qualifications

• Bachelor's degree in Computer Science, Computer Engineering, or a related field, or equivalent experience.
• 8+ years of relevant software engineering experience, including hands-on instrumentation in production ML or systems software.
• Proficiency in CUDA, and working knowledge of ML frameworks like PyTorch or JAX, plus NVTX APIs.
• Experience maintaining data quality in a pipeline with validation, checks, and provenance.
• Ability to measure instrumentation overhead and trade-offs among overhead, fidelity, sampling, and aggregation.
• Experience implementing changes across teams by defining interfaces and delivering.
• Strong debugging across component boundaries with tests and automated tools.
• Excellent communication skills to convey evidence and next steps.

Responsibilities

• Define and maintain performance data requirements from profiling libraries, frameworks, runtimes, and telemetry.
• Identify visibility gaps and collaborate to address or build solutions.
• Inform build of low-overhead instrumentation and annotation APIs for consumers.
• Ensure data quality at ingestion with validation and alerting mechanisms.
• Measure overhead of instrumentation and manage observer effects on real workloads.
• Collaborate with multiple partner teams and manage commitments.

Skills

Python

C++

CUDA

PyTorch or JAX

Education

Bachelor's degree in CS/CE or related

Tools

NVTX

Profiling/Instrumentation

Job description

NVIDIA is seeking a Software Engineer, Performance Instrumentation, to join our team in Santa Clara, CA. You will define data requirements, identify visibility gaps, and implement low-overhead instrumentation across ML and systems software.

Responsibilities include ensuring data quality, measuring overhead, and collaborating with partner teams to deliver reliable telemetry and insights. Strong Python/C++ and CUDA skills are essential.