Senior Performance Instrumentation Engineer (CUDA/ML)

NVIDIA Corporation; Santa Clara (CA); On-site; USD 168,000 - 270,000; Full time

Benefits offered by this job

Equity

Benefits

Job summary

NVIDIA Corporation in Santa Clara, CA is seeking a Software Engineer, Performance Instrumentation to define and maintain performance data requirements across profiling libraries, runtimes, and telemetry. You will identify visibility gaps and collaborate with teams to address them while keeping overhead low.

Requirements include a Bachelor’s degree in CS/CE or equivalent, 8+ years in production software with instrumentation, proficiency in Python and C++, CUDA, NVTX, PyTorch or JAX, and strong

Qualifications

• Bachelor’s degree in Computer Science, Computer Engineering, or a related field (or equivalent experience).
• 8+ years of relevant software engineering experience, including instrumentation or tracing in production ML or systems software.
• Proficiency in CUDA, and working knowledge of PyTorch or JAX, plus NVTX APIs.
• Experience maintaining data quality in a pipeline with validation, checks, and provenance.
• Ability to measure instrumentation overhead and trade-offs between overhead, fidelity, sampling, and aggregation.
• Experience implementing changes across codebases managed by other groups with clear interfaces.
• Strong debugging skills across component boundaries and in code reviews, tests, and measurements.
• Excellent communication skills for conveying evidence and next steps; proficient in Python and C++.

Responsibilities

• Define and maintain the performance data requirements from profiling libraries, annotation APIs, frameworks, runtimes, and telemetry.
• Identify and prove visibility gaps with concrete evidence.
• Collaborate with teams to address gaps or build solutions.
• Inform the build of low-overhead instrumentation and annotation APIs from the consumer perspective.
• Ensure data quality at ingestion with validation, sanitization, and alerting.
• Measure and manage instrumentation overhead on real workloads.
• Collaborate with multiple partner teams simultaneously to ensure progress.

Skills

Instrumentation/Tracing experience

Python

C++

Communication skills

Education

Bachelor's degree in Computer Science, Computer Engineering, or related field

Tools

CUDA

NVTX

PyTorch

JAX

Job description

NVIDIA Corporation in Santa Clara, CA is seeking a Software Engineer, Performance Instrumentation to define and maintain performance data requirements across profiling libraries, runtimes, and telemetry. You will identify visibility gaps and collaborate with teams to address them while keeping overhead low.

Requirements include a Bachelor’s degree in CS/CE or equivalent, 8+ years in production software with instrumentation, proficiency in Python and C++, CUDA, NVTX, PyTorch or JAX, and strong