Sr. Staff Performance Tuning Engineer - CPU PMU & Virtualization

NIO; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Medical plans

Dental & vision

401(k) with company match

Paid holidays

Job summary

NIO is seeking a Senior Performance Engineer in San Francisco to lead advanced system performance analysis and optimization across CPUs, memory hierarchies, and I/O subsystems. You will leverage PMU counters and Linux performance tools to drive throughput, latency, and determinism for both native and virtualization workloads.

You will profile cross-component interactions, optimize vCPU scheduling, huge pages, and IRQ affinity, and correlate hardware metrics with kernel traces and source code

Qualifications

• 8+ years in system performance engineering or related roles.
• Deep understanding of CPU pipelines, memory hierarchy, and interrupts.
• Proficiency with Linux performance tooling and scripting.

Responsibilities

• Identify and analyze performance bottlenecks across CPU, memory, cache, and I/O subsystems.
• Profile, measure, and optimize IPC, CPI, MPKI, and memory bandwidth.
• Tune virtualization overhead, vCPU placement, huge pages, and IRQ affinity for workloads.
• Correlate PMU metrics with kernel traces and source code paths.

Skills

Linux performance

C/C++

Python

Performance tooling

Education

Bachelor's, Master's, or Ph.D. in CS/EE

Tools

perf

eBPF/BCC/bpftrace

ftrace

VM tooling

Job description

NIO Inc. is a pioneer and a leading company in the global smart electric vehicle market. Founded in November 2014, NIO aspires to shape a sustainable and brighter future with the mission of "Blue Sky Coming". NIO envisions itself as a user enterprise where innovative technology meets experience excellence.

NIO designs, develops, manufactures and sells smart electric vehicles, driving innovations in next-generation core technologies. NIO distinguishes itself through continuous technological breakthroughs and innovations, exceptional products and services, and a community for shared growth.

NIO provides premium smart electric vehicles under the NIO brand, premium smart electric vehicles for families through the ONVO brand, and small smart high-end electric cars with the FIREFLY brand.

Responsibilities

Performance Analysis and Optimization

• Identify and analyze performance bottlenecks across CPU, memory hierarchy, cache, interconnect, and interrupt subsystems.
• Leverage hardware PMU counters to measure and interpret cycles, instructions, cache misses, branch mispredictions, TLB misses, and stall cycles.
• Compute and analyze performance metrics such as IPC, CPI, MPKI, memory bandwidth utilization, and stall breakdowns.
• Optimize system software, applications, and drivers for throughput, latency, and performance determinism.

Virtualization Performance Tuning

• Profile and reduce hypervisor overhead, including VM exits, interrupt injection, and stage-2 MMU translation costs.
• Optimize vCPU placement strategies, huge page usage, and IRQ affinity for performance-sensitive workloads.
• Analyze and tune virtio and IOMMU performance paths.
• Investigate cross-core and cross-cluster latency issues, including IPI overhead and interrupt routing on ARM-based systems.

CPU PMU and Microarchitectural Analysis

• Configure and interpret ARM PMUv3 and related vendor-specific performance monitoring extensions.
• Apply Top-Down Microarchitecture Analysis (TMA) to identify frontend/backend bottlenecks.
• Correlate PMU-derived metrics with kernel traces, source code paths, and system behavior.
• Design and implement custom event sets tailored to workload-specific performance characterization.

Required Qualifications

• Bachelor's, Master's, or Ph.D. in Computer Science, Electrical Engineering, or a related field.
• 8+ years of experience in system performance engineering or related roles.
• Strong understanding of computer architecture, including:
• CPU pipelines, execution engines, and memory hierarchy
• Interrupt architectures (e.g., GICv3/GICv4, APIC concepts)
• MMU, virtual memory, and page table structures
• Deep expertise with Linux performance tooling:
• perf
• eBPF / BCC / bpftrace
• ftrace / trace-cmd
• Proficiency in C/C++, Python, and shell scripting for performance tooling and automation.

Preferred Qualifications

• Hands-on experience with ARM PMUv3 on Cortex-A55/A76/A78 or Neoverse platforms.
• Experience with one or more virtualization stacks:
• KVM/QEMU
• Xen
• seL4 or other RTOS/hypervisor environments

Along with competitive pay, as a full-time NIO employee, you are eligible for the following benefits on the first day you join NIO:

• Anthem Blue Cross, HSA, and Kaiser HMO medical plans with $0 for Employee Only Coverage.
• Dental (including orthodontic coverage) and vision plan. Both provide options with a $0 paycheck contribution covering you and your eligible dependents.
• Company Paid HSA (Health Savings Account) Contribution when enrolled in the High Deductible Anthem Blue Cross medical plan
• Healthcare and Dependent Care Flexible Spending Accounts (FSA)
• 401(k) with Brokerage Link option
• Company paid Basic Life, AD&D, short-term and long-term disability insurance
• Employee Assistance Program
• Sick and Vacation time
• 13 Paid Holidays a year
• Paid Parental Leave for first 8 weeks at full pay (eligible after 90 days of employment with NIO)
• Paid Disability Leave for first 6 weeks at full pay (eligible after 90 days of employment with NIO)
• Voluntary benefits including: Voluntary Life and AD&D options for you, your spouse/domestic partner and dependent child(ren), pet insurance