Production Systems Engineer

Meta; Menlo Park (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Meta in Menlo Park, CA, seeks a Production Systems Engineer to own end-to-end hardware validation and deployment for data center servers. You will work with HW/SW co-design teams, hardware designers, and vendors to enable new systems for production data centers.

The role emphasizes diagnostic tooling, robust test plans, and driving improvements in error handling, telemetry, and health signals across compute and storage platforms.

Qualifications

• Bachelor's degree in Computer Science, Computer Engineering, a relevant technical field, or equivalent practical experience.
• Degree must be completed prior to joining Meta.
• 3+ years of work experience in ASIC, compute, AI/ML hardware, storage, memory, or related domains.

Responsibilities

• Support end-to-end system validation and deployment for Compute and Storage hardware platforms in data center environments.
• Proactively create experiments and tooling to diagnose hardware, firmware, and software failures using telemetry and logs.
• Design robust system-level test plans including functional, stress, and performance tests.
• Drive vendors and internal teams to resolve hardware and firmware issues identified during production operations.
• Develop Hardware Fault Management and improve error reporting/handling for better operation quality.

Skills

Python

C/C++

Linux

System-level debugging

Troubleshooting

Education

Bachelor's degree in CS/CE or related field

Tools

JTAG

GDB

Trace32

DSTREAM

CI/CD

Job description

About

Meta is seeking a highly skilled and experienced Systems/Hardware engineer to join our Release to Production (RTP) team. The RTP team is responsible for the end-to-end hardware lifecycle of all Meta servers, including prototyping, pre-production hands-on system validation, hardware debugging, and stress testing. As a Production Systems Engineer, you will work closely with various teams, including HW/SW co-design teams, hardware designers, networking teams, system manufacturers, component vendors, capacity engineering, production engineering, production services, and data center operations teams to enable new systems that will be deployed in our production data centers.

Responsibilities

• Support end-to-end system validation, and deployment for cutting-edge Compute and Storage hardware platforms, including CPUs, flash, and data center processing units (DPUs) in data center environments
• Proactively create experiments and tooling to diagnose and root cause hardware, firmware, and software failures across compute, storage, and networking systems using telemetry, logs, and hands-on debugging
• Design and implement robust system-level test plans, including functional, stress, and performance tests
• Drive component vendors and system manufacturers to resolve hardware and firmware issues identified during production operations
• Interface with external vendors and internal teams to understand the system architecture and develop Hardware Fault Management for various server products
• Drive new platform enablement, hardware validation, tooling specification, and integration, customer workload testing, and experiment creation to detect and diagnose hardware/firmware/software health issues
• Leverage understanding of RAS (reliability, availability, serviceability) to improve error reporting and error handling mechanisms for better operation quality and cost/efficiency
• Analyze production failure data to identify systemic trends and drive corrective actions with internal teams and external vendors
• Create and maintain dashboards and data visualizations to surface hardware health signals and track remediation progress across the fleet
• Drive necessary discussions with external and internal teams on test specifications and methodologies to improve test quality continuously, both within Meta, and in the broader industry

Minimum Qualifications

• Currently has, or is in the process of obtaining a Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience. Degree must be completed prior to joining Meta
• Bachelor's degree in Computer Science, Computer Engineering, a relevant technical field, or equivalent practical experience
• 3+ years of work experience in one or more domains such as: ASIC development, compute, AI-ML hardware/software, storage, memory, network, server interconnect technologies, or similar
• Knowledge of architecture and components on one of the following products: server, PC, or laptop
• Development or debug experience in one or more areas: hardware fault management, error reporting, error handling on hardware products
• Experience with Python, C/C++ and/or similar languages, within a Linux environment, for server system management, automation, version control, CI/CD, or similar
• Excellent problem-solving skills, with the ability to troubleshoot complex technical issues
• Experience communicating technical findings and coordinating with cross-functional teams such as hardware design, firmware, and vendor engineering to drive issue resolution
• Experience working in a matrix organization 5+ years of experience with a subset of one of the following domains: Compute Systems, Storage Systems, Accelerated Compute Systems/HPC, Kernel/Firmware Development and/or testing, Post Silicon Bringup
• Experience with x86 or ARM-based CPUs and their subsystems (e.g., memory, inter-chiplet communications, RAS/DFT, power management), SSDs, or DPUs/SmartNIC
• 3+ years of experience supporting ASIC development (silicon bringup, characterization)
• Familiarity with debugging tools for systems-on-chip (SoCs) — e.g. JTAG, GDB, DSTREAM, Trace32
• 2+ years of experience scripting automation in Python or equivalent
• Hands-on experience in troubleshooting problems at the system level, crossing across multiple components, as well as hardware/firmware/software boundaries. Hands-on experience in managing/debugging Linux servers
• Understanding of the hardware development process and how it pertains to test strategy. Experience authoring test plans for hardware systems or components
• Working/functional knowledge of common bus protocols such as I2C, SPI, USB, LPDDR, and/or PCIe