Principal AI Infrastructure & Accelerator Architect

EPAM Systems; New York (NY); On-site; USD 180,000 - 220,000; Full time

Benefits offered by this job

Medical Insurance

401(k) Retirement Savings Plan

Paid Time Off

Stock Purchase Program

Employee Discounts

LinkedIn learning access

Job summary

EPAM Systems in New York City seeks a Principal Architect to drive AI performance, optimization, and hardware–software co-design across training and serving infrastructures.

You will define the architectural vision, lead the Center of Excellence, and guide roadmaps spanning TPU/GPU fleets, ML models, and compiler toolchains, delivering scalable impact across enterprise and research partners.

Qualifications

• 15+ years of software engineering experience, with HPC or AI infrastructure focus.
• 8+ years in distributed systems and AI infrastructure.
• 5+ years leading architecture and delivery of large-scale software.
• Proven track record architecting performance-critical systems bridging hardware accelerators and software frameworks.
• Strong leadership and collaboration across research, framework, and compiler teams.

Responsibilities

• Define multi-year roadmap for high-performance AI kernels, ops, and hardware–software co-design targeting TPU/GPU.
• Mentor and scale a technical practice with governance and engineering standards.
• Collaborate with ML researchers, framework architects (JAX, PyTorch), and compilers teams (XLA, MLIR) to remove bottlenecks.
• Architect foundational infrastructure including benchmarking suites, autotuning frameworks, and docs.
• Track hardware/ model trends to unlock efficiency gains in training and inference.

Job description

Are you a visionary technical leader passionate about defining the future of AI infrastructure and squeezing every drop of performance out of advanced hardware accelerators at scale? We are seeking a Principal Architect to lead, shape, and execute our technical strategy for AI performance, optimization, and hardware-software co-design. In this elite, highly visible role, you will define the architectural vision for both AI training and serving infrastructure, delivering massive industry-wide impact. You will spearhead our Center of Excellence (CoE), scaling our practice and guiding the technical roadmap across next-generation Tensor Processing Units (TPUs), Graphics Processing Unit (GPU) fleets, state-of-the-art ML models, and advanced compiler toolchains. Your architectural decisions will directly enable cutting-edge AI research and large-scale production deployments across Google Cloud, major enterprise customers, and the broader open-source ecosystem. If you thrive on solving intractable performance bottlenecks and redefining what is physically and computationally possible in AI infrastructure, this is your platform.

Req# 1067369299

Responsibilities

• Define and drive the multi-year technical roadmap for high-performance AI kernels, custom operations, and hardware-software co-design targeting TPU and GPU architectures
• Scale and mentor a world-class technical practice, establishing architectural governance, engineering standards, and best practices across the organization
• Act as the principal technical liaison partnering with ML researchers, core framework architects (JAX, PyTorch), and compiler engineering teams (XLA, MLIR) to eliminate systemic bottlenecks and shape future hardware/software requirements
• Architect foundational infrastructure—including enterprise-grade benchmarking suites, automated autotuning frameworks, regression analysis pipelines, and comprehensive documentation—empowering the global developer community
• Anticipate industry shifts by tracking advancements in hardware architectures, emerging model topologies, and compiler innovations to unlock step-changes in AI training and inference efficiency

Requirements

• Bachelor's degree in Computer Science, Electrical Engineering, or equivalent practical experience (Master's or Ph.D. preferred)
• 15+ years of software engineering experience, with 8+ years focused on distributed systems, AI infrastructure, or high-performance computing (HPC) architecture
• 7+ years of experience designing and developing complex software systems in C++ or Python
• 5+ years of experience leading the architecture, design, and delivery of large-scale software products, frameworks, or developer ecosystems from inception to production
• Proven track record of architecting performance-critical systems at the kernel level, bridging hardware accelerators and high-level software frameworks

Nice to have

• Deep expertise in optimizing TPU/GPU execution, leveraging low-level kernel languages/abstractions such as Pallas, Mosaic, Triton, or CUDA
• Comprehensive knowledge of modern ML frameworks (JAX, PyTorch), attention mechanisms, Mixture of Experts (MoEs), model quantization, and low-precision arithmetic
• Advanced understanding of modern accelerator architectures, including heterogeneous compute, complex memory hierarchies, data movement optimization, and multi-node scale-out fabrics
• Deep familiarity with compiler principles, code generation, and modern toolchains such as MLIR, OpenXLA, and LLVM
• Demonstrated leadership in building and scaling developer infrastructure, widely adopted Open-Source Software (OSS) libraries, and extensible high-performance APIs
• Exceptional strategic communication and stakeholder management skills, with a history of influencing cross-functional engineering teams, researchers, and executive leadership

We offer

• Medical, Dental and Vision Insurance (Subsidized)
• Health Savings Account
• Flexible Spending Accounts (Healthcare, Dependent Care, Commuter)
• Short-Term and Long-Term Disability (Company Provided)
• Life and AD&D Insurance (Company Provided)
• Employee Assistance Program
• Unlimited access to LinkedIn learning solutions
• Matched 401(k) Retirement Savings Plan
• Paid Time Off – the employee will be eligible to accrue 15-25 paid days, depending on specific level and tenure with EPAM (accrual eligibility may change over time)
• Paid Holidays - nine (9) total per year
• Legal Plan and Identity Theft Protection
• Accident Insurance
• Employee Discounts
• Pet Insurance
• Employee Stock Purchase Program
• If otherwise eligible, participation in the discretionary annual bonus program
• If otherwise eligible and hired into a qualifying level, participation in the discretionary Long-Term Incentive (LTI) Program

For remote work in New York City only.

EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our clients, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that deliver the most creative and cutting-edge solutions, and have an opportunity to continuously learn and grow. No matter where you are located, you will join a dedicated, creative, and diverse community that will help you discover your fullest potential.

Engineer the Future with a Career at EPAM

This posting includes a good faith range of the salary EPAM would reasonably expect to pay the selected candidate. The range provided reflects base salary only. Individual compensation offers within the range are based on a variety of factors, including, but not limited to: geographic location, experience, credentials, education, training; the demand for the role; and overall business and labor market considerations. Most candidates are hired at a salary within the range disclosed. Salary range: $180,000 - $220,000. In addition, the details highlighted in this job posting above are a general description of all other expected benefits and compensation for the position.

Applications will be accepted on a rolling basis.

EPAM will not provide new H-1B visa sponsorship for this position. Candidates with existing transferable H-1B status may be considered.

It is unlawful in Massachusetts to require or administer a lie detector test as a condition of employment or continued employment. An employer who violates this law shall be subject to criminal penalties and civil liability.