Senior AI System Engineer - LLM Training System

Huawei Canada

British Columbia

On-site

CAD 127,000 - 225,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Huawei Canada is hiring a Senior System Engineer to drive core compute operator development and scalable training across foundation models on Ascend NPUs. The role emphasizes optimizing distributed pipelines, high-performance kernels, and low-precision training while profiling and tuning for peak MFU efficiency.

You will work with MindSpeed, Megatron-LM, PyTorch FSDP, veRL, and TileLang on multi-node clusters, ensuring numerical parity between training and serving runtimes and contributing to

Qualifications

  • Bachelor’s, Master’s, or PhD in CS/CE/EE or related field
  • 3+ years of HPC, AI accelerator operator development, or distributed large-model training systems
  • Strong experience in AscendC / CANN, or CUDA C/C++ with ability to transition to AscendC
  • Experience with MindSpeed, Megatron-LM / Megatron-Core, PyTorch FSDP, veRL, or vLLM / DeepSpeed
  • Proficiency in C/C++ and Python, with PyTorch custom extensions
  • Performance profiling and debugging on AI hardware (Ascend Profiler, PyTorch Profiler)

Responsibilities

  • Drive core compute operator development across foundation model pre-training and post-training workloads
  • Identify bottlenecks and implement high-performance kernels using AscendC, Triton, and TileLang
  • Optimize distributed acceleration suites (MindSpeed / Megatron frameworks)
  • Tune multi-dimensional hybrid parallelism and achieve MFU/compute-communication overlap
  • Profile performance on Ascend hardware using msprof, msprobe, and PyTorch Profiler to reduce stalls
  • Contribute high-performance operators and accelerations to open-source Ascend communities

Skills

HPC experience
CUDA/C++
PyTorch experience
Triton/TileLang
Distributed training frameworks
Python (expert)
PyTorch extensions

Education

Bachelor/Master/PhD in CS/CE/EE or related field

Tools

AscendC
CANN
Megatron-LM / Megatron-Core
PyTorch custom C++/CUDA extensions

Job description

Huawei Canada has an immediate permanent opening for a Senior Engineer.

About the team:

The Computing Data Application Acceleration Lab aims to create a leading global data analytics platform organized into three specialized teams using innovative programming technologies. This team focuses on full-stack innovations, including software-hardware co-design and optimizing data efficiency at both the storage and runtime layers. This team also develops next-generation GPU architecture for gaming, cloud rendering, VR/AR, and Metaverse applications. One of the goals of this lab are to enhance algorithm performance and training efficiency across industries, fostering long-term competitiveness.

About the job:

As a Senior System Engineer, you will drive core compute operator development and distributed scaling performance across foundation model pre-training and post-training workloads on Ascend NPUs. You will identify computational and communication bottlenecks, implement high-performance kernels using AscendC, Triton, and TileLang, and optimize distributed acceleration suites (MindSpeed / open-source training frameworks).

  • Ascend High-Performance Kernel Development: Design, implement, and tune core operators (FlashAttention variants, FP8/MXFP8/W4A8 GEMM, Fused RMSNorm, dynamic MoE routing/dispatch) using AscendC, with explorations in Triton (Triton-Ascend backend) and TileLang.
  • Pre-training & Post-training System Optimization: Architect, scale, and optimize distributed pre-training and RL post-training pipelines using MindSpeed (MindSpeed-LLM / MindSpeed-MM) alongside upstream frameworks (Megatron-LM / Megatron-Core, PyTorch FSDP, veRL, vLLM, Ray).
  • Distributed Parallelism & Training Efficiency: Tune multi-dimensional hybrid parallelism (TP, PP, CP, EP, ZeRO/FSDP) and RL rollout/actor pipelining on Ascend NPU clusters, achieving optimal MFU and compute-communication overlap.
  • Low-Precision & Numerical Parity: Implement low-precision training/inference schemes (FP8, MXFP8, mixed-precision), guaranteeing numerical stability, batch-invariance, and strict precision parity between training and serving runtimes.
  • Profiling & Hardware-Aware Tuning: Conduct deep performance profiling on Ascend hardware using Ascend Profiler (msprof), msprobe, and PyTorch Profiler to eliminate pipeline bubbles, AICore/AIVector stalls, and HCCL communication overhead.
  • Open Source & Ecosystem Enablement: Contribute high-performance operators, bug fixes, and acceleration features to open-source Ascend communities and upstream AI acceleration projects.

The total target annual compensation ranges from $127,000 to $225,000 depending on education, experience, and demonstrated expertise.

Job requirements
About the ideal candidate:
  • Bachelor’s, Master’s, or PhD in Computer Science, Computer Engineering, Electrical Engineering, or related technical disciplines, 3+ years of experience in high-performance computing (HPC), AI accelerator operator development, or distributed large model training systems.
  • Hands-on Operator Programming: Solid development experience in AscendC / CANN, or extensive experience in CUDA C/C++ with a proven ability to transition to AscendC. Experience or strong interest in Triton or TileLang for accelerator backends.
  • Distributed Frameworks: Deep practical experience with distributed pre-training and/or post-training frameworks, such as MindSpeed, Megatron-LM / Megatron-Core, PyTorch FSDP, veRL, or vLLM / DeepSpeed. Systems Programming: Strong programming proficiency in C/C++ and modern Python, with hands-on experience building PyTorch custom C++/CANN extensions.
  • Performance Analysis: Proficiency in performance profiling and debugging tools on AI hardware (e.g., Ascend Profiler/msprof, PyTorch Profiler, or Nsight toolchains).
Preferred / Standout Qualifications:
  • Direct experience porting or optimizing Triton or TileLang frontends/backends for non-CUDA accelerators (especially Ascend NPU).
  • Deep understanding of Ascend NPU microarchitecture (Cube & Vector compute units, local memory hierarchy/UB, memory movement engines, and HCCL collective communication). Hands-on experience optimizing end-to-end MFU for massive MoE or dense foundation models on multi-node NPU/GPU clusters.
  • Experience in low-precision numerical alignment (FP8, MXFP8, W4A8/W8A8) and eliminating training-inference discrepancy. Active open-source contributions to Ascend repos (MindSpeed, CANN ops, Ascend-vLLM) or upstream projects (Megatron-LM, Triton, TileLang, veRL).
Additional Information:

Huawei Canada is committed to a fair, inclusive, and accessible recruitment process. If you require accommodation during any stage of the hiring process, please let us know and we will work with you to meet your needs.

All applications for this position are reviewed directly by our hiring team, we do not use artificial intelligence tools to screen or select candidates.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI System Engineer - LLM Training System
Senior AI System Engineer - LLM Training System

Huawei Technologies Co. • Vancouver

On-site
CAD 127,000 - 225,000
Researcher - AI Computing System
Researcher - AI Computing System

Huawei Canada • Vancouver

On-site
CAD 106,238 - 204,304
Senior Technical VP - AI & Efficient Deep Learning
Senior Technical VP - AI & Efficient Deep Learning

Huawei Canada • Markham

On-site
CAD 150,000 - 200,000
Intern Researcher – AI Foundation Model Training
Intern Researcher – AI Foundation Model Training

Huawei Canada • Markham

On-site
CAD 58,000 - 104,000
Senior Researcher – GPU, AI & Hardware Architecture
Senior Researcher – GPU, AI & Hardware Architecture

Huawei Canada • Edmonton

On-site
CAD 120,000 - 170,000
Research Engineer - AI Workload & Systems
Research Engineer - AI Workload & Systems

Huawei Canada • Markham

On-site
CAD 178,070 - 315,479
Senior Engineer-Cloud AI Infrastructure
Senior Engineer-Cloud AI Infrastructure

Huawei Canada • Markham

On-site
CAD 127,000 - 225,000
AI Research Engineer - Computer Vision & Neural Rendering
AI Research Engineer - Computer Vision & Neural Rendering

Huawei Technologies Canada Co., Ltd. • Edmonton

On-site
CAD 90,000 - 130,000
Intern Researcher - AI Computing System
Intern Researcher - AI Computing System

Huawei Canada • Vancouver

On-site
CAD 78,000 - 150,000
Principal Scientist - Software/Hardware Co-design
Principal Scientist - Software/Hardware Co-design

Huawei Canada • Markham

On-site
CAD 120,000 - 160,000