Member of Technical Staff, ML Compilers & Code Generation

General Diffusion, Inc.; San Francisco, Northern (CA, KY); Hybrid; USD 180,000 - 260,000; Full time

Job summary

General Diffusion, Inc. is seeking a Member of Technical Staff to translate workloads into efficient executables across diverse accelerators. You will shape IR contracts, implement compiler passes, and verify numerical correctness with automated tests.

Ideal candidates bring MLIR/LLVM, C++, Python, or Rust expertise and a track record in performance tuning, cross-target validation, and collaboration with runtime and safety teams.

Qualifications

• Experience shipping compiler passes, lowering pipelines, code generation, or accelerator backends—not solely using compiler frameworks.
• Strong command of MLIR, LLVM, XLA/StableHLO, Triton, TVM, Mojo, or comparable compiler stacks, including IR design and dialect/lowering mechanics.
• Proven performance-engineering practice: profiling and benchmarks to distinguish graph, memory, runtime, and generated-code effects.
• Ability to reason rigorously about compiler correctness, numerical behavior, shape/layout constraints, and target-specific legality across multiple hardware classes.
• Production-quality systems engineering in C++, Rust, Python, or similar, with disciplined debugging, tests, and reproducible performance investigations.

Responsibilities

• Design and evolve target-independent IR contracts, legality rules, and staged lowering boundaries that preserve workload semantics while allowing backend specialization.
• Implement and maintain compiler passes, analyses, and target backends that lower ML workloads into executable programs for heterogeneous accelerators.
• Build generated-code conformance, semantic-equivalence, and numerical-correctness tests against defined references; make unsupported cases and assumptions explicit.
• Profile compiled workloads to identify graph, memory, scheduling, and code-generation bottlenecks, then turn findings into reproducible optimizations and benchmark regressions.
• Define machine-readable backend capability, compilation-cost, and execution-characteristic reports for Runtime & Placement.
• Collaborate with Kernels on compiler-to-kernel interfaces and with Safety & Formal Verification on evidence and hooks for enforcement.

Skills

MLIR

LLVM

XLA/StableHLO

Triton

TVM

C++

Python

Rust

Job description

Member of Technical Staff, ML Compilers & Code Generation

Translate workloads into dependable executable programs across unlike architectures.

Status Open

Area Compilers

Build the compiler path that turns a workload into a correct, target-aware executable across the unlike silicon represented in General Diffusion’s multi-architecture testbed. You will make lowering boundaries, backend behavior, and generated-code evidence legible to the runtime and placement teams, while keeping compiler-generated programs within the correctness checks required before execution. This role owns compilation and code generation—not fleet placement policy, hand-authored kernel hot paths, or the independent safety decision to permit an action.

01 / The work

What you’ll work on

• Design and evolve target-independent IR contracts, legality rules, and staged lowering boundaries that preserve workload semantics while allowing backend specialization.
• Implement and maintain compiler passes, analyses, and target backends that lower supported ML workloads into executable programs for heterogeneous external accelerators.
• Build generated-code conformance, semantic-equivalence, and numerical-correctness tests against defined references; make unsupported cases and assumptions explicit.
• Profile compiled workloads to identify graph, memory, scheduling, and code-generation bottlenecks, then turn findings into reproducible optimizations and benchmark regressions.
• Define machine-readable backend capability, compilation-cost, and execution-characteristic reports that Runtime & Placement can consume without owning their placement decisions.
• Work with Kernels on clear compiler-to-kernel interfaces and fallback paths, and with Safety & Formal Verification on evidence and hooks needed for independent enforcement.

02 / The background

What you bring

• Experience shipping compiler passes, lowering pipelines, code generation, or accelerator backends—not solely using compiler frameworks.
• Strong command of MLIR, LLVM, XLA/StableHLO, Triton, TVM, Mojo, or comparable compiler stacks, including IR design and dialect/lowering mechanics.
• Proven performance-engineering practice: use profiling and controlled benchmarks to distinguish graph, memory, runtime, and generated-code effects.
• Ability to reason rigorously about compiler correctness, numerical behavior, shape/layout constraints, and target-specific legality across more than one hardware class.
• Production-quality systems engineering in C++, Rust, Python, or a comparable language, with disciplined debugging, tests, and reproducible performance investigations.

03 / The evidence

What progress looks like

• A documented compilation path for an agreed representative workload set has explicit IR/lowering contracts, target coverage, unsupported-case behavior, and automated legality checks.
• A cross-target evidence suite routinely compares generated programs with defined references, records numerical and semantic results, and turns discovered failures into minimized regression cases.
• Runtime and placement collaborators can consume versioned backend capability and compilation/performance signals, with demonstrated traceability from a workload decision to its compiled artifact and measured outcome.

04 / In the system

Where this role fits

Compiler ownership stops short of production placement policy and independent safety enforcement.