ML Engineer

Breakout Ventures; United States; Hybrid; USD 180,000 - 260,000; Full time

Job summary

Noetik is seeking highly technical ML engineers to join a hybrid South San Francisco team building scalable AI systems for oncology. You will design, train, and deploy production-grade workflows powering large foundation models and data-driven drug discovery.

Join a cross-disciplinary group focused on reproducibility, rapid iteration, and close collaboration with ML researchers. The role emphasizes scalable infrastructure, observability, and high-performance GPU workloads in a hybrid setup.

Qualifications

• Strong software engineering fundamentals for production ML systems.
• Experience with distributed model training in large-scale environments.
• Reading, profiling, debugging, and modifying model training and inference code.

Responsibilities

• Design, build, and evolve systems for distributed model training and batch inference.
• Develop workflows for experiment tracking, model evaluation, and registry management.
• Profile and optimize training and inference performance across PyTorch, data loading, GPU utilization, and memory.
• Build researcher-facing CLIs, SDKs, and job submission tools for scalable runs.
• Develop and operate cost-efficient GPU infrastructure using Kubernetes, AWS, Ray, and Terraform.
• Collaborate with ML researchers to translate needs into platform improvements.
• Use AI development tools to accelerate implementation, debugging, profiling and testing.

Skills

Distributed training

PyTorch

GPU systems

Kubernetes

Cloud infrastructure

Profiling tools

Debugging complex systems

Collaboration with researchers

Tools

Terraform

Docker

Ray

MLflow

Job description

WHY NOETIK

NOETIK is a frontier therapeutics company harnessing AI to deeply understand human biology and redefine clinical outcomes in oncology. Leveraging the largest proprietary multimodal oncology dataset of its kind, uniquely integrating deep spatial profiling with routine clinical assays, NOETIK’s foundation AI models precisely match therapeutic targets to patient subpopulations and accurately predict clinical efficacy. This AI-first approach elevates therapeutic development from probabilistic guesswork to precision science.

THE OPPORTUNITY

At Noetik, you will work alongside a highly technical, interdisciplinary team tackling hard and impactful problems. We are looking for people who are excited to engage deeply and bring a strong sense of urgency to making an impact.

In this role, you’ll work with a high-caliber team building systems and conducting research at the forefront of AI. At Noetik, AI is at the heart of everything we do: we generate our own data purpose-built for machine learning, build systems for advanced machine learning research, train large foundation models from scratch, and center our drug discovery work on these models. Here, AI is not a buzzword. It is the core of our company and the key to deciphering complex biology to make a difference for patients.

KEY RESPONSIBILITIES

• Design, build, and evolve systems for distributed model training and batch inference as Noetik’s workloads grow in scale and complexity.

• Develop workflows for experiment tracking, model evaluation, and model registry management that support reproducibility across the ML lifecycle.

• Profile and optimize training and inference performance across PyTorch model code, data loading, GPU utilization, memory usage, distributed communication, and checkpointing.

• Build researcher-facing CLIs, SDKs, and job submission tools that make it easy to move from experimentation to large-scale runs with minimal infrastructure knowledge.

• Develop and operate reliable, observable, and cost-efficient GPU infrastructure using technologies such as Kubernetes, AWS, Ray, Kueue, Docker, Terraform, and high-performance networking.

• Work directly with ML researchers to troubleshoot issues, understand evolving research needs, and translate them into platform improvements that support rapid iteration.

• Use modern AI development tools and agents throughout the engineering workflow to accelerate implementation, debugging, profiling, testing, exploration, and maintenance.

QUALIFICATIONS

• Strong software engineering fundamentals and experience building and operating production machine learning systems.

• Hands-on experience with distributed model training in a production or large-scale research environment.

• Strong understanding of multi-GPU and distributed training concepts, including the performance and reliability challenges involved in scaling ML workloads across multiple machines.

• Experience working with PyTorch or a similar deep learning framework, with the ability to read, profile, debug, and modify model training and inference code.

• Experience identifying and resolving performance bottlenecks across ML workloads, including GPU utilization, data pipelines, memory, distributed communication, and infrastructure.

• Comfort working directly with cloud and compute infrastructure, including containers, Kubernetes, GPU systems, storage, networking, and distributed compute.

• Experience building tooling or platforms that improve the developer experience of machine learning researchers or engineers.

• Ability to debug complex problems spanning model code, distributed frameworks, infrastructure, and underlying compute resources.

• Ability to balance reliability and standardization with the speed and flexibility required in a fast-moving research environment.

• Comfort using AI coding tools and agents as a core part of the software development process while maintaining a high bar for correctness, maintainability, and understanding of the systems being built.

• Strong collaboration skills and the ability to work closely with ML researchers to understand their workflows, identify friction, and build systems that make them more effective.

• A collaborative, low-ego approach, with a focus on solving problems as a team and finding the best solution rather than being right.

• Typically, several years of experience working with production ML systems, distributed training, or ML infrastructure. We care more about demonstrated depth and impact than a specific number of years.

NICE TO HAVE

• Experience with Ray, Kubernetes, Kueue, MLflow, Terraform, or similar ML infrastructure and orchestration technologies.

• Experience with high-performance GPU networking and distributed communication technologies such as NCCL or EFA.

• Experience optimizing large-scale model training for throughput, memory efficiency, checkpointing, or compute utilization.

• Experience building large-scale batch inference systems or online model-serving infrastructure.

• Experience operating GPU clusters in AWS or another major cloud environment.

• Experience with modern distributed training approaches such as FSDP, DeepSpeed, or similar frameworks.

• Experience with profiling tools, CUDA-level performance debugging, or other low-level GPU optimization techniques.

Life@Noetik

Great science is a team sport. We are focused on creating a culture where brilliant people from different backgrounds can solve hard problems together. At Noetik, culture isn't just about perks; it’s about building a vibrant, collaborative community where everyone has the tools and the autonomy to thrive.

How We Work

Location & Collaboration: our primary lab and office space is located in South San Francisco. We operate as a hybrid organization, utilizing Slack, Zoom, and a modern tech stack to stay seamlessly connected regardless of time zone.

Intentional Connection: we believe that high-impact work requires intentional time together. We keep the team aligned through monthly All-Hands meetings and Quarterly Co-working Weeks at our South San Francisco office. Once a year, we gather for an Annual Reset - a week-long offsite dedicated to high-level strategy, collaboration and team bonding. Past locations have included Austin, TX, Park City, UT, and Boulder, CO.

Onsite Community: For team members working onsite, we foster a collaborative environment and refuel with daily catered lunches from great local restaurants and Boba Thursday - a small but meaningful tradition where we step away from our desks to connect over a break.

How We Support You

Health & Wellbeing: we provide generous health, dental, and vision insurance, with premiums fully covered for both employees and their dependents (depending on your plan choice). Our coverage is designed to be comprehensive and accessible, featuring modern care platforms like OneMedical, Midi Health, Spring Health, and XP Health. To further support your needs, we offer HSA and FSA accounts, dependent care savings, an EAP, resources for mental health (Rula), and health care navigation and advocacy (Rightway).

Time Off & Synchronized Breaks: we offer Flexible PTO to manage your personal time, alongside nine paid holidays. We also offer synchronized breaks so that the team can fully unplug simultaneously - two full-week company breaks (the first week of July and the last week of December), plus the Friday after Thanksgiving.

Compensation & Financial Benefits: our financial package includes a competitive base salary, stock options, and a 401(k) plan with an employer match. We provide a safety net through fully-covered disability, life and AD&D insurance, as well as generous parental leave. We also offer a suite of financial and lifestyle tools, including SoFi, Multiply Mortgage, and optional programs for pet insurance, legal services, and identity theft protection.

We believe that the highest performing teams include people from a wide variety of backgrounds and experiences who respectfully challenge each other. We are committed to an open, diverse and inclusive culture for all employees. Noetik is proud to be an equal opportunity employer and does not discriminate on the basis of race, color, citizenship status, national origin, ancestry, sex, sexual orientation, age, religion, creed, physical or mental disability, medical condition, marital status, veteran status and any other characteristics protected under applicable federal, state and local laws.