Senior AI Research Infra Engineer - Scalable Pipelines

Menlo Ventures; San Francisco (CA); On-site; USD 190,000 - 270,000; Full time

Benefits offered by this job

Comprehensive benefits

Equity options

Annual performance bonus

Job summary

Databricks is seeking a Staff Software Engineer for AI Research Infrastructure in San Francisco. The role involves designing and implementing systems for large-scale data processing, developing the research stack for AI research, and partnering with teams to create robust pipelines. You will need a BS/MS or PhD in Computer Science and 5+ years of experience in software engineering focusing on large-scale distributed systems. The position offers a competitive salary range of $190,000 to $270,000 USD.

Qualifications

• 5+ years of software engineering experience on large-scale distributed systems.
• Deep experience with building and operating distributed systems and data pipelines.
• Experience contributing to or building cluster schedulers.

Responsibilities

• Develop research infrastructure for Databricks AI.
• Design and implement systems for large-scale data processing.
• Mentor engineers and influence the research computation roadmap.

Skills

Large-scale distributed systems

Infrastructure programming languages (C++, Rust, Go, Java, Scala)

Cluster scheduling and resource management systems

ML training and inference workflows

Education

BS/MS or PhD in Computer Science or related field

Tools

Kubernetes

Slurm

Ray

Job description

Databricks is seeking a Staff Software Engineer for AI Research Infrastructure in San Francisco. The role involves designing and implementing systems for large-scale data processing, developing the research stack for AI research, and partnering with teams to create robust pipelines. You will need a BS/MS or PhD in Computer Science and 5+ years of experience in software engineering focusing on large-scale distributed systems. The position offers a competitive salary range of $190,000 to $270,000 USD.