Senior/Staff Machine Learning Engineer, Data Infrastructure

Jobtailor; California (MO); On-site; USD 120,000 - 160,000; Full time

Job summary

Jobtailor is seeking an experienced data infrastructure engineer to build and optimize large-scale batch and streaming data pipelines for ML training in a US setting. You will work with Flink, Spark, and Ray to generate datasets and integrate with Flyte and Airflow for reliable multi-stage workflows.

You will drive reproducibility, observability, and cost-aware optimization across distributed systems while collaborating with ML teams on experimentation and model iteration.

Qualifications

• :

Responsibilities

• Develop infrastructure supporting batch and stream big data processing using Flink, Spark, Ray, and similar technologies
• Design and operate large-scale data pipelines generating training datasets for machine learning training and experimentation
• Integrate data pipelines with workflow orchestration systems such as Flyte and Airflow for reliable multi-stage training workflows
• Improve pipeline reproducibility and observability through dataset validation, monitoring, and automated testing
• Optimize performance and resource utilization across distributed compute systems
• Partner with ML engineers to enable large-scale experimentation and model iteration
• Lead architectural improvements to keep offline data pipelines scalable, reliable, and cost-efficient

Skills

Python programming

Distributed computing

Data pipeline development

Machine learning feature engineering

Tools

Flyte

Airflow

Data Lakes

Data Warehouses

Job description

• Develop infrastructure supporting batch and stream big data processing using Flink, Spark, Ray, and similar technologies
• Design and operate large-scale data pipelines generating training datasets for machine learning training and experimentation
• Integrate data pipelines with workflow orchestration systems such as Flyte and Airflow for reliable multi-stage training workflows
• Improve pipeline reproducibility and observability through dataset validation, monitoring, and automated testing
• Optimize performance and resource utilization across distributed compute systems
• Partner with ML engineers to enable large-scale experimentation and model iteration
• Lead architectural improvements to keep offline data pipelines scalable, reliable, and cost-efficient

Requirements

• Experience working with distributed computing frameworks such as Flink, Spark, and Ray for distributed data processing
• Experience building infrastructure for training data generation, dataset preparation, or ML feature pipelines
• Experience optimizing big data pipelines and infrastructure for cost efficiency
• Strong programming skills in Python and experience with large-scale distributed workloads
• Experience with modern data infrastructure, including data lakes, warehouses, orchestration systems, and streaming platforms
• Strong systems thinking and ability to reason about performance, scalability, reliability, and cost tradeoffs in distributed systems
• Proven ability to lead technical direction and influence architectural decisions across teams without formal authority
• Sufficient knowledge of English for professional verbal and written exchanges
• Work visa/immigration sponsorship is not available for this position

Core Competencies

Demonstrates expertise in developing and optimizing large-scale data pipelines for machine learning, utilizing distributed computing frameworks like Flink, Spark, and Ray. Proficient in integrating orchestration systems and ensuring pipeline reliability and cost efficiency.

Highest-signal resume keywords

• Flink
• Spark
• Ray
• Python Programming
• Data Pipeline Optimization

ATS Optimization Keywords

Hard Skills

• Distributed Computing
• Data Pipeline Development
• Machine Learning Feature Engineering
• Dataset Validation
• Automated Testing

Soft Skills

• Systems Thinking
• Technical Leadership
• Influencing Architectural Decisions

Industry Keywords

• Big Data Processing
• Performance Optimization
• Resource Utilization
• Scalability
• Reliability

Tools & Technologies

• Flyte
• Airflow
• Data Lakes
• Data Warehouses
• Streaming Platforms