Senior ML Data Infrastructure Engineer - Scalable Pipelines

Jobtailor; California (MO); On-site; USD 120,000 - 160,000; Full time

Job summary

Jobtailor is seeking an experienced data infrastructure engineer to build and optimize large-scale batch and streaming data pipelines for ML training in a US setting. You will work with Flink, Spark, and Ray to generate datasets and integrate with Flyte and Airflow for reliable multi-stage workflows.

You will drive reproducibility, observability, and cost-aware optimization across distributed systems while collaborating with ML teams on experimentation and model iteration.

Qualifications

• :

Responsibilities

• Develop infrastructure supporting batch and stream big data processing using Flink, Spark, Ray, and similar technologies
• Design and operate large-scale data pipelines generating training datasets for machine learning training and experimentation
• Integrate data pipelines with workflow orchestration systems such as Flyte and Airflow for reliable multi-stage training workflows
• Improve pipeline reproducibility and observability through dataset validation, monitoring, and automated testing
• Optimize performance and resource utilization across distributed compute systems
• Partner with ML engineers to enable large-scale experimentation and model iteration
• Lead architectural improvements to keep offline data pipelines scalable, reliable, and cost-efficient

Skills

Python programming

Distributed computing

Data pipeline development

Machine learning feature engineering

Tools

Flyte

Airflow

Data Lakes

Data Warehouses

Job description

Jobtailor is seeking an experienced data infrastructure engineer to build and optimize large-scale batch and streaming data pipelines for ML training in a US setting. You will work with Flink, Spark, and Ray to generate datasets and integrate with Flyte and Airflow for reliable multi-stage workflows.

You will drive reproducibility, observability, and cost-aware optimization across distributed systems while collaborating with ML teams on experimentation and model iteration.