Senior Data Infra Engineer for Scalable ML Pipelines

Unity; Mountain View (CA); On-site; USD 210,000 - 273,000; Full time

Benefits offered by this job

Health insurance

Commuter subsidy

Employee stock ownership

Retirement plan

Vacation and personal days

Parental leave

Office snacks

Mental health programs

ERGs

EAP

Training programs

Volunteer program

Job summary

Unity Vector builds a data platform powering analytics, experimentation, attribution, and AI-driven decisions across the company. We seek a senior data infra engineer to design and evolve a large-scale offline platform focusing on reliable infrastructure for training datasets and data workflows.

You will collaborate with ML engineers and platform teams to ensure pipelines scale with growing data and complex workloads.

Qualifications

• Experience with distributed computing frameworks (Flink, Spark, Ray).
• Experience building infrastructure for training data generation or ML feature pipelines.
• Experience optimizing big data pipelines for cost efficiency.
• Strong Python programming skills for large-scale workloads.
• Experience with modern data infrastructure (data lakes, warehouses, orchestration systems, streaming).
• Ability to influence architecture and technical direction without formal authority.

Responsibilities

• Develop infrastructure that supports batch and stream big data processing (Flink, Spark, Ray).
• Design and operate pipelines that generate training datasets for ML training and experimentation.
• Integrate data pipelines with workflow orchestration systems (Airflow, Flyte) for multi-stage training workflows.
• Improve reproducibility and observability of data pipelines via dataset validation, monitoring, and tests.
• Optimize performance and resource utilization across distributed compute systems.
• Partner with ML engineers to enable large-scale experimentation and model iteration.
• Lead architectural improvements to keep offline data pipelines scalable, reliable, and cost-efficient.

Skills

Distributed data processing (Flink, SP

Tools

Airflow

Flyte

Job description

Unity Vector builds a data platform powering analytics, experimentation, attribution, and AI-driven decisions across the company. We seek a senior data infra engineer to design and evolve a large-scale offline platform focusing on reliable infrastructure for training datasets and data workflows.

You will collaborate with ML engineers and platform teams to ensure pipelines scale with growing data and complex workloads.