Staff Data Engineer, AI Platform — Scale ML Data Pipelines

AlleyCorp; San Jose (CA); On-site; USD 175,000 - 215,000; Full time

Job summary

Archer is seeking a Staff Data Engineer for the AI Platform in San Jose to design, build, and operate data infrastructure powering large-scale model training and inference. You will own pipelines, storage, and data quality to ensure models train on clean, high-throughput, well-governed data.

You will partner with ML engineers to define feature stores, dataset versioning, and data contracts; collaborate with researchers and platform engineers to optimize queries and unblock training runs.

Qualifications

• 5+ years of professional data engineering experience.
• BS/MS/PhD in CS, Data Engineering, Software Engineering, or a related field.
• Hands-on experience building production pipelines with tools like Apache Spark, Flink, Airflow, dbt, or similar batch/streaming frameworks.
• Deep proficiency with columnar formats (Parquet), open table formats (Iceberg or Paimon), and object storage systems (S3 or equivalent).
• Experience with high-throughput messaging systems such as Apache Pulsar or Kafka for real-time data ingestion.
• Strong SQL skills; experience with StarRocks for large-scale analytical queries and real-time analytics over the lakehouse.
• Familiarity with AWS data services (S3, Glue, EMR) and containerized workloads (Docker/Kubernetes) in production environments. Airflow/Prefect/Dagster

Responsibilities

• Design and maintain high-throughput ingestion and transformation pipelines that feed training workloads at scale, with a focus on latency, throughput, and correctness.
• Build and operate the data lakehouse — defining table formats (Iceberg, Paimon, Parquet), partitioning strategies, and compaction policies optimized for ML consumption patterns.
• Instrument pipelines with data quality checks, lineage tracking, and anomaly detection so that model failures trace back to data problems quickly.
• Partner with ML engineers to define feature stores, dataset versioning, and experiment-to-production data contracts; integrate with tools like MLflow for dataset and artifact tracking.
• Work closely with AI researchers, platform engineers, and software engineers to understand data access patterns, optimize query performance, and unblock training runs.

Skills

5+ years data engineering

SQL proficiency

Spark/Flink/Airflow

Parquet/Iceberg

AWS data services

Education

BS/MS/PhD in CS/DE/SE

Tools

Apache Spark

Flink

Airflow

dbt

Iceberg

Parquet

Job description

Archer is seeking a Staff Data Engineer for the AI Platform in San Jose to design, build, and operate data infrastructure powering large-scale model training and inference. You will own pipelines, storage, and data quality to ensure models train on clean, high-throughput, well-governed data.

You will partner with ML engineers to define feature stores, dataset versioning, and data contracts; collaborate with researchers and platform engineers to optimize queries and unblock training runs.