Senior Data Platform Engineer - Lead Data Pipelines & AI

Apply; San Francisco (CA); On-site; USD 150,000 - 210,000; Full time

Benefits offered by this job

Equity

401(k) match

Parental leave

Wellbeing benefits

Learning stipend

Remote & Office

Job summary

Handshake is hiring a Senior Data Platform Engineer to build and operate the data infrastructure powering our career marketplace and AI products. You will own systems that orchestrate batch workloads and deliver reliable data to engineers, analysts, and product teams.

You will set technical direction for our workflow orchestration platform, data ingestion, data warehouse, and streaming data infrastructure. Collaborate with ML engineers and data scientists to maximize velocity of the data

Qualifications

• Strong Python development skills and production data systems experience.
• Deep Airflow expertise: scheduling, deployment, scaling, monitoring.
• Experience with event-driven or streaming data infra and message buses.
• understanding CDC, idempotency, schema evolution, and replay in production pipelines.
• Cloud infrastructure knowledge; IaC; Kubernetes; CI/CD; observability.

Responsibilities

• Lead the technical direction and roadmap for Airflow and Astronomer platform and manage production operations.
• Build reusable operators, testing, deployment, staging environments, and runbooks.
• Design and operate streaming pipelines using Pub/Sub, Dataflow/Beam, Datastream for analytics.
• Ensure data delivery resilience: retries, duplicates, latency, freshness, reliability targets.
• Improve cloud foundations behind data workloads with Kubernetes, Terraform, IAM, secrets, cost-aware capacity.
• Lead cross-team design decisions on data contracts and ownership; mentor engineers.
• Participate in on-call and troubleshoot production failures across systems.
• Build dependable data and orchestration foundations for AI workloads.

Skills

Python

Airflow

Streaming data

Cloud infrastructure

Kubernetes

CI/CD

Infrastructure as code

Data pipelines

Communication

Job description

Handshake is hiring a Senior Data Platform Engineer to build and operate the data infrastructure powering our career marketplace and AI products. You will own systems that orchestrate batch workloads and deliver reliable data to engineers, analysts, and product teams.

You will set technical direction for our workflow orchestration platform, data ingestion, data warehouse, and streaming data infrastructure. Collaborate with ML engineers and data scientists to maximize velocity of the data