Tech Lead, Data & Inference Engineer: Scale AI Pipelines

Catalyst Labs; Palo Alto (CA); On-site; USD 150,000 - 180,000; Full time

Job summary

A leading talent agency is seeking a Tech Lead, Data & Inference Engineer in Palo Alto, California. In this full-time role, you will design and scale an end-to-end data platform and build scalable data pipelines. The ideal candidate holds a relevant degree, has 6-12 years of experience in production-grade data systems, and excels in SQL and Python. You'll mentor engineers, enhance data quality, and make architectural decisions to drive innovation in data-driven solutions.

Qualifications

• 6 to 12 years of experience building and scaling production-grade data systems.
• Strong ownership and accountability in hybrid or distributed environments.
• Ability to teach, mentor, and document technical decisions clearly.

Responsibilities

• Lead the design, development, and scaling of an end-to-end data platform.
• Build and maintain scalable batch and streaming pipelines.
• Guide architectural decisions across the data lake and entire pipeline stack.
• Mentor engineers and promote data-driven best practices.

Skills

SQL query optimization on large datasets

Python

Data architecture

Pipeline design

Distributed data technologies (Spark, Flink, Kafka)

Modern orchestration tools (Airflow, Dagster, Prefect)

Kubernetes

Cloud infrastructure (AWS, GCP, or Azure)

Node.js

Education

Bachelor's or Master's degree in Computer Science, Computer Engineering, Electrical Engineering, or Mathematics

Job description

A leading talent agency is seeking a Tech Lead, Data & Inference Engineer in Palo Alto, California. In this full-time role, you will design and scale an end-to-end data platform and build scalable data pipelines. The ideal candidate holds a relevant degree, has 6-12 years of experience in production-grade data systems, and excels in SQL and Python. You'll mentor engineers, enhance data quality, and make architectural decisions to drive innovation in data-driven solutions.