Platform Lead, Data Pipelines & AI Inference

Catalyst Labs; San Francisco (CA); On-site; USD 180,000 - 280,000; Full time

Job summary

Catalyst Labs in San Francisco seeks a Tech Lead, Data & Inference Engineer to own the design, development and scaling of a data platform spanning ingestion to insights, with a focus on speed, reliability and business usability.

You will build batch and streaming pipelines, manage reliability and cost, and lead data-as-a-product initiatives, including inference workflows with large language models. Strong leadership and hands-on expertise in modern data stacks are required.

Qualifications

• Bachelor's or Master's degree in CS, CE, EE or Math.
• Excellent written and verbal communication; collaborative mindset.
• Comfortable in hybrid or distributed environments with strong ownership.
• Founders-level bias for actionable insights and iterative improvement.
• Demonstrated ability to teach, mentor, and document decisions clearly.

Responsibilities

• Lead the design, development and scaling of an end-to-end data platform from ingestion to insights.
• Build and maintain scalable batch and streaming pipelines from diverse data sources.
• Own reliability, uptime targets (99.9%), latency and cost optimization; perform root cause analysis.
• Operate inference pipelines using LLMs and retrieval-augmented generation; manage versioning and evaluation loops.
• Deliver data as a product with clear data contracts, ownership models and lifecycle processes.
• Guide architectural decisions across the data lake and pipeline stack; document trade-offs.
• Scale integration with APIs and internal services for real-time and batch use cases.
• Mentor engineers, review code and promote data-driven best practices.

Skills

SQL

Python

Data Architecture

Data Modeling

Performance Tuning

Version Control

Education

Bachelor's or Master's in CS/CE/EE/Math

Tools

Spark

Flink

Kafka

Airflow

Dagster

Prefect

Kubernetes

Cloud (AWS/GCP/Azure)

Job description

Catalyst Labs in San Francisco seeks a Tech Lead, Data & Inference Engineer to own the design, development and scaling of a data platform spanning ingestion to insights, with a focus on speed, reliability and business usability.

You will build batch and streaming pipelines, manage reliability and cost, and lead data-as-a-product initiatives, including inference workflows with large language models. Strong leadership and hands-on expertise in modern data stacks are required.