Tech Lead, Data & Inference Engineer

Catalyst Labs; San Francisco (CA); On-site; USD 180,000 - 280,000; Full time

Job summary

Catalyst Labs in San Francisco seeks a Tech Lead, Data & Inference Engineer to own the design, development and scaling of a data platform spanning ingestion to insights, with a focus on speed, reliability and business usability.

You will build batch and streaming pipelines, manage reliability and cost, and lead data-as-a-product initiatives, including inference workflows with large language models. Strong leadership and hands-on expertise in modern data stacks are required.

Qualifications

• Bachelor's or Master's degree in CS, CE, EE or Math.
• Excellent written and verbal communication; collaborative mindset.
• Comfortable in hybrid or distributed environments with strong ownership.
• Founders-level bias for actionable insights and iterative improvement.
• Demonstrated ability to teach, mentor, and document decisions clearly.

Responsibilities

• Lead the design, development and scaling of an end-to-end data platform from ingestion to insights.
• Build and maintain scalable batch and streaming pipelines from diverse data sources.
• Own reliability, uptime targets (99.9%), latency and cost optimization; perform root cause analysis.
• Operate inference pipelines using LLMs and retrieval-augmented generation; manage versioning and evaluation loops.
• Deliver data as a product with clear data contracts, ownership models and lifecycle processes.
• Guide architectural decisions across the data lake and pipeline stack; document trade-offs.
• Scale integration with APIs and internal services for real-time and batch use cases.
• Mentor engineers, review code and promote data-driven best practices.

Skills

SQL

Python

Data Architecture

Data Modeling

Performance Tuning

Version Control

Education

Bachelor's or Master's in CS/CE/EE/Math

Tools

Spark

Flink

Kafka

Airflow

Dagster

Prefect

Kubernetes

Cloud (AWS/GCP/Azure)

Job description

Tech Lead, Data & Inference Engineer

Location: San Francisco

Work type: Full Time

Compensation: above market base + bonus + equity

Roles & Responsibilities

• Lead the design, development and scaling of an end‑to‑end data platform from ingestion to insights, ensuring that data is fast, reliable and ready for business use.
• Build and maintain scalable batch and streaming pipelines, transforming diverse data sources and third‑party APIs into trusted and low‑latency systems.
• Take full ownership of reliability, cost and service level objectives, including achieving 99.9 percent uptime, maintaining minutes‑level latency and optimizing cost per terabyte; conduct root‑cause analysis and provide long‑lasting solutions.
• Operate inference pipelines that enhance and enrich data, including enrichment, scoring and quality assurance using large language models and retrieval‑augmented generation; manage version control, caching and evaluation loops.
• Work across teams to deliver data as a product through the creation of clear data contracts, ownership models, lifecycle processes and usage‑based decision making.
• Guide architectural decisions across the data lake and the entire pipeline stack; document lineage, trade‑offs and reversibility while making practical decisions on whether to build internally or buy externally.
• Scale integration with APIs and internal services while ensuring data consistency, high data quality and support for both real‑time and batch‑oriented use cases.
• Mentor engineers, review code and raise the overall technical standard across teams; promote data‑driven best practices throughout the organization.

Qualifications

• Bachelor's or Master's degree in Computer Science, Computer Engineering, Electrical Engineering, or Mathematics.
• Excellent written and verbal communication; proactive and collaborative mindset.
• Comfortable in hybrid or distributed environments with strong ownership and accountability.
• Founders‑level bias for actionable insights, identifying bottlenecks, automating workflows, and iterating rapidly based on measurable outcomes.
• Demonstrated ability to teach, mentor, and document technical decisions and schemas clearly.

Core Experience

• 6 to 12 years of experience building and scaling production‑grade data systems, with deep expertise in data architecture, modeling and pipeline design.
• Expert SQL (query optimization on large datasets) and Python skills.
• Hands‑on experience with distributed data technologies (Spark, Flink, Kafka) and modern orchestration tools (Airflow, Dagster, Prefect).
• Familiarity with dbt, DuckDB and the modern data stack; experience with IaC, CI/CD and observability.
• Exposure to Kubernetes and cloud infrastructure (AWS, GCP or Azure).
• Bonus: Strong Node.js skills for faster onboarding and system integration.
• Previous experience at a high‑growth startup (10 to 200 people) or early‑stage environment with a strong product mindset.