Lead Platform Engineer, Data Infra & AI Pipelines

True; San Francisco (CA); On-site; USD 220,000 - 300,000; Full time

Job summary

Confidential company in San Francisco, CA seeks a platform/infra engineer to own architecture, security, and scaling of a data platform. You will build pipelines, APIs, and storage with strong focus on provenance and async workloads.

The role requires 3–7 years in platform engineering, Kubernetes/Terraform/Ansible expertise, and comfort with Go, Python, and modern observability tools. On-site in San Francisco with relocation support.

Qualifications

• 3–7 years in platform or infrastructure engineering.
• Hands-on production backend infrastructure, data platforms, or distributed systems.
• Proficiency with Kubernetes, Terraform, Ansible, and modern CI/CD.
• Experience designing ETL/ELT pipelines for complex data.
• Strong software and infrastructure engineering background.
• Experience with Go, Python, FastAPI, TypeDB/TypeQL, Qdrant, GitHub Actions, OpenTelemetry, Prometheus.
• Biotechnology or computational biology data compliance or infra experience helpful.

Responsibilities

• Own architecture, delivery, security posture, and scaling of the core data platform.
• Build and maintain ingestion pipelines, APIs, microservices, and storage layers that preserve data provenance.
• Manage knowledge graph and vector search for AI agent workloads.
• Develop high-concurrency services in Go and Python with agent writes and isolated execution.
• Establish platform security practices: authentication, tenant isolation, secrets management, encryption, zero-trust.
• Own cloud infrastructure, CI/CD, observability, and define service objectives and internal workflows.
• Set engineering patterns, review code, mentor teammates, recruit the platform team.
• Partner with leadership on team design and interviewing.

Skills

Go

Python

FastAPI

Kubernetes

Terraform

Ansible

CI/CD

ETL/ELT design

Tools

TypeDB/TypeQL

Qdrant

GitHub Actions

OpenTelemetry

Prometheus

Job description

Confidential company in San Francisco, CA seeks a platform/infra engineer to own architecture, security, and scaling of a data platform. You will build pipelines, APIs, and storage with strong focus on provenance and async workloads.

The role requires 3–7 years in platform engineering, Kubernetes/Terraform/Ansible expertise, and comfort with Go, Python, and modern observability tools. On-site in San Francisco with relocation support.