Lead Data Infra Engineer: Streaming & Batch Pipelines

Francisco Partners; Mountain View (CA); On-site; USD 175,000 - 221,000; Full time

Benefits offered by this job

Health, dental, and vision care

Top-tier 401(K) plan with company matching

Paid time off and holidays

FSA, HSA and commuter benefits

Team activity budget

Job summary

Francisco Partners is looking for a Senior Engineer to own and scale data pipelines, impacting millions of users. This role requires a deep understanding of Spark and Kafka, with a focus on optimizing performance and ensuring data quality.

The ideal candidate will have over 5 years of data engineering experience, a track record of improving data system efficiency, and a BS/MS in Computer Science.

Benefits include health coverage, a top-tier 401(K), and generous paid time off.

Qualifications

• 5+ years of data engineering experience, with at least 3 years owning production pipelines.
• Strong hands-on experience with Spark and Kafka at scale.
• Experience with big-data technologies like Hadoop, Presto/Trino, and Flink.

Responsibilities

• Own the streaming data backbone, designing Kafka pipelines.
• Build batch pipelines, optimizing Spark jobs for billions of rows daily.
• Drive pipeline observability and define SLOs for data freshness.

Skills

Data engineering experience (5+ years)

Hands-on experience with Spark

Experience with Kafka

Proficiency in Python and SQL

Experience with big-data technologies

Cloud experience, preferably AWS

Strong communication skills

Education

BS or MS in Computer Science or equivalent

Tools

Spark

Kafka

AWS

Hadoop

Presto/Trino

Flink

Job description

Francisco Partners is looking for a Senior Engineer to own and scale data pipelines, impacting millions of users. This role requires a deep understanding of Spark and Kafka, with a focus on optimizing performance and ensuring data quality.

The ideal candidate will have over 5 years of data engineering experience, a track record of improving data system efficiency, and a BS/MS in Computer Science.

Benefits include health coverage, a top-tier 401(K), and generous paid time off.