Senior Data Engineer

DataJobs; Redwood City (CA); Hybrid; USD 205,000 - 240,000; Full time

Job summary

MyHealthTeam is seeking a Senior Data Engineer in a hybrid role based in Redwood City, CA. You will own the data platform end-to-end, building and operating ingestion, transformation, storage, and access while guiding the platform roadmap to support product, analytics, and operations teams.

You will design batch and streaming pipelines, model the warehouse, and ensure security, compliance, and scalable data operations. Collaboration across teams and mentoring others are key aspects of the role.

Qualifications

• 5+ years of production data pipelines and platforms experience.
• Strong Python (PySpark) and SQL skills with Spark tuning.
• Experience with Java and Spring Boot services.
• Hands-on with Spark/EMR, Kinesis, and S3.
• Postgres performance and indexing expertise.
• Experience with Iceberg and Trino.
• Familiar with CI/CD and Terraform.
• Experience using AI to accelerate pipelines.

Responsibilities

• Own the data platform end-to-end and its roadmap.
• Build and evolve batch and streaming pipelines.
• Model the warehouse with SCD and event tables.
• Collaborate with cross-functional teams and provide self-serve tooling.
• Ensure security and compliance with audit logging and access controls.
• Optimize backend query performance and IO instrumentation.
• Lead incident remediation for data infra issues.
• Mentor engineers and analysts.
• Use AI tooling to accelerate pipelines.

Skills

Python (PySpark)

SQL

Distributed computing

AI tooling

Tools

EMR

Kinesis

Lambda

Step Functions

Postgres

Salesforce

Iceberg

Trino

S3

Java

Spring Boot

CI/CD

Terraform

Job description

MyHealthTeam is hiring a Senior Data Engineer (hybrid, Redwood City, CA) to own the data platform end-to-end. In this role, you will build and operate ingestion, transformation, storage, query, access, and the platform roadmap so product, analytics, and operations teams can deliver faster while meeting security and compliance needs.

Role Overview

You will cover both production engineering and data engineering responsibilities across the data and product services ecosystem. The work includes pipeline development for batch and streaming workloads, warehouse modeling and conventions, performance optimization at the data layer, and reliable operations when infrastructure issues arise.

Key Responsibilities

• Own the data platform end-to-end, including ingestion, transformation, storage, query, and access, and lead its roadmap as data demands grow.
• Build and evolve batch and streaming pipelines using PySpark/EMR, Kinesis, Lambda, and Step Functions, ingesting from Postgres, Salesforce, third‑party vendors, and product event streams into an Iceberg-based lake.
• Model the warehouse by designing SCD tables, event tables, and the conventions used by engineers and analysts when adding new datasets.
• Collaborate with product, engineering, analytics, and operations stakeholders to scope data requests into reliable pipelines, and provide documentation and tooling for self‑serve use.
• Own the security and compliance backbone of data systems, including audit logging, access control, and temporary‑access workflows.
• Optimize backend query performance where the data layer meets product code, including read‑replica routing, indexing, caching, and IO instrumentation in Java/Spring services.
• Lead investigation and remediation for data infrastructure issues such as IOPS spikes, pipeline failures, schema drift, and late data, with durable fixes.
• Use AI to accelerate pipeline scaffolding, schema work, and ad‑hoc investigations, including shipping internal AI tooling for other teams.
• Mentor engineers and analysts on how to use and work effectively with the data platform.

Required Qualifications

• 5+ years of experience building production data pipelines and platforms.
• Strong Python (PySpark) and SQL skills, including tuning Spark jobs at scale.
• Ability and willingness to work on services around the data layer, including Java and Spring Boot.
• Hands‑on experience with distributed compute (Spark/EMR), streaming (Kinesis), and object storage (S3).
• Solid fundamentals in Postgres, including query optimization, indexing, replication, replica routing, and diagnosing bottlenecks.
• Experience with Iceberg and Trino (or comparable tooling).
• Comfort working with CI/CD and Terraform.
• Experience building with AI, including frontier models, agentic coding tools, or practical prototypes.
• Proven track record of collaborating across teams with different priorities and language (for example product and operations).

Technology Stack

PySpark, EMR, Kinesis, Lambda, Step Functions, Postgres, Salesforce, Iceberg, Trino, S3, Java, Spring Boot, CI/CD, Terraform, SQL

About the Role and Team Impact

As a Senior Data Engineer, you will build the platform layer that enables other teams to understand the business, ship product features, and meet compliance obligations. The goal is for the systems you create to scale beyond individual tickets, supporting product, analytics, operations, and data science over time. The work sits at the intersection of production engineering and data engineering, with impact measured by whether teams move faster due to the platform.

What Success Looks Like

• Own platform reliability, cost, accessibility, and compliance beyond day‑to‑day ticket delivery.
• Treat security and PII/PHI handling as core engineering work.
• Leave behind documentation, tools, and workflows that increase speed for engineers and analysts.
• Ship iteratively with instrumentation, favoring quick demos of rough versions over long polished releases.
• Prioritize effectively under urgency with good judgment.

Compensation

USD 205,000 - 240,000 per year

Hybrid location: Redwood City, CA, US