Lead Data Platform Engineer – Scalable Pipelines & AI

Swoop Airlines and Aviation; Redwood City (CA); On-site; USD 170,000 - 210,000; Full time

Job summary

NimbleRx is seeking a Senior Data Engineer to own the data platform end-to-end, shaping ingestion, transformation, storage, and access for the company. You’ll build batch and streaming pipelines using PySpark/EMR, Kinesis, Lambda, and Step Functions, feeding an Iceberg-based lake from Postgres, Salesforce, and event streams.

You’ll collaborate with product, engineering, analytics, and operations to deliver reliable pipelines, security, and scalable data tooling, while mentoring teams and

Qualifications

• 5+ years of experience building production data pipelines and platforms.
• Deep Python (PySpark) and SQL fluency, including tuning Spark jobs at scale.
• The skills and willingness to work on the services around the data layer (Java, Spring Boot)
• Hands-on experience with distributed compute (Spark/EMR), streaming (Kinesis), and object storage (S3)
• Solid Postgres fundamentals — query optimization, indexing, replication, replica routing, and a feel for when the database is the bottleneck
• Experience with Iceberg and Trino, or similar
• Comfort with CI/CD and Terraform
• Already building with AI — frontier models, agentic coding tools, or something you hacked together last weekend
• Track record of working across teams that don’t speak your language (product, ops, etc.)

Responsibilities

• Own the data platform end-to-end — ingestion, transformation, storage, query, and access — and drive its roadmap as the company’s data needs grow.
• Build and evolve batch and streaming pipelines on PySpark/EMR, Kinesis, Lambda, and Step Functions, ingesting from Postgres, Salesforce, third-party vendors, and product event streams into our Iceberg-based lake.
• Model the warehouse: design SCD tables, event tables, and the conventions other engineers and analysts follow when adding new data.
• Partner with product, engineering, analytics, and operations stakeholders across the company to turn data requests into well-scoped, reliable pipelines — and write the docs and tooling that let them self-serve next time.
• Own the security and compliance backbone of our data systems, including audit logging, access control, and temporary-access workflows.
• Optimize backend query performance where the data layer meets product code — read-replica routing, indexing, caching, and IO instrumentation in our Java/Spring services.
• Lead investigation and remediation when data infrastructure misbehaves — IOPS spikes, pipeline failures, schema drift, late data — and make the fixes durable.
• Use AI as a daily accelerant for pipeline scaffolding, schema work, and ad-hoc investigations — and ship internal AI tooling that lets other teams do the same.
• Mentor engineers and analysts across the company on how to work with the data platform.

Skills

Python

SQL

Cross-team collaboration

AI tooling

Tools

Java

Spring Boot

Spark/EMR

Kinesis

S3

Iceberg

Trino

Terraform

CI/CD

AI tooling

Job description

NimbleRx is seeking a Senior Data Engineer to own the data platform end-to-end, shaping ingestion, transformation, storage, and access for the company. You’ll build batch and streaming pipelines using PySpark/EMR, Kinesis, Lambda, and Step Functions, feeding an Iceberg-based lake from Postgres, Salesforce, and event streams.

You’ll collaborate with product, engineering, analytics, and operations to deliver reliable pipelines, security, and scalable data tooling, while mentoring teams and