Platform Data Engineer: Scale Petabyte Pipelines with Spark

The Cypress Group; United States; On-site; USD 140,000 - 190,000; Full time

Job summary

The Cypress Group is seeking a Platform Software Engineer to design, develop, and scale a core data platform powering our AI-driven products. You will build distributed systems capable of processing petabyte-scale datasets and develop infrastructure used by software engineers, data scientists, analysts, and AI teams.

You will design, build, and optimize ETL pipelines, collaborate with cross-functional teams, and leverage modern cloud technologies to accelerate software delivery.

Qualifications

• 4+ years of professional software engineering experience.
• Experience building and maintaining large-scale software platforms.
• Strong experience developing ETL and data ingestion pipelines.
• Strong knowledge of Apache Spark.
• Understanding of distributed data processing systems.
• Experience with data modeling concepts and software design patterns.
• Strong collaboration and communication skills.

Responsibilities

• Design, build, and maintain core data platform software supporting large-scale distributed systems.
• Develop and optimize ETL pipelines that ingest, transform, and normalize massive datasets.
• Build high-performance data processing applications using Apache Spark, Databricks, and Scala.
• Collaborate with software engineers, data scientists, and business stakeholders on high-impact technical initiatives.
• Improve platform reliability, scalability, and developer productivity.
• Participate in architecture discussions, code reviews, and continuous improvement initiatives.
• Utilize AI-assisted development tools such as Claude Code and GitHub Copilot to improve engineering efficiency.

Skills

Distributed systems

ETL pipelines

Spark

Data modeling

Collaboration

Tools

Databricks

AWS

Terraform

Claude Code

GitHub Copilot

Job description

The Cypress Group is seeking a Platform Software Engineer to design, develop, and scale a core data platform powering our AI-driven products. You will build distributed systems capable of processing petabyte-scale datasets and develop infrastructure used by software engineers, data scientists, analysts, and AI teams.

You will design, build, and optimize ETL pipelines, collaborate with cross-functional teams, and leverage modern cloud technologies to accelerate software delivery.