Remote - Senior PySpark Developer

Resource Informatics Group, Inc; United States; On-site; USD 140,000 - 200,000; Full time

Job summary

Resource Informatics Group, Inc is seeking a senior Data Engineer to lead the hands-on creation and optimization of heavy-duty data pipelines for a large cloud transformation project. You will move enterprise data into a modern cloud platform while optimizing performance, data quality, and automation.

Responsibilities include building high-performance PySpark pipelines, architecting data lakehouse models with Iceberg on CDP and AWS, and automating workflows with Airflow integrated into CI/CD

Qualifications

• 12+ years in IT with 10+ years in Data Engineering/Big Data.
• Strong PySpark, CDP, Iceberg and Airflow expertise.
• Deep SQL, data modeling, ETL/ELT design, and Git/CI/CD experience.

Responsibilities

• Build high-performance PySpark pipelines to move and transform large data volumes.
• Architect modern data lakehouse models using Iceberg on CDP and AWS.
• Automate workflows with Airflow and integrate into CI/CD pipelines.
• Collaborate with architects, analysts, and cloud teams end-to-end.

Skills

PySpark

CDP

Iceberg

Airflow

SQL

Data Modeling

ETL/ELT

Git/CI/CD

Education

Bachelor's or equivalent in CS/IT/Engineering

Tools

AWS

Kafka

Cloudera Data Platform

S3

Glue

EMR

Lambda

Job description

Contract: Long-term | Process: 2 Video Interviews | Pay: Depends on Experience

What You’ll Be Doing

You will lead the hands-on creation and optimization of heavy-duty data pipelines for a large cloud transformation project. You'll move enterprise data into a modern cloud platform while optimizing performance, data quality, and automation.

• Build & Scale: Develop high-performance PySpark applications and ETL/ELT pipelines to process huge volumes of data.
• Architect Modern Tech: Design data lakehouse models using Apache Iceberg on Cloudera and AWS.
• Automate & Orchestrate: Create and manage automated workflows using Apache Airflow, integrated with CI/CD pipelines.
• Collaborate: Partner with architects, analysts, and cloud teams to design solutions end-to-end (from requirements to production support).

What You Need to Apply

• Experience: 12+ years in IT, with 10+ years focused on Data Engineering and Big Data.
• Core Tech Stack: Advanced PySpark expertise, Cloudera Data Platform (CDP), Apache Iceberg, and Apache Airflow.
• Foundations: Deep knowledge of SQL, data modeling, ETL/ELT design, and Git/CI/CD practices.
• Education: Degree in Computer Science, IT, Engineering, or a related field.

Preferred Skillset

Hands-on AWS (S3, Glue, EMR, Lambda, ECS/EKS, RDS) & Cloud-Native Lakehouse architecture.

Performance:

Advanced query optimization, data partitioning, and performance tuning.

Additional Tech:

Real-time streaming (Kafka), Python/Java development, or data governance/quality tools.

Domain & Certs:

Healthcare, Medicaid, or Claims industry background; AWS or Cloudera certifications.