Senior PySpark Engineer: Cloud Data Pipelines & Lakehouse

Resource Informatics Group, Inc; United States; On-site; USD 140,000 - 200,000; Full time

Job summary

Resource Informatics Group, Inc is seeking a senior Data Engineer to lead the hands-on creation and optimization of heavy-duty data pipelines for a large cloud transformation project. You will move enterprise data into a modern cloud platform while optimizing performance, data quality, and automation.

Responsibilities include building high-performance PySpark pipelines, architecting data lakehouse models with Iceberg on CDP and AWS, and automating workflows with Airflow integrated into CI/CD

Qualifications

• 12+ years in IT with 10+ years in Data Engineering/Big Data.
• Strong PySpark, CDP, Iceberg and Airflow expertise.
• Deep SQL, data modeling, ETL/ELT design, and Git/CI/CD experience.

Responsibilities

• Build high-performance PySpark pipelines to move and transform large data volumes.
• Architect modern data lakehouse models using Iceberg on CDP and AWS.
• Automate workflows with Airflow and integrate into CI/CD pipelines.
• Collaborate with architects, analysts, and cloud teams end-to-end.

Skills

PySpark

CDP

Iceberg

Airflow

SQL

Data Modeling

ETL/ELT

Git/CI/CD

Education

Bachelor's or equivalent in CS/IT/Engineering

Tools

AWS

Kafka

Cloudera Data Platform

S3

Glue

EMR

Lambda

Job description

Resource Informatics Group, Inc is seeking a senior Data Engineer to lead the hands-on creation and optimization of heavy-duty data pipelines for a large cloud transformation project. You will move enterprise data into a modern cloud platform while optimizing performance, data quality, and automation.

Responsibilities include building high-performance PySpark pipelines, architecting data lakehouse models with Iceberg on CDP and AWS, and automating workflows with Airflow integrated into CI/CD