Data Pipeline Engineer II: ETL, Airflow & Spark

PTR Global; West Des Moines (IA); On-site; USD 55,104 - 61,992; Part time

Job summary

A leading technology provider seeks a Software Engineer 2 in West Des Moines, Iowa. This contingent role involves designing and developing ETL/ELT workflows and optimizing data structures for performance. Candidates should have around 2 years of Software Engineering experience, strong skills in data pipelines, and proficiency in tools like Apache Airflow and PySpark. The role offers a pay range of $40 to $45 per hour.

Qualifications

• 2 years of Software Engineering experience or equivalent.
• Strong understanding of data structures and lifecycle management.
• Hands-on experience designing and managing ETL/ELT data pipelines.
• Advanced skills in distributed data processing with PySpark.
• Knowledge of Hadoop Ecosystem components.
• Programming expertise in Python, Spark, and SQL.

Responsibilities

• Design and develop ETL/ELT workflows for data processing.
• Build and maintain data pipelines using open-source frameworks.
• Implement operational data stores leveraging Delta Lake.
• Collaborate with teams to align with architecture.
• Troubleshoot data pipeline issues and ensure reliability.

Skills

Software Engineering experience

Data structures understanding

ETL/ELT data pipelines

Distributed data processing with PySpark

Cloud platforms (GCP)

Apache Airflow

Python

Education

2 years of relevant experience

Tools

Apache Airflow

PySpark

SQL

NetApp Iceberg

Hadoop Ecosystem (HDFS, Hive)

Job description

A leading technology provider seeks a Software Engineer 2 in West Des Moines, Iowa. This contingent role involves designing and developing ETL/ELT workflows and optimizing data structures for performance. Candidates should have around 2 years of Software Engineering experience, strong skills in data pipelines, and proficiency in tools like Apache Airflow and PySpark. The role offers a pay range of $40 to $45 per hour.