Senior Data Engineer: Informatica & PySpark Leader

Pittsburgh Tomorrow; Pittsburgh (Allegheny County); On-site; USD 120,000 - 180,000; Full time

Job summary

Pittsburgh Tomorrow seeks a Senior Data Engineer to design and optimize ETL pipelines using Informatica PowerCenter/IDQ and PySpark, across large data platforms in a fast-paced environment.

You will work with Hadoop ecosystems (HDFS, Hive, Sqoop, MapReduce), and leverage Python and Kafka for streaming pipelines, while aligning with Agile processes and client-facing leadership across the SDLC.

Qualifications

• 8–10 years in Data Engineering and Data Analysis.
• Strong hands-on experience in Informatica PowerCenter/IDQ for ETL design, development, and optimization.
• Advanced PySpark development for large-scale data processing and analytics.
• Knowledge of Hadoop ecosystem (HDFS, Hive, Sqoop, MapReduce).
• Proficiency in Python and Kafka for streaming and batch pipelines.
• Understanding of data modeling and ETL workflows.
• Experience with Agile delivery and Jira for project management.
• Client-facing experience with strong communication and leadership to coordinate across SDLC.
• Preferred: AWS components, ML/AI concepts, Erwin data modeling.

Responsibilities

• Design, develop, and optimize ETL pipelines using Informatica and PySpark.
• Collaborate with data scientists and analytics teams across SDLC.
• Implement large-scale data processing on Hadoop ecosystem.
• Develop and enforce data quality checks and metadata handling.
• Lead agile delivery and coordinate via Jira.

Skills

Data Engineering

Data Analysis

Python

PySpark

Kafka

Agile / Jira

Client Liaison

Education

Master’s or Bachelor’s degree in Computer Science or related field

Tools

Informatica PowerCenter

Informatica IDQ

Hadoop

Hive

Sqoop

MapReduce

Erwin

Job description

Pittsburgh Tomorrow seeks a Senior Data Engineer to design and optimize ETL pipelines using Informatica PowerCenter/IDQ and PySpark, across large data platforms in a fast-paced environment.

You will work with Hadoop ecosystems (HDFS, Hive, Sqoop, MapReduce), and leverage Python and Kafka for streaming pipelines, while aligning with Agile processes and client-facing leadership across the SDLC.