Senior Data Engineer (Informatica & PySpark)

Life in Pittsburgh; Pittsburgh (Allegheny County); Presencial; USD 120.000 - 180.000; Jornada completa

Descripción de la vacante

Pittsburgh Tomorrow seeks a Senior Data Engineer to design and optimize ETL pipelines using Informatica PowerCenter/IDQ and PySpark, across large data platforms in a fast-paced environment.

You will work with Hadoop ecosystems (HDFS, Hive, Sqoop, MapReduce), and leverage Python and Kafka for streaming pipelines, while aligning with Agile processes and client-facing leadership across the SDLC.

Formación

• 8–10 years in Data Engineering and Data Analysis.
• Strong hands-on experience in Informatica PowerCenter/IDQ for ETL design, development, and optimization.
• Advanced PySpark development for large-scale data processing and analytics.
• Knowledge of Hadoop ecosystem (HDFS, Hive, Sqoop, MapReduce).
• Proficiency in Python and Kafka for streaming and batch pipelines.
• Understanding of data modeling and ETL workflows.
• Experience with Agile delivery and Jira for project management.
• Client-facing experience with strong communication and leadership to coordinate across SDLC.
• Preferred: AWS components, ML/AI concepts, Erwin data modeling.

Responsabilidades

• Design, develop, and optimize ETL pipelines using Informatica and PySpark.
• Collaborate with data scientists and analytics teams across SDLC.
• Implement large-scale data processing on Hadoop ecosystem.
• Develop and enforce data quality checks and metadata handling.
• Lead agile delivery and coordinate via Jira.

Conocimientos

Data Engineering

Data Analysis

Python

PySpark

Kafka

Agile / Jira

Client Liaison

Educación

Master’s or Bachelor’s degree in Computer Science or related field

Herramientas

Informatica PowerCenter

Informatica IDQ

Hadoop

Hive

Sqoop

MapReduce

Erwin

Descripción del empleo

Senior Data Engineer (Informatica & PySpark)

Position Title: Senior Data Engineer

• Experience: 8–10 years in Data Engineering and Data Analysis.
• Informatica Expertise: Strong hands‑on experience in Informatica PowerCenter/IDQ for ETL design, development, and optimization.
• PySpark Development: Advanced skills in PySpark for large‑scale data processing, transformation, and analytics.
• Hadoop Ecosystem: Solid working knowledge of Hadoop technologies (HDFS, Hive, Sqoop, MapReduce).
• Programming Skills: Proficiency in Python and Kafka for streaming and batch data pipelines.
• Database & Modeling: Strong understanding of database concepts, data design, data modeling, and ETL workflows.
• ETL Lifecycle: Experience in analyzing, designing, and coding ETL programs including data extraction, ingestion, quality checks, normalization, and loading.
• Agile Delivery: Hands‑on experience with Agile methodology and Jira for project delivery.
• Client Interaction: Proven ability in client‑facing roles with strong communication and leadership skills to coordinate across SDLC.
• Preferred Skills:
  • Exposure to AWS data components and analytics.
  • Familiarity with machine learning models and AI concepts.
  • Experience with data modeling tools such as Erwin.

Qualifications

• Master’s or Bachelor’s degree in Computer Science or related field.
• Strong problem-solving skills and ability to work in cross-functional teams.