Senior Data Engineer (Informatica & PySpark)

ThoughtStorm Inc.; Cleveland (OH); On-site; USD 110,000 - 170,000; Full time

Job summary

ThoughtStorm Inc. is seeking an experienced data engineer to drive data pipelines, analytics, and ETL architectures. You will design and optimize large-scale data processes, working across teams to deliver reliable data products.

You will leverage Informatica PowerCenter/IDQ, PySpark, and Hadoop ecosystem tools, collaborating with stakeholders and applying agile practices. Experience with AWS data services and ML concepts is advantageous.

Qualifications

• 8–10 years in data engineering and data analysis.
• Strong hands-on Informatica PowerCenter/IDQ for ETL design, development, and optimization.
• Advanced PySpark development for large-scale processing, transformation, and analytics.
• Solid Hadoop ecosystem knowledge (HDFS, Hive, Sqoop, MapReduce).
• Proficiency in Python and Kafka for streaming and batch pipelines.
• Well-versed in data modeling, ETL workflows, data quality checks, and loading.
• Experience with Agile delivery and Jira for project coordination.
• Client-facing roles with clear communication and leadership across SDLC.
• Exposure to AWS data components and analytics; familiarity with ML concepts.
• Experience with Erwin as a data modeling tool.

Skills

Data Engineering

ETL Design

Big Data Analytics

Python

Kafka

PySpark

Communication

Agile

Client Interaction

Problem Solving

Education

Master's or Bachelor's in CS

Tools

Informatica PowerCenter/IDQ

Hadoop (HDFS, Hive, Sqoop, MapReduce)

Erwin

PySpark

Kafka

Jira

AWS data services

Job description

• Experience: 8–10 years in Data Engineering and Data Analysis.
• Informatica Expertise: Strong hands-on experience in Informatica PowerCenter/IDQ for ETL design, development, and optimization.
• PySpark Development: Advanced skills in PySpark for large-scale data processing, transformation, and analytics.
• Hadoop Ecosystem: Solid working knowledge of Hadoop technologies (HDFS, Hive, Sqoop, MapReduce).
• Programming Skills: Proficiency in Python and Kafka for streaming and batch data pipelines.
• Database & Modeling: Strong understanding of database concepts, data design, data modeling, and ETL workflows.
• ETL Lifecycle: Experience in analyzing, designing, and coding ETL programs including data extraction, ingestion, quality checks, normalization, and loading.
• Agile Delivery: Hands-on experience with Agile methodology and Jira for project delivery.
• Client Interaction: Proven ability in client-facing roles with strong communication and leadership skills to coordinate across SDLC.
• Exposure to AWS data components and analytics.
• Familiarity with machine learning models and AI concepts.
• Experience with data modeling tools such as Erwin.

Qualifications

• Master’s or Bachelor’s degree in Computer Science or related field.
• Strong problem-solving skills and ability to work in cross-functional teams.