Data Engineer

Compunnel, Inc.; Charlotte (NC); On-site; USD 90,000 - 120,000; Full time

Job summary

A prominent tech company seeks a skilled Data Engineer to design and implement data integration solutions. The role demands proficiency in Python and PySpark, along with experience in AWS services. Responsibilities include developing ETL pipelines, collaborating with teams, and ensuring scalable data solutions. Ideal candidates should have strong SQL skills and experience with Agile methodologies.

Qualifications

• Hands-on expertise in Python and PySpark is a must.
• Experience with various AWS services required.
• Strong SQL programming skills, including performance tuning.

Responsibilities

• Design and implement data integration solutions.
• Develop and maintain ETL pipelines for data processing.
• Collaborate with cross-functional teams to deliver data solutions.

Skills

Python

PySpark

AWS Glue

AWS Lambda

SQL programming

UNIX scripting

REST APIs

Agile

Tools

Postgres

SQL Server

Oracle

DataStage

Informatica

Pentaho

Bitbucket

Jenkins

Job description

The Data Engineer plays a key role in designing and implementing data integration solutions that align with business objectives.

This role involves developing data pipelines, performing data manipulation and mining, and managing large-scale data processing using Python, PySpark, and AWS services.

The engineer collaborates with cross-functional teams to deliver scalable and reliable data solutions that support enterprise-wide initiatives.

Key Responsibilities

• Provide scoping, estimation, planning, design, development, and support services for data projects.
• Create detailed technical design documentation.
• Design, configure, deploy, and maintain custom ETL infrastructure.
• Develop and maintain ETL pipelines for data processing, transformation, and loading into large data domains.
• Collaborate with business analysts to understand and prioritize user requirements.
• Develop, test, and implement application code following software development lifecycle standards.
• Conduct quality analysis, defect tracking, and classification.
• Monitor project progress and resolve or escalation issues as needed.
• Coordinate with vendors and contractors for specific systems or projects.
• Stay updated on industry best practices and emerging technologies.

Required Qualifications

• Hands‑on expertise in Python and PySpark.
• Experience with AWS services such as Glue, Lambda, MSK (Kafka), S3, Step Functions, RDS, EKS.
• Proficiency in relational databases including Postgres, SQL Server, Oracle, Sybase.
• Strong SQL programming skills including performance tuning, stored procedures, views, and triggers.
• Experience in UNIX scripting and Oracle SQL/PL‑SQL.
• Familiarity with REST APIs and workload automation tools like Control‑M or Autosys.
• Knowledge of CI/CD tools such as Bitbucket, GitHub, Jenkins.
• Experience with Agile/SCRUM methodologies.

Preferred Qualifications

• Experience with other ETL tools such as DataStage, Informatica, or Pentaho.
• Knowledge of Master Data Management (MDM), Data Warehousing, and Data Analytics.