Data Engineer - Python, SQL, AWS

Compunnel, Inc.; Durham (NC); On-site; USD 95,000 - 120,000; Full time

Job summary

A technology solutions firm in Durham, NC is seeking an experienced Data Engineer to design scalable data pipelines and contribute to an enterprise-wide Data Lake on AWS. The role requires a strong background in Python, SQL, and AWS services, with responsibilities including building cloud infrastructure for data analytics, optimizing data models, and ensuring data integrity. The ideal candidate is passionate about data and enjoys working in a collaborative environment.

Qualifications

• Extensive experience with relational databases such as Oracle or Snowflake.
• Hands-on experience with AWS services including S3, Glue, Redshift, etc.
• Experience with ETL technologies such as Informatica or SnapLogic.

Responsibilities

• Design and implement scalable ETL/ELT pipelines using AWS services.
• Integrate structured and unstructured data into data lakes and warehouses.
• Monitor data pipelines and troubleshoot issues to ensure reliability.

Skills

Python

SQL

AWS

Data modeling

ETL/ELT processes

Agile methodologies

Education

Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field

Tools

AWS Glue

Terraform

Apache Airflow

Informatica

Job description

We are seeking an experienced Data Engineer with a strong background in Python, SQL, and AWS to join our team.

This role involves designing and developing scalable data pipelines, performing data analysis and modeling, and contributing to the creation of an enterprise-wide Data Lake on AWS.

The ideal candidate will be passionate about data, enjoy working in collaborative environments, and have a strong desire to innovate and learn.

Key Responsibilities

• Design and implement scalable ETL/ELT pipelines using AWS Glue, Lambda, Step Functions, and other AWS services.
• Integrate structured and unstructured data from diverse sources into data lakes and warehouses (e.g., S3, Redshift, RDS, Athena).
• Build and maintain cloud infrastructure for data analytics platforms using Terraform, CloudFormation, or similar IaC tools.
• Collaborate with data engineers, data scientists, and analysts to deliver high-quality platforms for data loading, reporting, and machine learning.
• Optimize data models and queries for performance and scalability.
• Monitor data pipelines and troubleshoot issues to ensure reliability and data integrity.
• Implement CI/CD pipelines for data engineering workflows using GitLab, Bitbucket, Jenkins, or GitHub Actions.
• Ensure compliance with data governance and security best practices.
• Implement access controls and encryption for sensitive data.

Required Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• Extensive experience with relational databases such as Oracle or Snowflake.
• Experience in data warehousing, data modeling, and creation of data marts.
• Hands‑on experience with AWS services including S3, Glue, Lambda, Redshift, RDS, Athena, and Step Functions.
• Experience with ETL technologies such as Informatica or SnapLogic.
• Proficiency in SQL and PySpark.
• Familiarity with orchestration tools like Apache Airflow or MWAA.
• Understanding of DevOps tools and practices (CDK, CI/CD, Git, Terraform).
• Experience with Agile methodologies (Kanban and SCRUM).

Preferred Qualifications

• Experience with big data tools (Spark, Hive, Kafka).
• Knowledge of containerization (Docker, Kubernetes).
• Familiarity with data visualization tools (e.g., Power BI).
• AWS certifications (e.g., AWS Certified Data Analytics – Specialty).
• Experience with Business Intelligence and dashboard development.
• Exposure to DevOps, Continuous Integration, and Continuous Delivery tools (Maven, Jenkins, Stash, Ansible, Docker).