Data Engineer - AWS ETL & Real-Time Pipelines

Primary Talent Partners, LLC; Charlotte (NC); On-site; USD 121,000 - 135,000; Full time

Job summary

Primary Talent Partners is seeking a Data Engineer - iGrid for a 12-month contract in Charlotte, NC. This role focuses on designing and maintaining scalable ETL pipelines on AWS, real-time data streaming with Confluent Kafka, and PySpark development on EMR clusters.

You will build serverless workflows with Lambda and optimize Athena queries for analytics. The candidate will work on a W2 contract with no PTO or benefits; ACA-compliant supplemental package is available.

Qualifications

• Bachelor's degree or higher in Computer Science, Data Engineering, IT, or related field.
• Strong background in distributed computing and big data processing frameworks.
• Experience with Agile development methodologies.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration abilities.

Responsibilities

• Design, develop, and maintain scalable ETL pipelines using AWS Glue, EMR (PySpark), and Lambda functions.
• Implement real-time data streaming solutions using Confluent Kafka with source and sink connectors.
• Develop and optimize PySpark jobs on EMR clusters for large-scale data transformations.
• Build serverless data processing workflows using AWS Lambda.
• Design and optimize queries in Amazon Athena for analytical workloads.
• Manage data warehouse solutions in Amazon Redshift including data modeling and performance tuning.
• Implement monitoring and alerting using CloudWatch for data pipeline health and performance.
• Provision and manage AWS infrastructure using Terraform.
• Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements.
• Implement data governance and security best practices using Lake Formation, IAM, and KMS.

Skills

Python

PySpark

AWS Lambda

AWS Glue

Amazon EMR

Amazon Redshift

Kafka

CloudWatch

Terraform

Data modeling

ETL pipelines

Education

Bachelor's degree in Computer Science or related field

Tools

VS Code

Git

DBeaver

Data Studio

AWS CLI

boto3

Terraform

EMR Notebooks

Jupyter

Job description

Primary Talent Partners is seeking a Data Engineer - iGrid for a 12-month contract in Charlotte, NC. This role focuses on designing and maintaining scalable ETL pipelines on AWS, real-time data streaming with Confluent Kafka, and PySpark development on EMR clusters.

You will build serverless workflows with Lambda and optimize Athena queries for analytics. The candidate will work on a W2 contract with no PTO or benefits; ACA-compliant supplemental package is available.