Data Engineer - iGrid

Primary Talent Partners; Charlotte (NC); On-site; USD 121,000 - 135,000; Full time

Job summary

Primary Talent Partners is seeking a Data Engineer - iGrid for a 12-month contract in Charlotte, NC. This role focuses on designing and maintaining scalable ETL pipelines on AWS, real-time data streaming with Confluent Kafka, and PySpark development on EMR clusters.

You will build serverless workflows with Lambda and optimize Athena queries for analytics. The candidate will work on a W2 contract with no PTO or benefits; ACA-compliant supplemental package is available.

Qualifications

• Bachelor's degree or higher in Computer Science, Data Engineering, IT, or related field.
• Strong background in distributed computing and big data processing frameworks.
• Experience with Agile development methodologies.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration abilities.

Responsibilities

• Design, develop, and maintain scalable ETL pipelines using AWS Glue, EMR (PySpark), and Lambda functions.
• Implement real-time data streaming solutions using Confluent Kafka with source and sink connectors.
• Develop and optimize PySpark jobs on EMR clusters for large-scale data transformations.
• Build serverless data processing workflows using AWS Lambda.
• Design and optimize queries in Amazon Athena for analytical workloads.
• Manage data warehouse solutions in Amazon Redshift including data modeling and performance tuning.
• Implement monitoring and alerting using CloudWatch for data pipeline health and performance.
• Provision and manage AWS infrastructure using Terraform.
• Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements.
• Implement data governance and security best practices using Lake Formation, IAM, and KMS.

Skills

Python

PySpark

AWS Lambda

AWS Glue

Amazon EMR

Amazon Redshift

Kafka

CloudWatch

Terraform

Data modeling

ETL pipelines

Education

Bachelor's degree in Computer Science or related field

Tools

VS Code

Git

DBeaver

Data Studio

AWS CLI

boto3

Terraform

EMR Notebooks

Jupyter

Job description

Primary Talent Partners has a new contract opening for a Data Engineer - iGrid with our large power and utilities client in Charlotte, NC. This is a 12-month contract with a potential for extension.

Pay: $88.00 - $98.00/hr; W2 contract, no PTO, no Benefits. ACA-compliant supplemental package available for enrollment. Candidates must be legally authorized to work in the United States and must be able to sit on Primary Talent Partners W2 without sponsorship.

Description

Key Responsibilities

• Design, develop, and maintain scalable ETL pipelines using AWS Glue, EMR (PySpark), and Lambda functions
• Implement real-time data streaming solutions using Confluent Kafka with source and sink connectors
• Develop and optimize PySpark jobs on EMR clusters for large-scale data transformations
• Build serverless data processing workflows using AWS Lambda
• Design and optimize queries in Amazon Athena for analytical workloads
• Manage data warehouse solutions in Amazon Redshift including data modeling and performance tuning
• Implement monitoring and alerting using CloudWatch for data pipeline health and performance
• Provision and manage AWS infrastructure using Terraform
• Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements
• Implement data governance and security best practices using Lake Formation, IAM, and KMS

Additional Technical Experience

• Knowledge of AWS EventBridge and Step Functions for workflow orchestration
• Familiarity with EMR runtime roles and execution roles for secure data access
• Experience with DynamoDB for NoSQL data storage
• Knowledge of API Gateway for building data APIs

Preferred Qualifications

• Bachelor's degree in Computer Science, Data Engineering, Information Technology, or related field
• Strong understanding of distributed computing and big data processing frameworks
• Experience with Agile development methodologies
• Excellent problem-solving and analytical skills
• Strong communication and collaboration abilities

Development and Data Tools

• EMR Notebooks or Jupyter for interactive development
• VS Code or similar IDE for Python development
• Git for version control
• AWS CLI and boto3 for AWS service interaction
• Terraform for infrastructure provisioning
• DBeaver, Data Studio for database management

Required Technical Skills

• Python: Advanced proficiency in Python 3.8+ for data processing, scripting, and automation
• PySpark: Extensive experience with PySpark for large-scale data processing and transformations on EMR clusters
• AWS Lambda: Hands-on experience building serverless functions for data processing and orchestration
• AWS Glue: Expertise in AWS Glue for ETL jobs, Data Catalog management.
• Amazon Athena: Experience with Athena for SQL-based data analysis and querying data lakes
• Amazon EMR: Strong experience with EMR for running Spark jobs, EMR Notebooks, and cluster management
• Amazon Redshift: Knowledge of Redshift for data warehousing, query optimization, and integration with data lakes
• Kafka: Experience with Confluent Kafka for real-time data streaming, including Schema Registry, source/sink connectors, and producers/consumers
• Amazon S3: Deep understanding of S3 for data lake architecture, bucket policies, and data organization (raw, curated layers)
• CloudWatch: Proficiency in CloudWatch for logging, monitoring, and alerting on data pipelines
• Terraform: Experience with Infrastructure as Code using Terraform for provisioning AWS resources

Primary Talent Partners is an Equal Opportunity / Aff… All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, age, national origin, disability, protected veteran status, gender identity, or any other factor protected by applicable federal, state, or local laws. If you are a person with a disability needing assistance with the application or at any point in the hiring process, please contact us at [e-mail]