Senior AWS Data Engineer - ETL & Big Data

Valiant Technologies LLC; Chicago (IL); On-site; USD 120,000 - 180,000; Full time

Job summary

Valiant Technologies LLC in Chicago seeks an experienced Senior ETL Developer with hands-on AWS, Python, PySpark, and SQL. You will design, develop, and optimize enterprise-scale ETL pipelines and data analytics solutions within the AWS ecosystem.

Design, develop, implement, and optimize ETL pipelines using AWS Glue, Python, PySpark, and SQL; work with S3, Redshift, Lambda, and Step Functions. Collaborate with architects and stakeholders to deliver scalable data solutions.

Qualifications

• 6+ years in ETL development, data integration, transformation and scheduling.
• Strong SQL skills with complex query optimization.
• 3–4 years Python / PySpark with large-scale datasets.
• 3–4 years AWS experience (Glue, S3, Redshift, Lambda, Step Functions, RDS).
• 3+ years Big Data processing and scalable data architectures.
• Hands-on with Harness, YAML, and CloudFormation for CI/CD/Infra as code.
• Strong database troubleshooting and production support.
• Experience leading ETL or Data Engineering teams (preferred).
• Excellent analytical and communication skills.

Responsibilities

• Design, develop and optimize enterprise-scale ETL pipelines.
• Build scalable data pipelines using AWS Glue, Python, PySpark and SQL.
• Develop complex SQL queries and data transformation workflows.
• Work with AWS services including S3, Redshift, Lambda, Step Functions, RDS.
• Perform data cleansing, profiling, transformation and scheduling.
• Troubleshoot ETL failures and production incidents.
• Implement CI/CD and IaC with Harness, YAML and CloudFormation.
• Collaborate with architects, developers and stakeholders.
• Provide root cause analysis and production support.
• Maintain coding standards and ETL documentation.

Skills

ETL Development

SQL

Python / PySpark

Big Data

AWS Cloud

DevOps / CI/CD

Data Modeling

Leadership

Communication

Troubleshooting

Tools

AWS Glue

S3

Redshift

Lambda

Step Functions

RDS Aurora/MySQL

CloudWatch

SNS

SQS

EventBridge

Apache Iceberg

Dynatrace

Datadog

Harness

YAML

AWS CloudFormation

Job description

Valiant Technologies LLC in Chicago seeks an experienced Senior ETL Developer with hands-on AWS, Python, PySpark, and SQL. You will design, develop, and optimize enterprise-scale ETL pipelines and data analytics solutions within the AWS ecosystem.

Design, develop, implement, and optimize ETL pipelines using AWS Glue, Python, PySpark, and SQL; work with S3, Redshift, Lambda, and Step Functions. Collaborate with architects and stakeholders to deliver scalable data solutions.