AWS Data Engineer

VDart; Atlanta (GA); On-site; USD 100,000 - 150,000; Full time

Job summary

VDart is seeking a skilled Data Engineer to build and maintain ETL pipelines using Python and PySpark on AWS Glue and related platforms. You will design data flows, optimize queries, and ensure quality and reliability across production data assets.

Responsibilities include orchestrating workflows with AWS Step Functions, implementing event-driven patterns with SNS/SQS, and contributing to CI/CD and IaC for reproducible deployments.

Qualifications

• Strong experience with Python and PySpark for large-scale data processing.
• Proven hands-on experience with AWS services: Lambda, SNS, SQS, Glue, Redshift, Step Functions.
• Solid SQL skills and familiarity with data modeling and query optimization.
• Experience with ETL best practices, data quality checks, and monitoring/alerting.
• Familiarity with version control (Git) and basic DevOps/CI-CD workflows.

Responsibilities

• Build and maintain ETL pipelines using Python and PySpark on AWS Glue.
• Orchestrate workflows with AWS Step Functions and serverless components (Lambda).
• Implement messaging with AWS SNS and SQS; design event-driven patterns.
• Design and optimize data storage and querying in Amazon Redshift.
• Write performant SQL for data transformations, validation, and reporting.
• Ensure data quality, monitoring, error handling and ops support for pipelines.
• Collaborate with data consumers, engineers, and stakeholders to translate requirements into solutions.
• Contribute to CI/CD, infrastructure-as-code, and documentation for reproducible deployments.

Skills

Python

PySpark

SQL

Data modeling

ETL

CI/CD

Git

AWS Lambda

SQS

SNS

Tools

AWS Glue

Amazon Redshift

Step Functions

S3

CloudFormation

Job description

· Build and maintain ETL pipelines using Python and PySpark on AWS Glue and other compute platforms

· Orchestrate workflows with AWS Step Functions and serverless components (Lambda)

· Implement messaging and event-driven patterns using AWS SNS and SQS

· Design and optimize data storage and querying in Amazon Redshift

· Write performant SQL for data transformations, validation, and reporting

· Ensure data quality, monitoring, error handling and operational support for pipelines

· Collaborate with data consumers, engineers, and stakeholders to translate requirements into solutions

· Contribute to CI/CD, infrastructure-as-code, and documentation for reproducible deployments

Required Skills

· Strong experience with Python and Pyspark for large-scale data processing

· Proven hands-on experience with AWS services: Lambda, SNS, SQS, Glue, Redshift, Step Functions

· Solid SQLSQL skills and familiarity with data modeling and query optimization

· Experience with ETL best practices, data quality checks, and monitoring/alerting

· Familiarity with version control (Git) and basic DevOps/CI-CD workflows