Senior AWS Data Engineer: Big Data Pipelines & Python

Tech Mirrors; Atlanta (GA); On-site; USD 120,000 - 180,000; Part time

Job summary

Tech Mirrors, Atlanta, seeks an AWS Data Engineer with 10+ years of Python development experience to build scalable data pipelines in a cloud environment. You will work with Hadoop/EMR, Spark, Hive, and AWS services including Lambda, EventBridge, Step Functions, Redshift, S3, and Glue to process and analyze large datasets.

Ideal candidates have hands-on API integration, SQL querying, data transformation, and visualization skills, plus CI/CD experience with GitLab and Terraform.

Qualifications

• 10+ years hands-on Python development for big data applications.
• Experience with Hadoop/EMR, Spark, Hive for scalable data pipelines.
• AWS services: Lambda, EventBridge, Step Functions, Redshift, S3, Glue.
• API calls, SQL queries, data manipulation/transformation, analysis and visualization.
• CI/CD pipeline experience; GitLab and Terraform familiarity.

Responsibilities

• Develop scalable data processing pipelines using big data technologies.
• Analyze large datasets to derive meaningful insights.
• Collaborate with teams across a matrixed organization.
• Implement APIs, perform data manipulation and transformation tasks.
• Contribute to CI/CD processes and tooling.

Skills

Python

Big data

Spark

Hadoop

AWS

Lambda

EventBridge

Step Functions

Redshift

S3

Glue

SQL

Data analysis

API calls

CI/CD

GitLab

Terraform

Tools

GitLab

Terraform

EMR

Hive

Spark

Job description

Tech Mirrors, Atlanta, seeks an AWS Data Engineer with 10+ years of Python development experience to build scalable data pipelines in a cloud environment. You will work with Hadoop/EMR, Spark, Hive, and AWS services including Lambda, EventBridge, Step Functions, Redshift, S3, and Glue to process and analyze large datasets.

Ideal candidates have hands-on API integration, SQL querying, data transformation, and visualization skills, plus CI/CD experience with GitLab and Terraform.