Senior Python Big Data Engineer — AWS, Spark, EMR

Compunnel Inc.; Reston (VA); On-site; USD 140,000 - 180,000; Full time

Job summary

Compunnel, Inc. is seeking an experienced Python Big Data Engineer to design, develop, and optimize large-scale data processing solutions.

The role emphasizes Python, AWS cloud services, big data technologies, and best practices to build scalable data pipelines and analytics. You will design and maintain scalable applications with Python, build pipelines using Hadoop, EMR, Spark, and Hive, and develop cloud-native data solutions on AWS.

Qualifications

• 10+ years of Python development for big data apps.
• Extensive experience building pipelines with Hadoop, EMR, Spark, Hive.
• 5+ years AWS services: EMR, Lambda, EventBridge, Step Functions, Redshift, S3, Glue.
• 5+ years API integrations, SQL, Spark, data manipulation, transformation, analysis, visualization.
• CI/CD pipelines with GitLab, Terraform or similar tools.
• Strong analytical and cross-functional collaboration skills.

Responsibilities

• Design, develop, and maintain scalable big data applications using Python.
• Build and optimize data pipelines using Hadoop, EMR, Spark, and Hive.
• Develop cloud-native data solutions using AWS services.
• Design and implement data ingestion, transformation, and processing workflows.
• Develop and maintain integrations using APIs and SQL queries.
• Perform large-scale data manipulation, transformation, analysis, and visualization.
• Analyze large datasets to extract meaningful business insights.
• Design, implement, and support CI/CD pipelines for automated deployment.
• Collaborate with cross-functional teams to deliver scalable and reliable data solutions.
• Apply analytical and problem-solving skills to resolve complex technical challenges.

Skills

Python

Big Data

Hadoop

Spark

Hive

AWS (EMR, S3, Redshift, Glue)

API integrations

SQL

CI/CD

GitLab

Terraform

Tools

GitLab

Terraform

EMR

Job description

We are seeking an experienced Python Big Data Engineer with expertise in designing, developing, and optimizing large-scale data processing solutions. The ideal candidate will have extensive experience with Python, AWS cloud services, big data technologies, and data engineering best practices to build scalable, high-performance data pipelines and analytics solutions.

Key Responsibilities

• Design, develop, and maintain scalable big data applications using Python.

• Build and optimize data processing pipelines using Hadoop, EMR, Spark, and Hive.

• Develop and support cloud-native data solutions using AWS services.

• Design and implement data ingestion, transformation, and processing workflows.

• Develop and maintain integrations using APIs and SQL queries.

• Perform large-scale data manipulation, transformation, analysis, and visualization.

• Analyze large datasets to extract meaningful business insights.

• Design, implement, and support CI/CD pipelines for automated deployment.

• Collaborate with cross-functional teams to deliver scalable and reliable data solutions.

• Apply analytical and problem-solving skills to identify and resolve complex technical challenges.

Required Qualifications

• 10+ years of hands-on Python development experience for big data applications.

• Extensive experience implementing scalable data processing pipelines using Hadoop, Amazon EMR, Spark, and Hive.

• 5+ years of development experience with AWS services including EMR, Lambda, EventBridge, Step Functions, Redshift, Amazon S3, and AWS Glue.

• 5+ years of experience developing solutions involving API integrations, SQL queries, Spark, data manipulation, data transformation, analysis, and visualization.

• Strong experience analyzing large datasets and extracting meaningful insights.

• Hands-on experience implementing CI/CD pipelines using GitLab, Terraform, or similar tools.

• Strong analytical, critical thinking, and problem-solving skills.

• Ability to collaborate effectively across cross-functional and matrixed teams.

Preferred Qualifications

• Experience with infrastructure as code (IaC).

• Experience optimizing large-scale cloud-based data engineering solutions.