Senior Big Data Engineer: Spark, Hadoop & Cloud Pipelines

Compunnel, Inc.; Tysons (VA); On-site; USD 150,000 - 190,000; Full time

Job summary

Compunnel, Inc. is seeking a Big Data Engineer to design, develop, and optimize large-scale data processing systems at petabyte scale.

You will build scalable data pipelines using Spark, PySpark, Hadoop, Hive, Trino, and AWS services, collaborating with data scientists and analysts to translate business needs into robust solutions. The role emphasizes performance tuning, reliability, automation, and data quality, with opportunities to apply AI tooling and modern development practices in a

Qualifications

• Bachelor's degree with 5+ years of related experience.
• Strong hands-on experience with Big Data technologies including Hadoop and Apache Spark.
• Experience with Spark development, internals, and performance tuning.
• Experience with AWS services such as S3, EMR, Glue, Lambda, and Athena.
• Experience developing scalable data processing solutions using Python, Scala, or Java.
• Experience with test automation, build automation, and configuration management practices.

Responsibilities

• Design, develop, and maintain large-scale data processing pipelines using Hadoop, Spark, Python, and Scala.
• Implement scalable and reliable data ingestion, storage, transformation, and analysis solutions.
• Develop and optimize data pipelines designed to process massive data volumes at petabyte scale.
• Optimize Spark applications through partitioning, caching, broadcast joins, and other performance tuning techniques.
• Analyze and troubleshoot Spark architecture components including executors, tasks, stages, and DAGs.
• Work with AWS services including S3, EMR, Glue, Lambda, and Athena to build and support cloud-based data solutions.
• Develop scalable Python, PySpark, or Scala-based data processing applications.
• Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
• Develop automated testing frameworks and implement continuous testing for data quality assurance.

Skills

Big Data tech

Spark

Python

Scala

Java

SQL

AWS

Agile

CI/CD

Testing

AI tools

Education

Bachelor's degree in Computer Science / related field

Tools

Hadoop

Apache Spark

Hive

Trino

Python

Scala

Job description

Compunnel, Inc. is seeking a Big Data Engineer to design, develop, and optimize large-scale data processing systems at petabyte scale.

You will build scalable data pipelines using Spark, PySpark, Hadoop, Hive, Trino, and AWS services, collaborating with data scientists and analysts to translate business needs into robust solutions. The role emphasizes performance tuning, reliability, automation, and data quality, with opportunities to apply AI tooling and modern development practices in a