Cloud-Native Big Data Platform Engineer

Compunnel, Inc.; Rockville (MD); On-site; USD 140,000 - 190,000; Full time

Job summary

Compunnel, Inc. seeks a seasoned Big Data Platform Engineer to design, develop, and optimize large-scale data processing platforms in a cloud-native environment. You will build scalable data pipelines and architect Kubernetes infra to run Spark workloads on AWS, including EMR on EKS.

The role requires deep expertise in Spark, Kubernetes, Hadoop, Hive, Trino, and modern DevOps practices, with a focus on performance, cost efficiency, and reliability.

Qualifications

• Design and develop enterprise Big Data solutions on Kubernetes with Spark.
• Experience running Spark on Kubernetes and EMR on EKS.
• Strong AWS services and security in cloud environments.
• Proficient in Python or Scala for data processing.
• Experience with Hadoop, Hive, and Trino ecosystems.
• Proven ability to optimize data pipelines for scale and cost.

Responsibilities

• Design, develop, and maintain large-scale data pipelines using Spark, Hadoop, Python, and Scala.
• Architect, deploy, and optimize containerized big data workloads on EMR on EKS.
• Build Kubernetes infrastructure to support Spark apps and data platforms.
• Develop scalable data ingestion, storage, transformation, and analytics.
• Monitor, troubleshoot, and resolve production data pipelines and platforms.
• Create automated tests to ensure data quality and reliability.
• Collaborate with architects, data scientists, and engineers on enterprise solutions.
• Research emerging Big Data and cloud AI technologies to improve capabilities.

Skills

Kubernetes

Apache Spark

Python

Scala

AWS

Data pipelines

SQL

Performance tuning

Education

Bachelor's degree in CS/IS

Master's degree (preferred)

Tools

Docker

Terraform

CloudFormation

GitHub Actions

ArgoCD

Istio

Prometheus

Grafana

Job description

Compunnel, Inc. seeks a seasoned Big Data Platform Engineer to design, develop, and optimize large-scale data processing platforms in a cloud-native environment. You will build scalable data pipelines and architect Kubernetes infra to run Spark workloads on AWS, including EMR on EKS.

The role requires deep expertise in Spark, Kubernetes, Hadoop, Hive, Trino, and modern DevOps practices, with a focus on performance, cost efficiency, and reliability.