Senior Big Data Engineer – AWS, Hadoop & Spark

Signature IT World Inc; Missouri; On-site; USD 120,000 - 180,000; Full time

Job summary

Signature IT World Inc. is seeking a Senior Big Data Engineer in the Maryland Heights, MO area to lead design and optimization of data pipelines and lakehouse platforms leveraging AWS, Hadoop, Spark, Hive, and Iceberg. The role focuses on real-time and batch processing, data governance, and cloud-native migrations.

Responsibilities include building scalable pipelines, managing a data lakehouse, implementing CDC, and ensuring regulatory compliance while tuning Spark and JVM performance.

Qualifications

• Strong experience in AWS, Hadoop, Spark, and enterprise data lakehouse architectures.
• Hands-on with batch and real-time data pipelines using Spark, Java, and Kafka.
• Experience migrating Hadoop workloads to AWS cloud-native architectures.

Responsibilities

• Design, develop, and optimize batch and real-time data pipelines using Spark, Java, and Kafka.
• Manage data lakehouse platforms using AWS S3, Cloudera, Hive, and Apache Iceberg.
• Implement CDC and develop complex SQL queries to integrate relational databases into dimensional data warehouses.
• Automate data workflows using Python and Unix shell scripting.
• Migrate legacy Hadoop workloads to AWS cloud-native architectures.
• Monitor and tune Spark jobs and JVM performance.
• Ensure data governance, security, and regulatory compliance.

Skills

AWS

Hadoop

Spark

Java

Cloudera

Hive

Iceberg

S3

SQL

CDC

Oracle

SQL Server

Hadoop-to-AWS migration

Kafka

Tools

Kafka

Job description

Location: Maryland Heights, MO 63043

Type: Contract

Must-Have Skills:

• AWS and Hadoop
• Apache Spark (Scala/Python) and Core Java
• Cloudera, Hive, Iceberg, and S3
• SQL, CDC, Oracle, and SQL Server
• Hadoop-to-AWS migration

Job Description:

We are looking for a Senior Big Data Engineer with strong experience in AWS, Hadoop, Spark, and enterprise data lakehouse architectures.

Key Responsibilities:

• Design, develop, and optimize batch and real-time data pipelines using Spark, Java, and Kafka.
• Manage data lakehouse platforms using AWS S3, Cloudera, Hive, and Apache Iceberg.
• Implement CDC and develop complex SQL queries to integrate relational databases into dimensional data warehouses.
• Automate data workflows using Python and Unix shell scripting.
• Migrate legacy Hadoop workloads to AWS cloud-native architectures.
• Monitor and tune Spark jobs and JVM performance.
• Ensure data governance, security, and regulatory compliance.

Preferred Qualifications:

• Experience in the Telecom/Communications domain, including CDRs and network telemetry data.
• Strong understanding of data warehouse modeling and data governance.