Senior Big Data & AWS Platform Engineer

TechDigital Group; St. Louis (MO); On-site; USD 150,000 - 190,000; Full time

Job summary

TechDigital Group is seeking a Senior Big Data / AWS Administrator to lead hands-on administration of large-scale Hadoop and Kafka environments in a cloud-first setup. The role emphasizes reliability, automation, and scalable data platform operations across AWS, with expertise in Terraform/CloudFormation and CI/CD pipelines.

The ideal candidate brings 10+ years in system administration, deep AWS knowledge, and strong problem-solving skills to ensure high availability and peak performance of

Qualifications

• 10+ years in system administration with 5+ years in big data and cloud environments.
• Proven track record managing large Hadoop clusters and Kafka environments.
• Strong hands-on AWS experience and infrastructure as code expertise.
• Agentic AI experience is a plus.

Responsibilities

• Design, deploy, and manage scalable, secure AWS infrastructure.
• Administer AWS services: EC2, S3, EMR, EKS, IAM, VPC, CloudWatch, Lambda.
• Install, tune, and maintain Hadoop components (HDFS, YARN, Hive, HBase, Spark).
• Install, configure, and manage Apache Kafka clusters and topics.
• Build monitoring, logging, and alerting with Prometheus, Grafana, or Confluent Center.
• Develop CI/CD pipelines for infra and platform deployments.
• Automate tasks with Python/Shell; collaborate with DevOps.

Skills

Cloud architecture

Hadoop admin

Kafka administration

Automation & IaC

CI/CD pipelines

Monitoring & incident response

Performance tuning

Scripting (Python, Shell)

Education

Bachelor's in CS or related

Tools

AWS

EC2

S3

EMR

EKS

IAM

VPC

CloudWatch

Lambda

Hadoop

Hive

Spark

HDFS

YARN

Kafka

Terraform

CloudFormation

Job description

Role Overview
We are seeking a highly experienced Senior Big Data / AWS Administrator with 10+ years of hands-on experience in managing large-scale distributed systems. The ideal candidate will have deep expertise in AWS cloud infrastructure, Hadoop ecosystem, and Kafka administration, ensuring high availability, scalability, and performance of enterprise data platforms.
________________________________________
Key Responsibilities
Cloud & Infrastructure (AWS)
• Design, deploy, and manage scalable, secure, and highly available infrastructure on AWS.
• Administer core AWS services: EC2, S3, EMR, EKS, IAM, VPC, CloudWatch, Lambda.
• Implement automated infrastructure provisioning using Infrastructure as Code (Terraform/CloudFormation).
• Optimize cost, performance, and reliability of cloud environments.
• Develop disaster recovery and backup strategies.
Big Data Platform Administration (Hadoop)
• Install, configure, tune, and maintain Hadoop ecosystem components (HDFS, YARN, Hive, HBase, Spark).
• Monitor cluster health, performance, and capacity planning.
• Troubleshoot cluster issues and ensure high availability using HA configurations (NameNode HA, etc.).
• Manage data ingestion and processing workflows.
Streaming Platform Administration (Kafka)
• Install, configure, and manage Apache Kafka clusters.
• Ensure high throughput, low latency, and fault-tolerant streaming systems.
• Manage topics, partitions, brokers, and retention policies.
• Perform Kafka cluster scaling, upgrades, and performance tuning.
• Implement monitoring using tools like Prometheus, Grafana, or Confluent Control Center.
System Operations & Monitoring
• Implement monitoring, logging, and alerting for all systems.
• Conduct root cause analysis and performance tuning.
• Ensure system security, access controls, and compliance standards.
• Maintain detailed documentation and operational runbooks.
Automation & DevOps
• Build CI/CD pipelines for infrastructure and platform deployments.
• Automate routine administrative tasks using scripting (Python, Shell, etc.).
• Collaborate with DevOps and engineering teams for platform improvements.
________________________________________
Required Skills & Qualifications
Experience
• Minimum 10+ years of experience in system administration, with at least 5+ years in big data and cloud environments.
• Proven experience managing large-scale Hadoop clusters and Kafka environments.
• Strong hands-on experience with AWS cloud platform.
• Agentic AI experience

Technical Skills
• AWS services: AWS Administration, EC2, S3, EMR, IAM, VPC, CloudWatch, Lambda
• Big Data : Administration, Hadoop, HDFS, Hive, Spark,
• Kafka Administration