Big Data Engineer

HNM Systems; Hyderabad; Hybrid; INR 1,200,000 - 1,800,000; Full time

Job summary

HNM Systems, Inc. in Hyderabad, India is looking for a hands-on Big Data Engineer to design, build, and scale production-grade data platforms on AWS.

This role demands taking technical ownership while staying deeply involved in development, architecture, code reviews, and delivery. You will design scalable ETL pipelines with Python and Spark, build large S3 data lakes and lakehouse architectures, and lead architecture decisions with engineering teams.

Qualifications

• Expert-level Python and strong SQL.
• Deep hands-on experience with Apache Spark/PySpark, including Spark internals and performance tuning.
• Strong AWS experience, particularly Glue, EMR, S3, and Step Functions.
• Experience with Parquet, ORC, Avro, Iceberg, Hudi, and Delta Lake.
• Strong understanding of dimensional modeling, lakehouse, and medallion architectures.
• Experience with Terraform, AWS CDK, CloudFormation, CI/CD, Git, and Docker.
• Proven ability to deliver and support complex, production-grade data solutions.

Responsibilities

• Design and build scalable ETL and data pipelines using Python, Spark, AWS Glue, and Amazon EMR.
• Develop large-scale S3 data lakes and lakehouse architectures supporting batch and CDC ingestion.
• Build solutions using Step Functions, Athena, Redshift/Spectrum, Lambda, SQS, SNS, Kinesis, and MSK.
• Lead technical design, architecture, and code reviews while partnering closely with engineering teams.
• Drive production readiness across reliability, observability, data quality, performance, and cost optimization.
• Optimize distributed data workloads operating at multi-terabyte scale.

Skills

Python

SQL

Spark/PySpark

AWS Glue

AWS EMR

S3

Step Functions

Parquet/ORC/Avro

Iceberg/Hudi/Delta

Lakehouse architecture

Terraform

CloudFormation

CI/CD

Git

Docker

Dimensional modeling

Education

Bachelor's degree in CS/Engineering

Tools

AWS CDK

Job description

We are looking for Big Data Engineer's to help design, build, and scale production-grade data platforms on AWS. This is a highly hands-on engineering role for someone who can lead technical decisions while remaining deeply involved in development, architecture, code reviews, and production delivery.

What Youll Do

• Design and build scalable ETL and data pipelines using Python, Spark, AWS Glue, and Amazon EMR.
• Develop large-scale S3 data lakes and lakehouse architectures supporting batch and CDC ingestion.
• Build solutions using Step Functions, Athena, Redshift/Spectrum, Lambda, SQS, SNS, Kinesis, and MSK.
• Lead technical design, architecture, and code reviews while partnering closely with engineering teams.
• Drive production readiness across reliability, observability, data quality, performance, and cost optimization.
• Optimize distributed data workloads operating at multi-terabyte scale.

What We're Looking For

• Expert-level Python and strong SQL.
• Deep hands-on experience with Apache Spark/PySpark, including Spark internals and performance tuning.
• Strong AWS experience, particularly Glue, EMR, S3, and Step Functions.
• Experience with Parquet, ORC, Avro, Iceberg, Hudi, and Delta Lake.
• Strong understanding of dimensional modeling, lakehouse, and medallion architectures.
• Experience with Terraform, AWS CDK, CloudFormation, CI/CD, Git, and Docker.
• Proven ability to deliver and support complex, production-grade data solutions.

Preferred

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• AWS certifications.
• Experience leading architecture decisions and mentoring engineering teams.