Data Engineer

Trantor; Bengaluru; Remote; INR 1,200,000 - 1,800,000; Full time

Job summary

Trantor is seeking a Senior Data Engineer with hands-on experience in HBase, EMR, Hadoop, and Spark to support a production data platform modernization and migration program.

You will migrate and modernize distributed data platforms, ensuring data integrity, performance, and operational stability. Real production experience with HBase/EMR is required, not just Spark or ETL knowledge.

Qualifications

• Minimum 5+ years in Data/Big Data Engineering.
• Hands-on production experience with HBase.
• Strong experience with Amazon EMR.
• Strong experience with Apache Spark.
• Strong understanding of Hadoop ecosystem.
• Experience with Scala and/or Spark programming.
• Experience with S3 and AWS data platforms.
• Hands-on migration/ modernization experience (HBase/EMR).
• Experience with production data migration and cutover activities.
• Experience validating migrated data (row/region counts, checksums, reconciliation).
• Troubleshooting distributed data platforms at cluster level.
• Understanding dependencies between HBase, Spark, Hadoop, and workloads.
• Production support and migration-related troubleshooting.

Responsibilities

• Design and execute migration and modernization activities for HBase, EMR, Hadoop, and Spark environments.
• Support production migration of HBase/EMR workloads across environments.
• Perform cluster-level troubleshooting and resolve performance, availability, and compatibility issues.
• Plan and execute HBase snapshots, restore, and migration activities.
• Validate migrated data using row/region counts, checksums, reconciliation, and behavior checks.
• Support HBase data movement using Snapshot/ExportSnapshot/HFile mechanisms.
• Analyze dependencies between HBase, Spark, Hadoop, apps and platforms.
• Support EMR upgrades and platform modernization.
• Participate in production cutovers, maintenance windows, rollback planning, and post-cutover validation.
• Troubleshoot migration issues involving data integrity and connectivity.
• Collaborate with Backend, Cloud, Architect, DevSecOps, Security, and QA teams on cross-platform issues.
• Document migration procedures and results; contribute to automation.

Skills

Data Engineering

HBase

Amazon EMR

Apache Spark

Hadoop ecosystem

Scala

S3/AWS

Migration/ modernization

Data validation

Cluster troubleshooting

Dependency management

Production support

Job description

Location: Remote

Shift : 3pm onwards

Summary of the Role

We are looking for a Data Engineer with strong hands-on experience in HBase, Amazon EMR, Hadoop, and Spark to support a production data platform modernization and migration program.

The role will focus on migrating and modernizing distributed data platforms while maintaining application compatibility, data integrity, performance, and operational stability. The ideal candidate must have real production experience operating and migrating HBase/EMR environments, not only experience with Spark, Databricks, or ETL development.

The engineer will participate in migration planning, cluster-level troubleshooting, data validation, snapshot/restore activities, cutover, rollback, and post-migration verification.

Responsibilities

• Design and execute migration and modernization activities for HBase, EMR, Hadoop, and Spark environments.
• Support production migration of HBase/EMR workloads across environments.
• Perform cluster-level troubleshooting and resolve performance, availability, and compatibility issues.
• Plan and execute HBase snapshots, restore, and migration activities.
• Validate migrated data using row/region counts, checksums, reconciliation, application behavior, and other appropriate validation methods.
• Support HBase data movement using relevant migration mechanisms such as Snapshot/ExportSnapshot/HFile.
• Analyze dependencies between HBase, Spark, Hadoop, applications, and supporting platform components.
• Support EMR version upgrades and platform modernization.
• Participate in production cutovers, maintenance windows, rollback planning, and post-cutover validation.
• Troubleshoot migration issues involving data integrity, cluster configuration, application connectivity, performance, and compatibility.
• Work with Backend, Cloud, Architect, DevSecOps, Security, and QA teams to resolve cross-platform dependencies.
• Support data validation and reconciliation between legacy and target environments.
• Document migration procedures, validation results, issues, resolutions, and operational runbooks.
• Contribute to automation and repeatable processes for migration and validation activities.

Must-Have Skills

• 5+ years of experience in Data Engineering / Big Data Engineering.
• Strong hands-on production experience with HBase.
• Strong experience with Amazon EMR.
• Strong experience with Apache Spark.
• Strong understanding of Hadoop ecosystem.
• Experience with Scala and/or Spark programming.
• Experience with S3 and AWS-based data platforms.
• Hands-on experience with HBase/EMR migration or modernization.
• Experience with production data migration and cutover activities.
• Experience validating migrated data through row/region counts, reconciliation, checksums, or equivalent validation techniques.
• Experience troubleshooting distributed data platforms at the cluster level.
• Strong understanding of dependencies between HBase, Spark, Hadoop, and application workloads.
• Experience with production support and migration-related troubleshooting.

Good-to-Have Skills

• Experience with HBase 1.x and HBase 2.x.
• Experience with EMR 5.x and EMR 7.x.
• Experience with HBase Snapshot and ExportSnapshot.
• Experience working with HFiles and Coprocessors.
• Experience with Apache Phoenix.
• Experience with Hive, Presto, and/or Iceberg.
• Experience with Kerberos in Hadoop/HBase/EMR environments.
• Experience with KMS and encrypted data platforms.
• Experience with data integrity validation using checksums.
• Experience with rollback and snapshot/restore strategies.