Lead BigData Engineer with (Hadoop +Databricks)_ Exp: 7+ Years

Atyeti Inc.; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

Atyeti Inc. is seeking a Data Engineer to design and execute Hadoop to Databricks migration strategies for large-scale data platforms. You will migrate HDFS, Hive, Spark, and MapReduce workloads to Databricks and optimize Spark code for performance and cost efficiency.

You will develop scalable ETL/ELT pipelines using Spark, SQL, and Databricks, leverage Delta Lake features, and implement CI/CD for notebooks and jobs while aligning with security and governance standards for banking environments.

Responsibilities

• Design and execute Hadoop to Databricks migration strategies for large-scale data platforms.
• Migrate existing HDFS, Hive, Spark, and MapReduce workloads to Databricks.
• Data Engineering Hadoop to Databricks migration experience
• Strong hands-on experience in migrating and optimizing Spark code
• Analyze existing Hadoop/Spark applications and identify opportunities for modernization and optimization.
• Strong hands-on experience in migrating, refactoring, and optimizing Spark code for Databricks.
• Develop scalable ETL/ELT pipelines using Spark, SQL, and Databricks.
• Optimize Spark jobs by improving partitioning, caching, joins, shuffling, serialization, and memory utilization.
• Work with Databricks Delta Lake / Delta tables for reliable and scalable data processing.
• Implement Delta Lake features such as ACID transactions, schema evolution, time travel, and optimized data layouts.
• Convert legacy Hive SQL / Spark SQL workloads to Databricks SQL and optimize queries.
• Migrate data from HDFS/Hive to cloud-based storage and Databricks.
• Experience with Azure Data Lake Storage (ADLS), AWS S3, or equivalent cloud storage.
• Develop and maintain robust data pipelines using Databricks Workflows and job orchestration tools.
• Implement data quality, validation, reconciliation, and error-handling frameworks during migration.
• Perform source-to-target mapping, data profiling, transformation analysis, and migration validation.
• Troubleshoot performance issues across Spark clusters, notebooks, jobs, and data pipelines.
• Work closely with architects, business analysts, data scientists, and application teams to understand data requirements.
• Implement CI/CD practices for Databricks notebooks, jobs, and code using Git and DevOps tools.
• Follow enterprise security, governance, audit, and compliance standards applicable to banking environments.
• Ensure migrated workloads meet performance, scalability, availability, and data-quality requirements.

Job description

Role & responsibilities

Key Responsibilities

• Design and execute Hadoop to Databricks migration strategies for large-scale data platforms.
• Migrate existing HDFS, Hive, Spark, and MapReduce workloads to Databricks.
• Data Engineering Hadoop to Databricks migration experience
• Strong hands-on experience in migrating and optimizing Spark code
• Analyze existing Hadoop/Spark applications and identify opportunities for modernization and optimization.
• Strong hands-on experience in migrating, refactoring, and optimizing Spark code for Databricks.
• Develop scalable ETL/ELT pipelines using Spark, SQL, and Databricks.
• Optimize Spark jobs by improving partitioning, caching, joins, shuffling, serialization, and memory utilization.
• Work with Databricks Delta Lake / Delta tables for reliable and scalable data processing.
• Implement Delta Lake features such as ACID transactions, schema evolution, time travel, and optimized data layouts.
• Convert legacy Hive SQL / Spark SQL workloads to Databricks SQL and optimize queries.
• Migrate data from HDFS/Hive to cloud-based storage and Databricks.
• Experience with Azure Data Lake Storage (ADLS), AWS S3, or equivalent cloud storage.
• Develop and maintain robust data pipelines using Databricks Workflows and job orchestration tools.
• Implement data quality, validation, reconciliation, and error-handling frameworks during migration.
• Perform source-to-target mapping, data profiling, transformation analysis, and migration validation.
• Troubleshoot performance issues across Spark clusters, notebooks, jobs, and data pipelines.
• Work closely with architects, business analysts, data scientists, and application teams to understand data requirements.
• Implement CI/CD practices for Databricks notebooks, jobs, and code using Git and DevOps tools.
• Follow enterprise security, governance, audit, and compliance standards applicable to banking environments.
• Ensure migrated workloads meet performance, scalability, availability, and data-quality requirements.