Lead Data Engineer with (Hadoop +Databricks)_ Exp: 7+ Years

Atyeti Inc.; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

Atyeti Inc. seeks a data engineering specialist to migrate large Hadoop/Spark workloads to Databricks and modernize data platforms. You will design strategies, refactor Spark code, and implement scalable ETL/ELT pipelines on Databricks, with Delta Lake and cloud storage integration.

You will collaborate with architects, data scientists, and analysts to ensure performance, reliability, and governance in banking-scale environments, leveraging CI/CD and DevOps to streamline notebooks and jobs.

Qualifications

• Experience migrating large Hadoop/Spark workloads to Databricks.
• Proven ability to optimize Spark code and memory usage.
• Ability to design scalable ETL/ELT pipelines on Databricks.
• Experience with Delta Lake features and cloud storage.

Responsibilities

• Design and execute Hadoop to Databricks migration strategies for large-scale data platforms.
• Migrate existing HDFS, Hive, Spark, and MapReduce workloads to Databricks.
• Develop scalable ETL/ELT pipelines using Spark, SQL, and Databricks.
• Optimize Spark jobs by improving partitioning, caching, joins, shuffling, serialization, and memory utilization.
• Work with Databricks Delta Lake / Delta tables for reliable and scalable data processing.
• Implement Delta Lake features such as ACID transactions, schema evolution, time travel.
• Convert legacy Hive SQL / Spark SQL workloads to Databricks SQL and optimize queries.
• Migrate data to cloud-based storage (ADLS/S3) and Databricks.
• Develop and maintain robust data pipelines using Databricks Workflows and orchestration tools.
• Implement data quality, validation, reconciliation, and error-handling frameworks.
• Troubleshoot performance across Spark clusters, notebooks, and pipelines.
• Collaborate with architects, business analysts, data scientists, and teams to meet data requirements.
• Implement CI/CD for Databricks notebooks, jobs, and code using Git and DevOps tools.
• Adhere to security, governance, and compliance standards for banking environments.

Skills

Databricks migration

Spark optimization

ETL/ELT pipelines

Hadoop to Databricks

Tools

Databricks

Hadoop

Spark

Job description

Role & responsibilities

Key Responsibilities

• Design and execute Hadoop to Databricks migration strategies for large-scale data platforms.
• Migrate existing HDFS, Hive, Spark, and MapReduce workloads to Databricks.
• Data Engineering Hadoop to Databricks migration experience
• Strong hands-on experience in migrating and optimizing Spark code
• Analyze existing Hadoop/Spark applications and identify opportunities for modernization and optimization.
• Strong hands-on experience in migrating, refactoring, and optimizing Spark code for Databricks.
• Develop scalable ETL/ELT pipelines using Spark, SQL, and Databricks.
• Optimize Spark jobs by improving partitioning, caching, joins, shuffling, serialization, and memory utilization.
• Work with Databricks Delta Lake / Delta tables for reliable and scalable data processing.
• Implement Delta Lake features such as ACID transactions, schema evolution, time travel, and optimized data layouts.
• Convert legacy Hive SQL / Spark SQL workloads to Databricks SQL and optimize queries.
• Migrate data from HDFS/Hive to cloud-based storage and Databricks.
• Experience with Azure Data Lake Storage (ADLS), AWS S3, or equivalent cloud storage.
• Develop and maintain robust data pipelines using Databricks Workflows and job orchestration tools.
• Implement data quality, validation, reconciliation, and error-handling frameworks during migration.
• Perform source-to-target mapping, data profiling, transformation analysis, and migration validation.
• Troubleshoot performance issues across Spark clusters, notebooks, jobs, and data pipelines.
• Work closely with architects, business analysts, data scientists, and application teams to understand data requirements.
• Implement CI/CD practices for Databricks notebooks, jobs, and code using Git and DevOps tools.
• Follow enterprise security, governance, audit, and compliance standards applicable to banking environments.
• Ensure migrated workloads meet performance, scalability, availability, and data-quality requirements.