Lead Data Engineer: Hadoop-to-Databricks Migration & Spark

Atyeti Inc.; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

Atyeti Inc. seeks a data engineering specialist to migrate large Hadoop/Spark workloads to Databricks and modernize data platforms. You will design strategies, refactor Spark code, and implement scalable ETL/ELT pipelines on Databricks, with Delta Lake and cloud storage integration.

You will collaborate with architects, data scientists, and analysts to ensure performance, reliability, and governance in banking-scale environments, leveraging CI/CD and DevOps to streamline notebooks and jobs.

Qualifications

• Experience migrating large Hadoop/Spark workloads to Databricks.
• Proven ability to optimize Spark code and memory usage.
• Ability to design scalable ETL/ELT pipelines on Databricks.
• Experience with Delta Lake features and cloud storage.

Responsibilities

• Design and execute Hadoop to Databricks migration strategies for large-scale data platforms.
• Migrate existing HDFS, Hive, Spark, and MapReduce workloads to Databricks.
• Develop scalable ETL/ELT pipelines using Spark, SQL, and Databricks.
• Optimize Spark jobs by improving partitioning, caching, joins, shuffling, serialization, and memory utilization.
• Work with Databricks Delta Lake / Delta tables for reliable and scalable data processing.
• Implement Delta Lake features such as ACID transactions, schema evolution, time travel.
• Convert legacy Hive SQL / Spark SQL workloads to Databricks SQL and optimize queries.
• Migrate data to cloud-based storage (ADLS/S3) and Databricks.
• Develop and maintain robust data pipelines using Databricks Workflows and orchestration tools.
• Implement data quality, validation, reconciliation, and error-handling frameworks.
• Troubleshoot performance across Spark clusters, notebooks, and pipelines.
• Collaborate with architects, business analysts, data scientists, and teams to meet data requirements.
• Implement CI/CD for Databricks notebooks, jobs, and code using Git and DevOps tools.
• Adhere to security, governance, and compliance standards for banking environments.

Skills

Databricks migration

Spark optimization

ETL/ELT pipelines

Hadoop to Databricks

Tools

Databricks

Hadoop

Spark

Job description

Atyeti Inc. seeks a data engineering specialist to migrate large Hadoop/Spark workloads to Databricks and modernize data platforms. You will design strategies, refactor Spark code, and implement scalable ETL/ELT pipelines on Databricks, with Delta Lake and cloud storage integration.

You will collaborate with architects, data scientists, and analysts to ensure performance, reliability, and governance in banking-scale environments, leveraging CI/CD and DevOps to streamline notebooks and jobs.