Senior Data Engineer, Databricks & Delta Lake Architect

NTT DATA, Inc.; Indianapolis (IN); On-site; USD 120,000 - 170,000; Full time

Job summary

NTT DATA, Inc. is seeking a senior Databricks engineer to design and optimize data pipelines migrating legacy DB2/Guidewire data to the cloud.

You will build reliable, scalable ETL/ELT workflows, integrate with Snowflake and dbt, and implement Delta Lake patterns with ACID transactions. You will also focus on governance, schema evolution, and cost-optimized Spark workloads while collaborating across teams and ensuring data quality from source to analytics.

Qualifications

• 8+ years of experience in Databricks with legacy data models migrating to the cloud.

Responsibilities

• Develop Databricks notebooks, jobs, and workflows to replicate and enhance DB2/Guidewire-based pipelines and transformations.
• Implement Delta Lake tables and patterns (bronze/silver/gold, ACID, time travel, schema evolution) for migrated data.
• Integrate Databricks with AWS/S3 or Azure ADLS, ADF/Synapse, Key Vault, and Snowflake as required.
• Optimize Databricks clusters, jobs, and queries for performance and cost.
• Implement incremental loads, CDC patterns, and batch schedules for large datasets.
• Collaborate with Snowflake and dbt teams to ensure consistent data models and data contracts.
• Participate in data validation and reconciliation between DB2 400 / Guiderwire and Databricks outputs.
• Follow coding standards, version control, and CI/CD practices using Git/Azure DevOps.
• Provide defect fixes and support during SIT/UAT and post go-live stabilization.

Skills

Databricks

ETL/ELT design

PySpark

Spark SQL

Scala

Delta Lake

Data governance

CI/CD (Git/Azure DevOps)

Tools

Databricks

Azure DevOps

Git

Snowflake

dbt

Unity Catalog

Spark

Job description

NTT DATA, Inc. is seeking a senior Databricks engineer to design and optimize data pipelines migrating legacy DB2/Guidewire data to the cloud.

You will build reliable, scalable ETL/ELT workflows, integrate with Snowflake and dbt, and implement Delta Lake patterns with ACID transactions. You will also focus on governance, schema evolution, and cost-optimized Spark workloads while collaborating across teams and ensuring data quality from source to analytics.