Data Engineer Advisor

NTT DATA BUSINESS SOLUTIONS; Chennai District; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

NTT DATA BUSINESS SOLUTIONS in Chennai, Tamil Nadu, is seeking a Data Engineer Advisor to join our analytics team. You will design and implement Databricks-based data platforms, optimize pipelines, and collaborate with cloud infra teams.

The ideal candidate has deep experience with Unity Catalog, Delta Lake, Delta Live Tables, PySpark, SQL, and Terraform, plus hands-on cloud skills across AWS, Azure, or GCP, to scale data architectures and control costs.

Qualifications

• Expertise with Databricks Unity Catalog, Delta Lake, DABs, and Delta Live Tables (DLT).
• Proficient in Python (PySpark), SQL and shell scripting (Bash/PowerShell).
• Hands-on Terraform, Git workflows and CI/CD tools.
• Experience administering data infrastructure on AWS/Azure/GCP.
• Solid understanding of Spark core concepts, memory management and tuning.

Responsibilities

• Architect and provision the core Databricks ecosystem across multi-workspace environments using cloud-native services.
• Establish shared data structures and optimized storage strategies with Delta Lake, Auto Loader and DLT.
• Implement cost-management strategies by configuring cluster sizing, autoscaling and serverless SQL warehouses.

Skills

Databricks

Python (PySpark)

SQL

Shell scripting

Terraform

CI/CD

Cloud platforms

Apache Spark

Tools

Auto Loader

Delta Lake

Delta Live Tables

Unity Catalog

Git workflows

Azure DevOps

GitHub Actions

GitLab CI

Job description

Req ID: phone_number

We are currently seeking a Data Engineer Advisor to join our team in Chennai, Tamil Nadu (IN-TN), India (IN).

Technical Expertise

• Databricks Ecosystem: Mastery of Unity Catalog, Delta Lake, Photon engine, Serverless Compute, Databricks Asset Bundles (DABs), and Delta Live Tables (DLT).
• Core Languages: Advanced proficiency in Python (PySpark), SQL, and standard shell scripting languages (Bash / PowerShell).
• Infrastructure DevOps: Strong experience with Terraform, git workflows, and CI/CD tools (e.g., Azure DevOps, GitHub Actions, or GitLab CI).
• Cloud Platforms: Hand-on experience administering data infrastructure on at least one cloud provider (AWS, Azure, or GCP).
• Distributed Systems: Complete understanding of Apache Spark core concepts, memory management, optimizations (Z-order, data skipping), and tuning.

Key Responsibilities Platform Architecture Foundation Build

• Architect and provision the core Databricks ecosystem across multi-workspace environments using cloud-native services (Azure, AWS, or GCP).
• Establish shared data structures and optimized storage strategies utilizing the Delta Lake format, Auto Loader, and Delta Live Tables (DLT).
• Implement cost-management strategies, configuring optimal cluster sizing, autoscaling, and serverless SQL warehouses to balance performance against cloud spend.