Data Engineer Advisor

NTT DATA, Inc.; Hyderabad; On-site; INR 3,000,000 - 4,500,000; Full time

Job summary

NTT DATA, Inc. in Hyderabad, India seeks a Hands-On Data Tech Lead with 8–10 years of experience in SSIS/SQL Server and cloud-based modern data platforms.

You will drive migration to Azure Databricks, architect MDW/Medallion patterns, and mentor a team of data engineers while actively coding. You will design scalable ETL/ELT pipelines in PySpark, optimize Spark configurations, implement Delta Lake, and enable CI/CD automation with Azure DevOps or GitHub Actions, ensuring governance and

Qualifications

• 8–10 years of hands-on data engineering and leadership experience.
• Strong expertise in Azure Databricks, Delta Lake, and PySpark.
• Proficient in SQL Server, SSIS, T-SQL, and data warehousing concepts.
• Experience building scalable ETL/ELT pipelines and mentoring teams.

Responsibilities

• Lead migration of legacy SSIS to cloud-native Azure Databricks/PySpark Lakehouse.
• Architect and implement end-to-end Modern Data Warehouse with MDW/Medallion layers.
• Mentor data engineers, enforce design patterns and coding standards.
• Collaborate with Data Architects, BAs, and DevOps to translate requirements.
• Develop scalable batch/real-time ETL/ELT pipelines using PySpark on Databricks.
• Refactor on-prem SSIS and optimize T-SQL, ensuring business continuity during transition.
• Implement Delta Lake features including schema evolution and time travel.
• Orchestrate workflows with ADF, ADLS Gen2, Databricks, and on-premise DBs.
• Optimize Spark configs and cluster usage to reduce compute overhead.

Skills

Azure Databricks

Delta Lake

PySpark

Python

SQL Server/SSIS

Azure Data Factory

CI/CD

Git

Data Modeling

Leadership

Mentoring

Tools

GitHub

Azure DevOps

Jira

Job description

Job Summary

A highly skilled 8 to 10 years of experience on SSIS. SQL Server, Azure Databricks, Pyspark,

• Lead the migration of legacy SQL Server Integration Services (SSIS) packages to modern, cloud-native Azure Databricks / PySpark Lakehouse environments.
• Architect end-to-end Modern Data Warehouse (MDW) and Medallion architecture systems (Bronze, Silver, Gold layers) within Azure.
• Mentor and guide a team of data engineers, conducting stringent code reviews, enforcing design patterns, and establishing development standards.
• Collaborate with stakeholders including Data Architects, Business Analysts, and DevOps teams to translate complex business requirements into technical blueprints.
• Develop and optimize scalable, modular, high-performance batch and real-time ETL/ELT pipelines using PySpark on Azure Databricks cluster environments.
• Maintain and refactor existing on-premise SSIS packages and complex T-SQL code (stored procedures, views, performance tuning) to ensure business continuity during transition phases.
• Implement Delta Lake features, managing transactions, schema evolution, and time-travel querying functionalities.
• Orchestrate workflows seamlessly by integrating Azure Data Factory (ADF) with on-premise databases, cloud storage (ADLS Gen2), and Databricks activities.
• Optimize Spark configurations and cluster utilization to drastically reduce cloud compute overhead and job execution windows.
• Enforce data quality frameworks encompassing automated data validation, error reconciliation, and operational audit logging.
• Implement CI/CD automations via Azure DevOps or GitHub Actions to manage environments, code repositories (Git), and deployment pipelines.

India T&M Special Bonus:

Job Summary

A highly skilled 8 to 10 years of experience on SSIS. SQL Server, Azure Databricks, Pyspark, Hands-On Data Tech Lead to drive the modernization and scaling of our enterprise data platform to provide technical leadership while remaining deeply technical and actively writing code. Also lead the migration, integration, and development of data pipelines across our legacy SQLServer/SSIS environments and our modern cloud infrastructure built on Azure Databricks and PySpark and act as the principal architect and developer for end-to-end ELT/ETL pipelines, ensuring optimal performance, data governance, and scalability.

• Lead the migration of legacy SQL Server Integration Services (SSIS) packages to modern, cloud-native Azure Databricks / PySpark Lakehouse environments.
• Architect end-to-end Modern Data Warehouse (MDW) and Medallion architecture systems (Bronze, Silver, Gold layers) within Azure.
• Mentor and guide a team of data engineers, conducting stringent code reviews, enforcing design patterns, and establishing development standards.
• Collaborate with stakeholders including Data Architects, Business Analysts, and DevOps teams to translate complex business requirements into technical blueprints.
• Develop and optimize scalable, modular, high-performance batch and real-time ETL/ELT pipelines using PySpark on Azure Databricks cluster environments.
• Maintain and refactor existing on-premise SSIS packages and complex T-SQL code (stored procedures, views, performance tuning) to ensure business continuity during transition phases.
• Implement Delta Lake features, managing transactions, schema evolution, and time-travel querying functionalities.
• Orchestrate workflows seamlessly by integrating Azure Data Factory (ADF) with on-premise databases, cloud storage (ADLS Gen2), and Databricks activities.
• Optimize Spark configurations and cluster utilization to drastically reduce cloud compute overhead and job execution windows.
• Enforce data quality frameworks encompassing automated data validation, error reconciliation, and operational audit logging.
• Implement CI/CD automations via Azure DevOps or GitHub Actions to manage environments, code repositories (Git), and deployment pipelines.

Skill Requirements

Technical Skills

• Azure Big Data Stack: Deep, production-level knowledge of Azure Databricks, Delta Lake, Unity Catalog, and Azure Data Lake Storage (ADLS Gen2).
• Programming Languages: Advanced proficiency in Python and PySpark core APIs (Spark SQL and DataFrames).
• Legacy ETL & Databases: Mastery of SQL Server, Advanced T-SQL writing, and extensive experience configuring, deploying, and troubleshooting SSIS packages.
• Orchestration: Hands-on experience mapping dependencies and scheduling triggers using Azure Data Factory (ADF).
• Data Modeling: Expert understanding of relational (OLTP) and dimensional (OLAP) schemas, explicitly Star and Snowflake architectures.
• DevOps & Security: Competency with Git branching, CI/CD automated release deployments, and secure data handling mechanisms (Azure Key Vault, network isolation).

Soft Skills & Leadership

• Proven track record as a coding lead who splits time between management/design and hands-on execution.
• Excellent cross-functional communication skills, comfortably discussing data strategies with both executive leaders and junior technical staff.
• Strong debugging and root-cause analysis (RCA) skills for multi-tier platform failures.
• Preferred Certifications: Microsoft Certified: Azure Data or Databricks