Lead Data Engineer

NB Healthcare Technologies; Hyderabad; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

NB Healthcare Technologies is seeking a Lead Data Engineer to drive data platform initiatives in Hyderabad. You will partner with scrum teams to deliver scalable ETL/ELT pipelines using Azure Databricks, PySpark/SQL, and Delta Lake, enabling real-time and batch processing with ACID tables.

The role requires 9–14 years of relevant experience, strong T-SQL, Azure cloud expertise, and exposure to DevOps practices.

Qualifications

• 9–14 years of experience as a Lead Data Engineer.
• Proven experience with T-SQL programming, Azure Databricks, Spark (PySpark/Scala), Delta Lake, Unity Catalog, ADLS Gen2.
• Microsoft TFS, Visual Studio, DevOps exposure.
• Experience with cloud platforms such as Azure or any.

Responsibilities

• Work with different scrum teams to develop all the quality database programming requirements of the sprint.
• Create and deploy scalable ETL/ELT pipelines with Azure Databricks by utilizing PySpark and SQL.
• Create Delta Lake tables with ACID transactions and schema evolution to support real-time and batch processing.
• Experience in Unity Catalog for centralized data governance, access control, and data lineage tracking.
• Independently analyse, solve, and correct issues in real time, providing problem resolution end-to-end.
• Develop unit tests to be able to test them automatically.
• Use SOLID development principles to maintain data integrity and cohesiveness.
• Interact with product owner and business representatives to determine and satisfy needs.

Skills

Azure Databricks

PySpark

Spark

Delta Lake

Unity Catalog

ADLS Gen2

T-SQL

Azure SQL

Data Factory

Data Lake

SQL

Python

Tools

Microsoft TFS

Visual Studio

DevOps

Azure Databricks

Azure Data Factory

Job description

Role & responsibilities

Work with different scrum teams to develop all the quality database programming requirements of the sprint.

• Experience in Azure cloud platforms like Advanced Python Programming, Databricks, Azure SQL , Data factory (ADF), Data Lake, Data storage, SSIS.
• Create and deploy scalable ETL/ELT pipelines with Azure Databricks by utilizing PySpark and SQL.
• Create Delta Lake tables with ACID transactions and schema evolution to support real-time and batch processing.
• Experience in Unity Catalog for centralized data governance, access control, and data lineage tracking.
• Independently analyse, solve, and correct issues in real time, providing problem resolution end-to-end.
• Develop unit tests to be able to test them automatically.
• Use SOLID development principles to maintain data integrity and cohesiveness.

Interact with product owner and business representatives to determine and satisfy needs.

• Sense of ownership and pride in your performance and its impact on companys success.
• Critical thinker and problem-solving skills.
• Team player.
• Good time-management skills.
• Great interpersonal and communication skills.

Preferred candidate profile

9-14 years of experience as a Lead Data Engineer.

Self-driven with minimal supervision. Proven experience with T-SQL programming, Azure Databricks, Spark (PySpark/Scala), Delta Lake, Unity Catalog, ADLS Gen2

Microsoft TFS, Visual Studio, Devops exposure.

Experience with cloud platforms such as Azure or any.

Analytical, problem-solving mindset.