Lead Data Engineer

NationsBenefits, LLC; Hyderabad; On-site; INR 3,000,000 - 5,400,000; Full time

Job summary

NationsBenefits, LLC in Hyderabad, India, is seeking a Lead Data Engineer to design and optimize scalable ETL/ELT pipelines using Azure Databricks, PySpark, and Delta Lake. You will govern data with Unity Catalog and support real-time and batch processing across scrum teams.

The role requires 7–10 years of experience, strong T-SQL, and hands-on Azure cloud tooling. Healthcare industry exposure is a plus, with a focus on data integrity and scalable solutions.

Qualifications

• 7-10 years of experience as a Lead Data Engineer.
• Proven experience with T-SQL, Azure Databricks, Spark, and Delta Lake.
• Hands-on with Unity Catalog for data governance and lineage.
• Experience with ADLS Gen2 and Azure data services.
• Strong problem-solving and collaboration across scrum teams.

Responsibilities

• Work with different scrum teams to develop all the quality database programming requirements of the sprint.
• Experience in Azure cloud platforms like Advanced Python Programming, Databricks, Azure SQL, Data factory (ADF), Data Lake, Data storage, SSIS.
• Create and deploy scalable ETL/ELT pipelines with Azure Databricks by utilizing PySpark and SQL.
• Create Delta Lake tables with ACID transactions and schema evolution to support real-time and batch processing.
• Experience in Unity Catalog for centralized data governance, access control, and data lineage tracking.
• Independently analyse, solve, and correct issues in real time, providing problem resolution end-to-end.
• Develop unit tests to be able to test them automatically.
• Use SOLID development principles to maintain data integrity and cohesiveness.
• Interact with product owner and business representatives to determine and satisfy needs.
• Sense of ownership and pride in your performance and its impact on company’s success.
• Critical thinker and problem-solving skills.
• Team player.
• Good time-management skills.
• Great interpersonal and communication skills.

Skills

T-SQL

Azure Databricks

PySpark

Delta Lake

Unity Catalog

ADLS Gen2

Spark

SQL

Tools

Azure Data Factory

Azure SQL

Data Lake Storage

DevOps

Job description

Job Title: Lead Data Engineer

Experience: 7-10 Years

Location: Hyderabad, India

Employment Type: Full-time

About NationsBenefits

At NationsBenefits, we are leading the transformation of the insurance industry by developing innovative benefits management solutions. We focus on modernizing complex back-office systems to create scalable, secure, and high-performing platforms that streamline operations for our clients. As part of our strategic growth, we are focused on platform modernization — transitioning legacy systems to modern, cloud-native architectures that support the scalability, reliability, and high performance of core backoffice functions in the insurance domain.

Position Overview

We are seeking a self-driven Lead Data Engineer with 7–10 years of experience to build and optimize scalable ETL/ELT pipelines using Azure Databricks, PySpark, and Delta Lake. The role involves working across scrum teams to develop data solutions, ensure data governance with Unity Catalog, and support real-time and batch processing. Strong problem-solving skills, T-SQL expertise, and hands-on experience with Azure cloud tools are essential. Healthcare domain knowledge is a plus.

Job Description

• Work with different scrum teams to develop all the quality database programming requirements of the sprint.
• Experience in Azure cloud platforms like Advanced Python Programming, Databricks, Azure SQL, Data factory (ADF), Data Lake, Data storage, SSIS.
• Create and deploy scalable ETL/ELT pipelines with Azure Databricks by utilizing PySpark and SQL.
• Create Delta Lake tables with ACID transactions and schema evolution to support real-time and batch processing.
• Experience in Unity Catalog for centralized data governance, access control, and data lineage tracking.
• Independently analyse, solve, and correct issues in real time, providing problem resolution end-to-end.
• Develop unit tests to be able to test them automatically.
• Use SOLID development principles to maintain data integrity and cohesiveness.
• Interact with product owner and business representatives to determine and satisfy needs.
• Sense of ownership and pride in your performance and its impact on company’s success.
• Critical thinker and problem-solving skills.
• Team player.
• Good time-management skills.
• Great interpersonal and communication skills.

Mandatory Qualifications

• 7-10 years of experience as a Lead Data Engineer.
• Self-driven with minimal supervision.
• Proven experience with T-SQL programming, Azure Databricks, Spark (PySpark/Scala), Delta Lake, Unity Catalog, ADLS Gen2.
• Microsoft TFS, Visual Studio, DevOps exposure.
• Experience with cloud platforms such as Azure or any.
• Analytical, problem-solving mindset.

Preferred Qualifications

• Healthcare domain knowledge.