Lead Data Engineer

LogixHealth; Bengaluru; Presencial; INR 4.000.000 - 6.000.000; Jornada completa

Descripción de la vacante

LogixHealth in Bengaluru is seeking a senior data engineer to lead and contribute to building a self-service analytics data platform. You will architect scalable data pipelines using Databricks, PySpark, Spark, Python, and Delta Lake within the Azure ecosystem to empower reporting across the business.

You will drive data quality, governance, and security in projects, mentor engineers, and collaborate with product teams to integrate Tableau, Power BI, APIs, and other systems.

Formación

• 8+ years of data engineering experience.
• 3+ years in a senior/principal engineer role leading projects.
• Strong hands-on Databricks with Delta Live Tables and related tooling.
• Expertise in PySpark and/or Scala; Spark/DataFrame programming.
• Knowledge of data security and governance; HIPAA or SOC 2 preferred.

Responsabilidades

• Lead and contribute to the development of a self-service data platform for reporting and analytics.
• Design data solutions using Databricks, SQL, Python, Spark, and Delta Lake in the Azure ecosystem.
• Implement ETL/ELT best practices for data quality, partitioning, maintainability, and reusability.
• Collaborate with engineers, product managers, and business leaders to integrate with Tableau, Power BI, APIs, and other systems.
• Establish CI/CD pipelines and use IaC tools for deployment and monitoring (Git, Terraform, Azure DevOps/GitHub Actions/Jenkins, Azure Monitor/Datadog).

Conocimientos

Leadership

Mentoring

Communication

Data modeling

Problem solving

Herramientas

Databricks

Delta Live Tables

Auto Loader

Databricks Workflows

Unity Catalog

Spark

Python

Scala

PySpark

Pandas

NumPy

Azure Data Factory

Azure Databricks

Descripción del empleo

Responsibilities

• Lead and contribute to the development of a self-service data platform for reporting and analytics.
• Design and build data solutions using Databricks, SQL, Python, Spark, and Delta Lake within the Azure ecosystem (Blob Storage, Data Factory, Event Hubs).
• Implement best practices for ETL/ELT processes, focusing on data quality management, partitioning, maintainability, and reusability.
• Collaborate with engineers, product managers, and business leaders to ensure seamless integration of the data platform with systems such as Tableau, Power BI, APIs, and other applications.
• Establish CI/CD pipelines and use infrastructure-as-code tools for deployment and monitoring (Git, Terraform, Azure DevOps/GitHub Actions/Jenkins, Azure Monitor/Datadog).
• Ensure adherence to organizational compliance standards and the Code of Conduct as defined in LogixGarden.

Requirements

• 8+ years of data engineering experience.
• 3+ years in a senior or principal engineer role, with proven experience leading projects.
• Strong hands-on experience with Databricks and related integrations (Delta Live Tables, Auto Loader, Databricks Workflows / Apache Airflow, Unity Catalogue).
• Expertise in PySpark and/or Scala.
• Strong DataFrame programming experience (Spark, Pandas, NumPy).
• Proven ability to design scalable, secure, and maintainable data pipelines.
• Excellent communication and leadership skills, with a passion for mentoring.
• Solid understanding of data security and governance frameworks, with exposure to HIPAA or SOC 2 compliance preferred.

Preferred Skills

• AWS cloud experience.
• AWS Databricks implementation expertise.
• Experience designing and implementing data security and governance frameworks.