We are seeking an experienced Data Engineering Tech Lead to lead the design, development, optimization, and maintenance of scalable enterprise data engineering and Lakehouse solutions on the Azure cloud platform. In this role, you will drive both batch and near-real-time data processing across complex enterprise data sources, enforce robust data governance, and mentor engineering teams to enable data transformation in the Healthcare sector.
Key Tech Stack & Environment
- Governance & Architecture: Unity Catalog, Delta Lake, Medallion Architecture (Bronze/Silver/Gold)
Must-Have Technical Qualifications
- 12+ overall years of experience in Data Engineering / IT industry roles.
- 5+ years of hands‑on experience across the Azure Cloud platform ecosystem.
- 4+ years of core experience with Azure Databricks and PySpark.
- 3+ years of experience with Azure Data Factory (ADF).
- 3+ years of experience with ADLS Gen2.
- 3+ years of experience with Azure SQL.
- 2+ years of experience in Databricks Workflows and Unity Catalog governance.
- 2+ years of experience in modular Python programming, package builds, and testing (e.g., PyTest/unittest).
Key Responsibilities
1. Data Engineering & Lakehouse Architecture
- Design, build, and optimize scalable Lakehouse solutions ingesting structured, semi-structured, and unstructured data from ERP platforms, APIs, relational databases, NoSQL stores, and on-premises sources.
- Optimize Spark jobs and handle large-scale data processing using DataFrame and RDD APIs.
- Implement performance tuning, partitioning strategies, file format optimizations (Parquet/Delta), and Spark SQL tuning.
- Implement data access controls, object lineage, audit trails, and security policies using Unity Catalog integrated with Azure AD.
- Architect multi-layered storage patterns (Bronze-Silver-Gold) on ADLS Gen2 using lifecycle policies, RBAC/ACLs, and folder-level security.
- Build modular, parameterized orchestration workflows using ADF and Databricks Workflows (custom activities, triggers, dynamic expressions, error handling).
3. Technical Leadership & DevOps
- Author High-Level Design (HLD) and Low-Level Design (LLD) documents, architectural diagrams, and technical specifications.
- Establish coding standards, peer code review processes, and integration with static analysis tools (e.g., SonarQube, PyLint).
- Build automated deployment pipelines using Azure DevOps, Git, and Infrastructure as Code (ARM/Bicep/Terraform).
- Mentor junior and mid-level engineers on technical design, optimization, and best practices.
Secondary / Good-to-Have Skills
- Identity & CI/CD: Microsoft Entra ID (Azure AD), GitHub Actions.
- CDC & Migration: Debezium CDC implementation; Synapse to Databricks migration experience.
- Multi-Cloud Storage: Exposure to Google Cloud Storage (GCS).