Senior Data Engineer

Allianz Services; Maharashtra; On-site; INR 350,000 - 700,000; Full time

Job summary

Allianz Services in Maharashtra, India seeks a Senior Data Engineer to lead the re-engineering of an enterprise data platform on Azure Synapse, with migration planning toward Databricks. You will audit existing architectures, validate data governance, and steer the technical migration strategy.

The role requires deep SQL/Python, Spark, and Databricks expertise, with hands-on leadership across data governance, CI/CD, and lakehouse patterns in a regulated, enterprise setting.

Qualifications

• Proven track record auditing and re-engineering complex data platforms.
• Deep expertise with Azure Synapse and Databricks migrations.
• Strong governance, reconciliation, and data quality controls.

Responsibilities

• Lead assessment and re-engineering of an existing enterprise data platform.
• Reverse-engineer pipelines, data models, and governance controls.
• Define and execute migration strategies to Databricks Lakehouse.
• Ensure data quality, reconciliation, and auditability across platforms.
• Collaborate with governance, architecture, and business stakeholders.

Skills

Azure Synapse

Databricks

Python

Spark (PySpark)

SQL

Delta Lake on Azure

SCD/CDP patterns

Data governance

Lakehouse architecture

CI/CD for data pipelines

Terraform/ARM

GitHub Actions/Azure DevOps

Tools

Docker

Terraform

ARM templates

Job description

Role Overview We are looking for a highly experienced Senior Data Engineer to lead the re-engineering of an existing enterprise data platform built on Azure Synapse Analytics. The role requires deep technical seniority to audit, understand, and validate a complex end-to-end data architecture spanning source ingestion through to consumption — and to drive a future migration of validated workloads to Databricks. This is not a greenfield role: it demands the ability to reverse-engineer existing implementations, assess their correctness, and own the technical migration strategy.

Key Responsibilities

• Lead the technical assessment and re-engineering of an existing enterprise data platform, spanning all layers from source ingestion through to data consumption
• Reverse-engineer, document, and validate existing pipeline logic, data models, transformation frameworks, and data governance controls
• Identify gaps, defects, and technical debt across the platform and remediate where implementations are incorrect or sub-optimal
• Ensure correctness of data processing patterns including change data capture, slowly changing dimensions, deduplication, and business reconciliation
• Design and implement target-state architectures aligned to modern lakehouse principles, ensuring feature parity and business logic fidelity during transitions
• Manage platform evolution initiatives, including parallel-run phases where multiple implementations operate simultaneously, validating output consistency before cutover
• Define and execute migration strategies for existing workloads to modern data platforms, preserving existing governance and control framework semantics
• Re-implement ingestion, transformation, and orchestration pipelines on target platforms, maintaining audit, quality, and reconciliation standards
• Collaborate with business, data governance, and architecture stakeholders to validate embedded business rules and data quality requirements
• Provide technical leadership across re-engineering and migration workstreams, contributing to decommission planning for legacy components

Core Technical Skills

• Azure Synapse & Data Platform Mandatory hands-on expertise with:
• Delta Lake on Azure (Synapse Lakehouse patterns)
• Azure Analysis Services and Power BI consumption layer patterns
• Deep understanding of medallion architecture: Raw / Harmonized / Conformed / Consumption layers
• Strong knowledge of SCD Type 0/1/2, CDC patterns, soft/hard delete, and retroactive change processing
• Experience with Synapse SQL Pool — stored procedures, control tables, and data quality validation patterns
• Experience with audit, balance, and control frameworks — parameterized, modular pipeline governance at enterprise scale
• Familiarity with config-driven and automation-first pipeline patterns (YAML, PySpark, SQL-driven generation from mapping documents)

Databricks & Lakehouse

• Hands-on experience with Azure Databricks (Delta Live Tables, Unity Catalog preferred)
• Strong Apache Spark skills (PySpark / Spark SQL)
• Experience migrating workloads from legacy data warehouse or Synapse environments to a Databricks Lakehouse
• Ability to re-implement governance and control frameworks natively in Databricks (audit logging, reconciliation, DQ checks)
• Experience with Delta Lake features: MERGE, CDC, time travel, schema enforcement
• Strong Python and SQL programming skills
• Experience with ETL/ELT at scale: denormalization, surrogate keys, directory tables, curated data models
• Experience integrating complex data sources: Oracle DB, SQL Server, Azure SQL DB, file systems, Salesforce, APIs
• Strong data modelling skills: relational, dimensional, and lakehouse-oriented

DevOps & Automation

• CI/CD pipelines for data engineering (Azure DevOps / GitHub Actions)
• Infrastructure as Code (Terraform or ARM)
• Containerization (Docker)
• Experience with automated testing frameworks for data pipelines (unit testing, reconciliation-based validation)

Nice to Have

• Experience with Unity Catalog for data governance and lineage
• Familiarity with Azure Purview for data cataloguing and governance
• Exposure to real-time and streaming pipelines (Event Hub / Kafka / Kinesis)
• Experience with GenAI or ML platform integration (MLOps, feature engineering pipelines)
• Familiarity with monitoring and observability tools (e.g., Dynatrace)
• Exposure to BI tools (Power BI, Tableau)

Experience & Profile

• 7+ years of experience in Data Engineering, with significant platform migration or re-engineering experience
• Proven track record auditing and taking ownership of existing, complex enterprise data platforms — not just building from scratch
• Deep knowledge of enterprise data governance patterns: audit trails, reconciliation, data quality controls, SCD versioning
• Strong analytical mindset: ability to read existing implementations, identify intent versus defect, and make sound re-engineering decisions
• Comfortable operating across both hands-on engineering and technical architecture
• Strong communication skills — able to engage business, governance, and engineering stakeholders with clarity
• Experience working in regulated or enterprise-scale environments (financial services a plus)