Lead Data Engineer

Kalpita Technologies; Bengaluru; On-site; INR 2,400,000 - 4,200,000; Full time

Job summary

Kalpita Technologies in Bengaluru is seeking an experienced Data Migration Lead to drive end-to-end migrations from legacy systems to cloud-based Azure SQL. You will profile databases, design mappings, and build repeatable ETL/ELT pipelines for extensive historical data loads.

You will define data quality rules, automate reconciliations, and manage cutover planning. Leadership of 1–2 engineers and coordination with stakeholders will be essential for success.

Qualifications

• 8–12 years in data engineering with at least two legacy-to-cloud migrations.
• Expert T-SQL skills, including complex views, procedures, CTEs, and indexing.
• Hands-on Azure SQL Database plus serverless or elastic tiers.
• Strong ETL/ELT tooling with Azure Data Factory and/or SSIS; Python/PowerShell optional.
• Data reconciliation, data modeling for OLTP and reporting workloads.
• Experience with JSON, data rules, quality checks, and automation.
• Experience with Git and CI/CD for database code and migrations.
• Clear communication with US client stakeholders; leadership/mentoring.
• Education-sector data experience and cloud-native architectures are a plus.

Responsibilities

• Own end-to-end migration strategy from legacy to cloud.
• Profile and reverse‑engineer legacy SQL Server databases and views.
• Build source-to-target mappings on Azure SQL for new platforms.
• Design and build repeatable ETL/ELT pipelines for 10‑year data loads.
• Define data quality rules and exception handling for bad data.
• Automate reconciliation using row counts, checksums, and comparisons.
• Plan production cutover with runbooks, mock migrations, rollback criteria.
• Lead and mentor 1–2 engineers, enforce coding/testing standards.
• Report migration status and risk to client stakeholders early.
• Collaborate with Solution Architect, BAs, and client SMEs.

Skills

T‑SQL

Azure SQL

Python

PowerShell

SSIS

Azure Data Factory

GitHub Actions

CI/CD

Leadership

Mentoring

Education

Microsoft Azure Data Engineer Associate (DP-203)

Tools

Terraform

Bicep

GitHub Actions

Azure DevOps

Job description

Role Requirements

• Own the end-to-end migration strategy, including extraction, mapping, transformation, load, reconciliation, and cutover.
• Profile and reverse-engineer legacy SQL Server databases, views, and materialized view outputs with little or no documentation.
• Build source-to-target mappings from the legacy model into the new operational and reporting databases on Azure SQL.
• Design and build repeatable, idempotent ETL/ELT pipelines for a 10-year historical data load.
• Define data quality rules, cleansing logic, and exception handling for bad, duplicate, and orphaned records.
• Build automated reconciliation using row counts, checksums, aggregates, and record-level comparisons between legacy and target systems.
• Run parallel legacy-versus-platform cycles, including comparison of JSON report outputs, and drive defects to closure.
• Plan and rehearse production cutover, including runbooks, mock migrations, timing, rollback, and sign-off criteria.
• Work with the Solution Architect, Business Analysts, and client SMEs to resolve data rules and business definitions.
• Lead and review the work of a Data Migration Engineer and set coding and testing standards.
• Tune load performance and query performance on large historical datasets.
• Report migration status, risks, and data issues to client stakeholders clearly and early.
• Have 8–12 years of experience in data engineering, with at least two large legacy-to-cloud migrations delivered end to end as the lead.
• Demonstrate expert T‑SQL skills, including complex views, stored procedures, CTEs, window functions, indexing, and query plans.
• Have deep SQL Server experience and hands‑on Azure SQL Database experience, including serverless or elastic tiers.
• Have strong ETL/ELT tooling experience with Azure Data Factory and/or SSIS, plus scripting in Python or PowerShell.
• Demonstrate data reconciliation and validation practices using automated checks rather than manual spot‑checking.
• Have data modeling experience for transactional (OLTP) and reporting workloads.
• Be comfortable reading and producing JSON and comparing structured outputs between systems.
• Have cutover planning experience, including mock runs, rollback plans, and go/no‑go criteria.
• Have experience with Git and CI/CD pipelines using GitHub Actions or Azure DevOps for database code and migration jobs.
• Have led or mentored at least 1–2 engineers.
• Demonstrate clear written and spoken English and the ability to work directly with US client stakeholders.
• Survey, assessment, or research data experience, including Qualtrics exports, respondent‑level data, and longitudinal datasets, is good to have.
• Education‑sector data experience, such as school or district reference data, is preferred.
• Experience with event‑driven or microservices architectures and understanding how data flows through queues is preferred.
• Experience with Azure Blob Storage, Key Vault, Azure Monitor/Log Analytics, dbt, Databricks, or Microsoft Fabric is good to have.
• Experience with infrastructure as code using Terraform or Bicep is preferred.
• Experience working with AI coding assistants such as Claude Code or GitHub Copilot is good to have.
• Microsoft Azure Data Engineer Associate (DP-203) certification or equivalent is preferred.