Data Engineer

LKQ India; Bengaluru; Vor Ort; INR 1.400.000 - 2.400.000; Vollzeit

Zusammenfassung

LKQ India is seeking a Data Engineer to design, build and support reliable data pipelines and lakehouse solutions on Microsoft Fabric and Azure. The role will integrate ERP, database and file-based sources; prepare governed data for analytics; and release changes safely across environments.

You will collaborate with platform, architecture, BI and business teams to deliver reusable, monitored data products and ensure data quality and security across the data platform.

Qualifikationen

• Bachelor's degree or equivalent practical experience in computer science or a related field.
• Hands-on experience with data engineering using Microsoft Fabric and Azure data services.

Aufgaben

• Design and develop ingestion and transformation pipelines in Microsoft Fabric using Data Factory pipelines, Lakehouse, OneLake, Delta tables, notebooks and Spark.
• Integrate SQL Server, Oracle/ERP, APIs and file-based sources using Azure services such as Azure Data Factory, ADLS Gen2 and Key Vault.
• Implement batch and incremental loads, watermark handling and schema evolution.
• Develop Bronze, Silver and Gold datasets with clear data contracts and documented lineage.
• Write and tune SQL and PySpark transformations; monitor pipeline health and performance.
• Apply access controls, secure credentials and data quality checks with stakeholders.
• Use Git, pull requests and CI/CD to deploy data assets across environments.
• Create technical documentation, runbooks and automated tests; support releases and incidents.
• Collaborate with Power BI developers on datasets and semantic models.

Kenntnisse

SQL

Python

PySpark

Data modelling

Delta Lake

Distributed processing

Ausbildung

Bachelor's degree in computer science/engineering/information systems

Tools

Microsoft Fabric

Azure Data Factory

ADLS Gen2

Azure Key Vault

Git

Azure DevOps

Delta tables

Jobbeschreibung

Job Summary:

The Data Engineer will design, build and support reliable data pipelines and lakehouse solutions on Microsoft Fabric and Azure. The role will integrate ERP, database and file-based sources; prepare governed data for analytics; and use practical DevOps methods to release changes safely across environments. The successful candidate will work with platform, architecture, BI and business teams to deliver reusable, monitored data products

Role & responsibilities

• Design and develop ingestion and transformation pipelines in Microsoft Fabric using Data Factory pipelines, Lakehouse, OneLake, Delta tables, notebooks and Spark.
• Integrate SQL Server, Oracle/ERP, APIs and file-based sources using suitable Azure services such as Azure Data Factory, ADLS Gen2 and Azure Key Vault.
• Implement batch and incremental loads, watermark handling, schema evolution, reconciliation, error recovery and appropriate raw-data history.
• Develop Bronze, Silver and Gold datasets with clear data contracts, business keys, dimensional models and documented lineage.
• Write and tune SQL and PySpark transformations; monitor pipeline health, investigate failures and improve performance and capacity use.
• Apply access controls, secure credentials and data quality checks in collaboration with platform, security and business owners.
• Use Git, pull requests, code reviews and CI/CD pipelines to deploy Fabric and Azure data assets through development, test and production environments.
• Create technical documentation, operational runbooks and automated tests; support SIT, UAT, production releases and incident resolution.
• Collaborate with Power BI developers on curated datasets and semantic-model requirements.

Preferred candidate profile

Basic Qualifications:

• Bachelors degree in computer science, engineering, information systems or a related discipline, or equivalent practical experience.
• 45 years of hands-on experience in data engineering, including substantive delivery using Microsoft Fabric and Azure data services.

Knowledge/Skills/Abilities:

• Strong SQL, Python and PySpark skills; practical experience with data modelling, Delta Lake and distributed processing.
• Hands-on Microsoft Fabric experience with Lakehouse, OneLake, pipelines, notebooks and Warehouse; familiarity with Dataflows Gen2 and Power BI integration.
• Hands-on Azure experience with ADLS Gen2, Azure Data Factory, Key Vault and identity/access patterns; experience with on-premises connectivity is beneficial.
• Working knowledge of Git branching, pull requests, Azure DevOps or equivalent CI/CD, environment configuration and automated deployment.
• Experience with incremental ingestion, orchestration, monitoring, data quality and performance tuning.
• Clear communication, troubleshooting and documentation skills; ability to work with cross functional teams.

Essential Physical Demands/Work Environment:

• Comfortable working in a cross-cultural, matrix and agile delivery environment.