Data Engineer ( Azure & Databricks )

Lufthansa Technik AG; Bengaluru; Hybrid; INR 1,800,000 - 3,200,000; Full time

Job summary

Lufthansa Technik AG is seeking a Data Engineer to design and maintain scalable data transformation pipelines using Python and SQL, integrated with Azure Databricks and Delta Lake. You will move data across Bronze to Silver/Gold layers and optimize for performance and cost.

You will collaborate with Data Science teams to prepare feature datasets and enable analytics workflows, while building CI/CD pipelines and DevOps practices for data workloads.

Qualifications

• Strong hands-on Python with data processing experience.
• Proven PySpark experience in large-scale ETL.
• Solid SQL skills for analytics and reporting.
• Experience building CI/CD pipelines and DevOps practices.
• Familiarity with cloud data platforms (Azure AWS GCP).
• Ability to design scalable data pipelines and data models.

Responsibilities

• Design, develop, and maintain scalable data transformation pipelines (Python/SQL).
• Move data Bronze to Silver/Gold layers following best practices.
• Collaborate with Data Science to prepare feature datasets.
• Build and manage Databricks notebooks, jobs, and Delta Lake tables.
• Implement CI/CD and platform automation for data workloads.
• Optimize pipelines for performance, cost, and reliability.
• Contribute to data-driven UI/UX components and dashboards.

Skills

Python

PySpark

SQL

CI/CD

DevOps

Data modeling

Cloud ETL

Data pipeline design

Monitoring & alerting

Tools

Azure Databricks

Delta Lake

Kubernetes

OpenShift

Azure Synapse

Azure Functions

AWS / Google Cloud

Job description

Roles & responsibilities :

Data Engineering & Transformation

• Design, develop, and maintain scalable data transformation pipelines using Python (with tools like PySpark, ADF) and SQL in Azure Databricks.
• Implement transformation logic to move data from Bronze to Silver/Gold layers following data engineering best practices.
• Apply strong data engineering principles to ensure data reliability, quality, performance, and reusability.
• Work with structured and semi-structured data at scale.

Databricks, Azure & Cloud ETL

• Build and manage Databricks notebooks, jobs, Delta Lake tables, and orchestrated workflows.
• Hands-on experience with Cloud-based ETL platforms.
• (Preferred: Microsoft Azure Databricks, Synapse, Azure Functions; otherwise AWS or Google Cloud.
• Optimize data pipelines for performance, scalability, and cost efficiency.

Python Applications, APIs & Automation

• Design, develop, and maintain Python applications, scripts, and APIs for data processing and automation.
• Write production-grade Python code with strong focus on readability, maintainability, and testing.
• Leverage Python for orchestration, validation, and integration with downstream systems.

Collaboration with Data Science & Engineering Teams

• Collaborate closely with Data Scientists and Data Analysts to understand data, analytical models, and consumption requirements.
• Enable and support advanced analytics and data science workflows by preparing high-quality feature datasets.
• Translate analytical needs into scalable data engineering solutions.

CI/CD, DevOps & Platform Engineering

• Build and maintain automated CI/CD pipelines for data and Databricks workloads.
• Hands-on experience with DevOps tools and practices, including Git-based version control.
• Exposure to containerization and orchestration platforms such as Kubernetes / OpenShift.
• Ensure smooth promotion of code and pipelines across environments (Dev/Test/Prod).

Data Modeling & Querying

• Design and implement robust data models optimized for analytics and reporting.
• Strong hands-on knowledge of SQL and exposure to KQL or other query languages.
• Apply best practices in data structures, indexing, and performance tuning UI / UX & Data Applications (Additional Advantage).
• Open to contributing to data-driven UI/UX components, dashboards, or lightweight data applications.
• Work with analytics and business teams to improve data usability and customer experience.

Desired candidate profile

• Strong hands-on expertise in Python (with frameworks like PySpark).
• Solid foundation in Data Engineering principles and large-scale data processing.
• Experience with Azure Databricks and cloud-based ETL platforms.
• Strong knowledge of SQL and data querying techniques.
• Experience with CI/CD pipelines and DevOps practices.
• Experience in pipeline monitoring and alerting.
• Ability to design efficient, scalable solutions to complex data problems.

Nice to have:-

• Experience with Azure Synapse and Azure Functions.
• Exposure to AWS or Google Cloud data platforms.
• Hands-on experience with OpenShift.
• Knowledge of data science concepts and workflows.
• Familiarity with analytics platforms, dashboards, and UI/UX considerations.