Lufthansa Technik Services - Data Engineer - Azure & Databricks

Lufthansa Technik Services India Pvt Ltd; Bengaluru; On-site; INR 800,000 - 1,200,000; Full time

Job summary

Lufthansa Technik Services India Pvt Ltd is hiring for a role focused on designing and maintaining data transformation pipelines. The position demands strong Python skills, particularly with PySpark, and proficiency in Azure Databricks, ensuring high-quality and reliable data pipelines.

Ideal candidates will have experience in cloud-based ETL platforms and a foundation in data engineering principles. Collaboration with data science teams is essential to support analytics workflows and provide scalable solutions to complex data problems.

Qualifications

• Strong expertise in Python with frameworks like PySpark.
• Solid foundation in data engineering principles and large-scale data processing.
• Experience with Azure Databricks and cloud-based ETL platforms.

Responsibilities

• Design, develop, and maintain scalable data transformation pipelines.
• Collaborate with Data Scientists and Data Analysts for data solutions.
• Build and manage Databricks notebooks, jobs, and orchestrated workflows.

Skills

Python (with frameworks like PySpark)

SQL

Azure Databricks

Data Engineering principles

CI/CD pipelines

DevOps practices

Tools

Azure Functions

AWS

Google Cloud

Kubernetes/OpenShift

Job description

Job Description

Responsible for designing, developing, and maintaining scalable data transformation pipelines using Python and SQL in Azure Databricks, ensuring data reliability, quality, performance, and reusability.

Data Engineering & Transformation

• Design, develop, and maintain scalable data transformation pipelines using Python (with PySpark, ADF) and SQL in Azure Databricks.
• Implement transformation logic to move data from Bronze to Silver/Gold layers following data engineering best practices.
• Apply strong data engineering principles to ensure data reliability, quality, performance, and reusability.
• Work with structured and semi‑structured data at scale.

Databricks, Azure & Cloud ETL

• Build and manage Databricks notebooks, jobs, Delta Lake tables, and orchestrated workflows.
• Hands‑on experience with Cloud‑based ETL platforms (Preferred: Microsoft Azure Databricks, Synapse, Azure Functions; otherwise AWS or Google Cloud).
• Optimize data pipelines for performance, scalability, and cost efficiency.

Python Applications, APIs & Automation

• Design, develop, and maintain Python applications, scripts, and APIs for data processing and automation.
• Write production‑grade Python code with a strong focus on readability, maintainability, and testing.
• Leverage Python for orchestration, validation, and integration with downstream systems.

Collaboration With Data Science & Engineering Teams

• Collaborate closely with Data Scientists and Data Analysts to understand data, analytical models, and consumption requirements.
• Enable and support advanced analytics and data science workflows by preparing high‑quality feature datasets.
• Translate analytical needs into scalable data engineering solutions.

CI/CD, DevOps & Platform Engineering

• Build and maintain automated CI/CD pipelines for data and Databricks workloads.
• Hands‑on experience with DevOps tools and practices, including Git‑based version control.
• Exposure to containerization and orchestration platforms such as Kubernetes/OpenShift.
• Ensure smooth promotion of code and pipelines across environments (Dev/Test/Prod).

Data Modeling & Querying

• Design and implement robust data models optimized for analytics and reporting.
• Strong hands‑on knowledge of SQL and exposure to KQL or other query languages.
• Apply best practices in data structures, indexing, and performance tuning.

UI / UX & Data Applications (Additional Advantage)

• Open to contributing to data‑driven UI/UX components, dashboards, or lightweight data applications.
• Work with analytics and business teams to improve data usability and customer experience.

Must‑Have

• Strong hands‑on expertise in Python (with frameworks like PySpark).
• Solid foundation in Data Engineering principles and large‑scale data processing.
• Experience with Azure Databricks and cloud‑based ETL platforms.
• Strong knowledge of SQL and data querying techniques.
• Experience with CI/CD pipelines and DevOps practices.
• Experience in pipeline monitoring and alerting.
• Ability to design efficient, scalable solutions to complex data problems.

Preferred Candidate Profile

• Experience with Azure Synapse, Azure Functions.
• Exposure to AWS or Google Cloud data platforms.
• Hands‑on experience with OpenShift.
• Knowledge of data science concepts and workflows.
• Familiarity with analytics platforms, dashboards, and UI/UX considerations.