Senior Databricks Data Engineer

Adesso SE; Bengaluru; On-site; INR 5,000,000 - 9,000,000; Full time

Job summary

Adesso SE invites a Senior Data Engineer to design and deliver enterprise-scale data solutions on Databricks Lakehouse using Delta Lake. You will build scalable batch and real-time pipelines with PySpark, Spark SQL, and Structured Streaming while governing security, lineage, and access via Unity Catalog.

You will lead technical initiatives, mentor teammates, and collaborate with global stakeholders to ensure reliable, cost-efficient platform delivery across Azure, AWS, or GCP environments.

Qualifications

• 6-8 years of experience in Data Engineering with Databricks and hands-on Python, PySpark, Spark SQL.
• Strong experience with Delta Lake, Unity Catalog, Databricks Workflows, and Delta Live Tables.
• Experience designing cloud-based data platforms on Azure, AWS, or GCP; data modeling and Lakehouse architectures.
• Knowledge of real-time data tech like Kafka, Event Hubs, and Structured Streaming; Git and DevOps practices.
• Ability to lead technical initiatives, mentor teammates, and collaborate with global stakeholders.
• Desirable exposure to MLflow, Microsoft Fabric, Azure Synapse, Snowflake, and data governance frameworks.

Responsibilities

• Design and deliver enterprise-scale data solutions using Databricks Lakehouse and Delta Lake.
• Lead scalable batch and real-time data pipelines with PySpark, Spark SQL, and Structured Streaming.
• Govern data security, lineage, access management, and compliance using Unity Catalog.
• Drive platform performance, scalability, and cost management initiatives.
• Establish and enhance CI/CD pipelines, DevOps practices, automated testing, and IaC implementations.
• Provide technical leadership via code reviews, mentoring, and engineering best practices.
• Partner with architects, stakeholders, and delivery teams to translate requirements into scalable data solutions.
• Troubleshoot complex production issues to ensure platform reliability and operational excellence.

Skills

Databricks

Python

PySpark

Spark SQL

Structured Streaming

Delta Lake

Unity Catalog

Databricks Workflows

Delta Live Tables

Kafka

Event Hubs

Git

DevOps

CI/CD

Infrastructure-as-Code

MLflow

Azure Synapse

Snowflake

Cloud Platforms (Azure/AWS/GCP)

Education

Bachelor's or Master's degree in Computer Science/Information Technology/related discipline

Tools

Databricks

Delta Lake

Unity Catalog

Git

Terraform / IaC

CI/CD tools

Job description

• Design and deliver enterprise-scale data solutions using Databricks Lakehouse architecture and Delta Lake.
• Lead the development of scalable batch and real-time data pipelines using PySpark, Spark SQL, and Structured Streaming.
• Implement and govern data security, lineage, access management, and compliance standards using Unity Catalog.
• Drive platform performance optimization, scalability improvements, and cost management initiatives.
• Establish and enhance CI/CD pipelines, DevOps practices, automated testing frameworks, and Infrastructure-as-Code implementations.
• Provide technical leadership through code reviews, mentoring, and engineering best practices.
• Partner with architects, business stakeholders, and delivery teams to translate business requirements into scalable data solutions.
• Troubleshoot and resolve complex production issues while ensuring platform reliability and operational excellence.
• Support solution architecture, technical estimations, framework development, and project delivery activities.

YOUR PROFILE

• 6-8 years of experience in Data Engineering, including significant hands-on experience with Databricks.Advanced expertise in Python, PySpark, Spark SQL, and large-scale data processing frameworks.
• Strong experience with Delta Lake, Unity Catalog, Databricks Workflows, and Delta Live Tables.
• Proven experience designing and implementing cloud-based data platforms on Azure, AWS, or GCP.In-depth understanding of data modeling, data warehousing, and modern Lakehouse architectures.
• Experience working with real-time data technologies such as Kafka, Event Hubs, and Structured Streaming.Strong knowledge of Git, DevOps practices, CI/CD pipelines, and Infrastructure-as-Code methodologies.
• Demonstrated ability to lead technical initiatives, mentor team members, and collaborate with global stakeholders.
• Exposure to MLflow, Microsoft Fabric, Azure Synapse, Snowflake, and enterprise data governance frameworks is desirable.
• Bachelor's or Master's degree in Computer Science, Information Technology, or a related discipline.

adesso India, 3B-2 Athulya, Third Floor, Wing B, SEZ, Infopark Phase 1, Cochin, Kerala 682042