Senior Databricks Data Engineer

Adesso SE; Pune District; On-site; INR 900,000 - 1,400,000; Full time

Job summary

adesso India is seeking an experienced Data Engineer to design and implement enterprise-scale data solutions on the Databricks Lakehouse platform. You will build scalable batch and real-time data pipelines using PySpark, Spark SQL, and Structured Streaming, while governing security and data access with Unity Catalog.

You will also drive CI/CD, DevOps practices, and Infrastructure-as-Code, mentor teammates, and collaborate with global stakeholders to ensure reliable, cost-efficient data platforms.

Qualifications

• 6–8 years of Data Engineering experience with hands-on Databricks.
• Advanced Python, PySpark, Spark SQL, and large-scale processing.
• Strong experience with Delta Lake, Unity Catalog, Databricks Workflows.
• Cloud data platform design on Azure/AWS/GCP; Lakehouse architecture.
• Experience with Kafka, Event Hubs, and Structured Streaming.
• Git, DevOps, CI/CD, and Infrastructure-as-Code knowledge.
• Mentoring and collaboration with global stakeholders.
• Bachelor's or Master's in CS/IT.

Responsibilities

• Design and deliver enterprise-scale data solutions on Databricks Lakehouse.
• Lead scalable batch and real-time data pipelines with PySpark.
• Govern data security, lineage, and access using Unity Catalog.
• Optimize platform performance and control costs.
• Establish CI/CD pipelines and IaC implementations.
• Provide technical leadership through code reviews and mentorship.
• Collaborate with architects, stakeholders, and delivery teams.
• Troubleshoot production issues ensuring reliability.
• Support solution architecture and project delivery activities.

Skills

Python

PySpark

Spark SQL

Large-scale data processing

Leadership & mentoring

CI/CD practices

Infrastructure-as-Code

Data governance

Education

Bachelor's or Master's degree in Computer Science or IT

Tools

Databricks

Delta Lake

Unity Catalog

Databricks Workflows

Delta Live Tables

Kafka

Event Hubs

Structured Streaming

Job description

• Design and deliver enterprise-scale data solutions using Databricks Lakehouse architecture and Delta Lake.
• Lead the development of scalable batch and real-time data pipelines using PySpark, Spark SQL, and Structured Streaming.
• Implement and govern data security, lineage, access management, and compliance standards using Unity Catalog.
• Drive platform performance optimization, scalability improvements, and cost management initiatives.
• Establish and enhance CI/CD pipelines, DevOps practices, automated testing frameworks, and Infrastructure-as-Code implementations.
• Provide technical leadership through code reviews, mentoring, and engineering best practices.
• Partner with architects, business stakeholders, and delivery teams to translate business requirements into scalable data solutions.
• Troubleshoot and resolve complex production issues while ensuring platform reliability and operational excellence.
• Support solution architecture, technical estimations, framework development, and project delivery activities.

YOUR PROFILE

• 6-8 years of experience in Data Engineering, including significant hands-on experience with Databricks.Advanced expertise in Python, PySpark, Spark SQL, and large-scale data processing frameworks.
• Strong experience with Delta Lake, Unity Catalog, Databricks Workflows, and Delta Live Tables.
• Proven experience designing and implementing cloud-based data platforms on Azure, AWS, or GCP.In-depth understanding of data modeling, data warehousing, and modern Lakehouse architectures.
• Experience working with real-time data technologies such as Kafka, Event Hubs, and Structured Streaming.Strong knowledge of Git, DevOps practices, CI/CD pipelines, and Infrastructure-as-Code methodologies.
• Demonstrated ability to lead technical initiatives, mentor team members, and collaborate with global stakeholders.
• Exposure to MLflow, Microsoft Fabric, Azure Synapse, Snowflake, and enterprise data governance frameworks is desirable.
• Bachelor's or Master's degree in Computer Science, Information Technology, or a related discipline.

adesso India, 3B-2 Athulya, Third Floor, Wing B, SEZ, Infopark Phase 1, Cochin, Kerala 682042