Databricks Data Engineer

Adesso SE; Pune District; On-site; INR 1,400,000 - 2,800,000; Full time

Job summary

Adesso SE is seeking a data engineer in India to design, develop, and maintain scalable batch and real-time data pipelines using Databricks, PySpark, and Spark SQL. You will implement Delta Lake-based Bronze/Silver/Gold architectures and orchestrate end-to-end workflows with scheduling tools.

The role emphasizes performance tuning, data quality, governance, and collaboration with data scientists, analysts, and stakeholders.

Qualifications

• 4-6 years of experience in Data Engineering, with strong hands-on expertise in Databricks.

Responsibilities

• Design, develop, and maintain scalable batch and real-time data pipelines.
• Build and implement Delta Lake-based solutions for Bronze/Silver/Gold layers.
• Develop and orchestrate end-to-end data workflows using Databricks Workflows and scheduling tools.
• Optimize Spark workloads for performance, scalability, and cost efficiency.
• Ensure data quality, reliability, and governance through validation, monitoring, and testing frameworks.
• Collaborate with cross-functional teams to deliver data-driven solutions.
• Contribute to code management, CI/CD, documentation, and continuous improvement.

Skills

Databricks

PySpark

SQL

Python

Spark Architecture

ETL/ELT

Education

Bachelor's degree in Computer Science/IT or related field

Tools

Delta Lake

Unity Catalog

Azure Data Factory

ADLS Gen2

Kafka

Event Hubs

Structured Streaming

Snowflake

Terraform

Git

Job description

Design, develop, and maintain scalable batch and real-time data pipelines using Databricks, PySpark, and Spark SQL.

Build and implement Delta Lake-based solutions, including data ingestion, transformation, and Medallion Architecture (Bronze, Silver, Gold) layers.

Develop and orchestrate end-to-end data workflows using Databricks Workflows and other scheduling tools.

Optimize Spark workloads for performance, scalability, and cost efficiency through effective cluster management and tuning practices.

Ensure data quality, reliability, and governance through validation, monitoring, and testing frameworks.Integrate and process data from multiple enterprise sources, including databases, APIs, files, and streaming platforms.

Collaborate with cross-functional teams, including data scientists, analysts, architects, and business stakeholders, to deliver data-driven solutions.

Contribute to code management, CI/CD implementation, documentation, and continuous improvement initiatives.

YOUR PROFILE

4-6 years of experience in Data Engineering, with strong hands-on expertise in Databricks.

Proficiency in Python, PySpark, and SQL for building scalable data processing solutions.

Strong understanding of Apache Spark architecture, Delta Lake, and modern data engineering practices.

Experience working with cloud platforms such as Azure, AWS, or GCP.Solid knowledge of ETL/ELT processes, data modeling, and data warehousing concepts.

Experience with version control systems such as Git and CI/CD processes.

Familiarity with technologies such as Unity Catalog, Azure Data Factory, ADLS Gen2, Kafka, Event Hubs, Structured Streaming, or Snowflake is an advantage.

Exposure to Infrastructure-as-Code tools such as Terraform is a plus.Databricks certifications will be considered an added advantage.

Bachelor's degree in Computer Science, Information Technology, or a related field.

adesso India, 3B-2 Athulya, Third Floor, Wing B, SEZ, Infopark Phase 1, Cochin, Kerala 682042