Senior Data Engineer Spark Scala Azure Databricks

Quess; Bengaluru; Hybrid; INR 3,500,000 - 5,500,000; Full time

Job summary

Quess is seeking an experienced Data Engineer to design, develop, and maintain scalable data pipelines in Bengaluru. The role focuses on Spark and Scala, with strong emphasis on Airflow-based orchestration and ecosystem familiarity (Azure Databricks, Delta Lake).

You will work with Hive, Kubernetes, Docker and GitHub Actions to deliver high-performance data solutions. The candidate should be comfortable with serverless compute and CI/CD practices across the SDLC.

Qualifications

• Strong hands-on experience with Spark and Scala.
• Experience in Azure Databricks ecosystem and Delta Lake.
• Proficient with Kubernetes, Docker, and GitHub Actions for data pipelines.

Responsibilities

• Design, develop and maintain data engineering solutions using Spark and Scala.
• Build and manage data pipelines with Airflow and orchestration tooling.
• Work with Hive, Kubernetes, Docker and GitHub Actions for scalable pipelines.

Skills

Spark

Scala

Airflow

Hive

Tools

Azure Databricks

ADLS

ADF

Delta Lake

Kubernetes

Docker

GitHub Actions

Job description

Key Responsibilities

• Design, develop and maintain scalable data engineering solutions using Apache Spark and Scala.
• Build and manage data pipelines using Airflow and related orchestration technologies.
• Work extensively with Hive, Kubernetes, Docker and GitHub Actions.
• Develop and optimize data solutions using Apache Iceberg.
• Design and implement data engineering solutions on Azure Databricks.
• Work with ADLS, ADF, Delta Lake, Databricks Jobs & Pipelines, Lakeflow Connect and Unity Catalog.
• Work with both Classic and Serverless Databricks Compute environments.
• Implement scalable, reliable and high-performance data pipelines.
• Follow CI/CD and DevOps best practices using GitHub Actions.
• Explore and implement AI-assisted development practices across SDLC and data engineering.
• Collaborate with cross-functional teams to deliver innovative data solutions aligned with business requirements.

Must-Have Skills

1. Spark, Scala, Airflow, Hive
2. Azure Databricks, ADLS, ADF, Delta Lake
3. Kubernetes, Docker, GitHub Actions

Strongly Preferred Skills

• Apache Iceberg
• Databricks Jobs & Pipelines
• Lakeflow Connect
• Unity Catalog
• Databricks Classic & Serverless Compute
• Azure Data Engineering
• CI/CD
• AI-assisted SDLC / AI for Data Engineering

Candidate Profile

• 6+ years of overall experience in Data Engineering.
• Strong hands-on experience with Spark and Scala.
• Good experience in Azure Databricks ecosystem.
• Strong understanding of data pipelines, orchestration and distributed data processing.
• Experience with containerization and DevOps tools.
• Ability to work from Whitefield office, Bangalore, 3 days per week.
• Candidates should be available for the Saturday interview.