Senior Data Engineer

Quess IT Staffing; Bengaluru; On-site; INR 2,500,000 - 4,200,000; Full time

Job summary

Quess IT Staffing in Bengaluru seeks a Data Engineer with 6+ years of hands-on experience in building scalable data pipelines. The candidate should have strong expertise in Spark, Scala, Airflow, Hive, Kubernetes, Iceberg, and Azure Databricks.

The role also requires knowledge of modern data platforms, cloud technologies, CI/CD, and applying AI across the SDLC to drive innovation and business value.

Qualifications

• 6+ years of relevant experience in Data Engineering.
• Strong hands-on experience with Apache Spark and Scala.
• Experience with Apache Airflow and Hive.
• Proficient in Kubernetes, Iceberg, Delta Lake, and Docker.
• CI/CD experience with GitHub Actions and Databricks CI/CD.
• Hands-on with Azure Databricks, ADLS, Data Factory, and Databricks Pipelines.

Responsibilities

• Design, develop, and maintain scalable data engineering pipelines.
• Build data processing solutions using Apache Spark and Scala.
• Develop and manage workflows using Apache Airflow.
• Work with Hive, Iceberg, Kubernetes and Docker in modern data platforms.
• Develop and maintain CI/CD pipelines using GitHub Actions.
• Design and implement data solutions using Azure Databricks.
• Collaborate with data scientists, architects, product teams, and other engineers.
• Apply AI capabilities across the SDLC and Data Engineering lifecycle.

Skills

Spark

Scala

Apache Airflow

Hive

Kubernetes

Apache Iceberg

Docker

GitHub Actions/CI-CD

Azure Databricks

ADLS

Databricks Jobs & Pipelines

Azure Data Factory

Delta Lake

Lakeflow Connect

Unity Catalog

Databricks Compute Classic/Serverless

AI in SDLC/Data Engineering

Tools

Docker

GitHub Actions

Azure Databricks

Unity Catalog

Job description

Data Engineer – Azure Databricks

Job Summary

We are looking for an experienced Data Engineer with 6+ years of hands‑on experience in building and managing scalable data engineering solutions. The ideal candidate should have strong expertise in Spark, Scala, Airflow, Hive, Kubernetes, Iceberg, and Azure Databricks.

The candidate should also have a good understanding of modern data platforms, cloud technologies, CI/CD practices, and the ability to leverage AI across the Software Development Lifecycle (SDLC) and Data Engineering to drive innovation and business value.

Key Responsibilities

• Design, develop, and maintain scalable and reliable data engineering pipelines.
• Build data processing solutions using Apache Spark and Scala.
• Develop and manage workflows using Apache Airflow.
• Work with Hive, Iceberg, Kubernetes, and Docker in modern data platforms.
• Develop and maintain CI/CD pipelines using GitHub Actions.
• Design and implement data solutions using Azure Databricks.
• Work with ADLS, Azure Data Factory (ADF), Delta Lake, Databricks Jobs and Pipelines.
• Utilize Lakeflow Connect, Unity Catalog, and Databricks Compute (Classic & Serverless).
• Implement data governance, security, and access controls using Unity Catalog.
• Optimize data pipelines and processing workloads for performance and scalability.
• Collaborate with data scientists, architects, product teams, and other engineering teams.
• Apply AI capabilities and tools across the SDLC and Data Engineering lifecycle to improve productivity, automation, and business outcomes.
• Follow engineering best practices around version control, testing, deployment, monitoring, and documentation.

Required Skills

Must-Have – Priority 1

• Strong hands-on experience with Apache Spark
• Scala
• Apache Airflow
• Hive
• Kubernetes
• Apache Iceberg
• Good knowledge of Docker
• Good experience with GitHub Actions / CI-CD

Must-Have – Priority 2

• Strong experience with Azure Databricks
• ADLS
• Databricks Jobs & Pipelines
• Azure Data Factory (ADF)
• Delta Lake
• Lakeflow Connect
• Unity Catalog
• Databricks Compute – Classic & Serverless

Must-Have – Priority 3

• Experience using AI tools/capabilities within SDLC and Data Engineering
• Ability to identify opportunities to use AI for automation, development, optimization, and innovation
• Strong problem-solving and analytical skills

Additional Requirements

• 6+ years of relevant experience in Data Engineering.
• Strong understanding of data engineering concepts and distributed data processing.
• Excellent communication and collaboration skills.
• Candidates must be based in or willing to relocate to Bengaluru.
• Candidates should be comfortable working from the office
• Willingness to attend 2 technical interview rounds.

Preferred Candidate Profile

Candidates with strong hands‑on experience across Spark/Scala/Airflow and the Azure Databricks ecosystem will be given priority. Experience in modern data lakehouse technologies, containerization, CI/CD, and practical application of AI in engineering environments will be an added advantage.