Senior Data Engineer - Azure Databricks, PySpark, ADF, Kafka

Bendito Soft Tech; Hyderabad; On-site; INR 3,000,000 - 5,400,000; Full time

Job summary

Bendito Soft Tech is seeking a Senior Data Engineer to design, build and run large‑scale batch and streaming data pipelines on Azure Databricks in Hyderabad. This hands‑on role requires delivering end‑to‑end data processing with quality checks and CI/CD support.

You will implement Delta Lake architectures, craft high‑perf SQL, tune Spark jobs, and mentor junior engineers while collaborating with stakeholders to turn requirements into robust data solutions.

Qualifications

• Qualification: B.E./B.Tech/MCA/M.Sc (Computer Science or equivalent) - mandatory.
• 7+ years total IT experience.

Responsibilities

• Design and build scalable ETL/ELT pipelines on Azure Databricks using PySpark and Spark SQL
• Build and orchestrate workflows in Azure Data Factory (pipelines, linked services, triggers, parameterisation, monitoring)
• Develop real-time and near-real-time ingestion using Apache Kafka with Spark Structured Streaming
• Implement Delta Lake and medallion (bronze/silver/gold) architecture
• Write complex, high-performance SQL for transformation, validation and reporting
• Tune Spark jobs and clusters for performance and cost
• Turn stakeholder requirements into technical design
• Own testing, CI/CD deployment, production support and data-quality checks
• Document pipelines and mentor junior engineers

Skills

Azure Databricks

PySpark

Spark SQL

Azure Data Factory

Delta Lake

SQL

Education

B.E./B.Tech./MCA/M.Sc (CS or equivalent)

Tools

Unity Catalog

Delta Live Tables

Databricks Workflows

ADLS Gen2

Azure Synapse

Event Hubs

Azure DevOps

GitHub

DP-203

Airflow

Snowflake

Job description

Senior Data Engineer - Databricks | Hyderabad | Immediate joiners to 30 days only

We are hiring an experienced Senior Data Engineer for an enterprise data engineering programme with a leading global client. You will design, build and run large-scale batch and streaming data pipelines on Azure Databricks. This is a hands‑on role for someone who has built production pipelines end to end.

Roles & Responsibilities:

• - Design and build scalable ETL/ELT pipelines on Azure Databricks using PySpark and Spark SQL


• - Build and orchestrate workflows in Azure Data Factory (pipelines, linked services, triggers, parameterisation, monitoring)


• - Develop real-time and near-real-time ingestion using Apache Kafka with Spark Structured Streaming


• - Implement Delta Lake and medallion (bronze/silver/gold) architecture


• - Write complex, high-performance SQL for transformation, validation and reporting


• - Tune Spark jobs and clusters for performance and costi>
  
• - Turn stakeholder requirements into technical design


• - Own testing, CI/CD deployment, production support and data-quality checks


• - Document pipelines and mentor junior engineers

Mandatory Skills (hands-on in ALL):

• - Azure Databricks (notebooks, jobs/workflows, Delta Lake, clusters)


• - PySpark / Spark SQL with performance tuning


• - Azure Data Factory - production ETL pipelines end to end


• - Advanced SQL (complex joins, window functions, query tuning)


• - Kafka - streaming ingestion (producers/consumers, topics, Spark Structured Streaming)


• - ETL design, data modelling and data warehousing

Good to Have:

• - Unity Catalog, Delta Live Tables, Databricks Workflows


• - ADLS Gen2, Azure Synapse, Event Hubs


• - CI/CD with Azure DevOps or GitHub


• - Databricks or DP-203 certification


• - Airflow or Snowflake

Qualification: B.E./B.Tech/MCA/M.Sc (Computer Science or equivalent) - mandatory. 7+ years total IT experience.

Notice period: Immediate to 30 days only.