Azure Data Engineer

Texplorers

Hyderabad

Hybrid

INR 4,000,000 - 7,000,000

Full time

4 days ago

Be an early applicant

Application generator

A complete application in a minute — tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Texplorers in Hyderabad is seeking an experienced Databricks Developer to design, build, and optimize data pipelines on the Databricks Lakehouse Platform. You will collaborate with data engineers, analysts, and stakeholders to transform raw data into actionable insights, ensuring performance, scalability, and reliability.

The role requires 8+ years in data engineering, strong Python/Scala, Spark SQL, Delta Lake, and cloud experience (Azure/AWS/GCP).

Qualifications

8+ years total experience in data engineering or related roles
5+ years hands-on with Databricks and Apache Spark
Strong Python and/or Scala proficiency
Experience with cloud platforms (Azure, AWS, or GCP) and data warehousing concepts

Responsibilities

Design, build, and maintain scalable ETL/ELT pipelines using Databricks, Spark, and Delta Lake
Develop and optimize PySpark and Spark SQL jobs for large-scale datasets
Ingest data from REST APIs, streaming platforms, and cloud storage into Lakehouse
Build and manage Unity Catalog for data governance, lineage, security, and access control
Collaborate with data scientists to enable ML workflows, feature stores, and real-time analytics
Automate workflows using Databricks Workflows, Jobs API, and CI/CD with Azure DevOps or GitHub Actions
Monitor cluster performance, troubleshoot bottlenecks, and optimize costs
Ensure data quality, validation, and testing frameworks across pipelines

Skills

Databricks

Apache Spark

Python

Scala

Spark SQL

Delta Lake

REST APIs

ML workflows

CI/CD

Git

Education

Bachelor’s degree in CS/Engineering

Tools

Unity Catalog

Delta Live Tables

dbt

Kafka

Event Hubs

Azure Data Factory

Power BI

Tableau

Azure DevOps

GitHub Actions

Job description

Position Summary

We are seeking a skilled Databricks Developer to design, develop, and optimize data
pipelines and analytics solutions on the Databricks Lakehouse Platform. The ideal
candidate will have 8 years of total experience in data engineering and analytics, with
proven expertise in building robust data pipelines, implementing advanced analytics, and enabling machine learning workflows. You will collaborate with data engineers, analysts, and stakeholders to transform raw data into actionable insights, ensuring performance, scalability, and reliability.

Key Responsibilities

Design, build, and maintain scalable ETL/ELT pipelines using Databricks, Apache
Spark, and Delta Lake.
Develop and optimize PySpark and Spark SQL jobs for performance, cost, and
reliability on large-scale datasets.
Implement data ingestion from various sources including REST APIs, streaming
platforms, and cloud storage into the Lakehouse architecture.
Build and manage Unity Catalog for data governance, lineage, security, and access
control.
Collaborate with data scientists and analysts to enable ML workflows, feature stores, and real-time analytics using Databricks.
Automate workflows using Databricks Workflows, Jobs API, and CI/CD pipelines with Azure DevOps or GitHub Actions.
Monitor cluster performance, troubleshoot bottlenecks, and implement best practices for cost optimization.
Ensure data quality, validation, and testing frameworks are implemented across all
pipelines.

Required Qualifications

Technical Skills