USEReady Technology - Senior Data Engineer - Databricks/Azure

USEReady; Kolkata District; On-site; INR 1,500,000 - 2,600,000; Full time

Job summary

USEReady is seeking a Senior Azure Data Engineer to design and optimize scalable data platforms using Databricks, ADF, and the Azure data ecosystem in a modern Lakehouse architecture. The role emphasizes real-time and batch processing, governance via Unity Catalog, and performance tuning of Spark workloads to deliver high-quality data solutions for business stakeholders.

You'll mentor junior engineers and collaborate across teams to ensure robust, secure data pipelines and cost-efficient

Qualifications

• 5+ years of Azure Data Engineering experience.
• Experience with Databricks, ADF, and Delta Lake at scale.
• Strong knowledge of Spark performance tuning.
• Experience with data governance and security using Unity Catalog.

Responsibilities

• Design and develop scalable data pipelines using Databricks and ADF.
• Build and maintain Delta Lake tables and Lakehouse solutions for analytics and reporting.
• Implement governance and security using Unity Catalog.
• Optimize Databricks jobs, Spark workloads, and SQL queries for performance and cost efficiency.
• Develop and support real-time and batch data processing solutions.
• Configure Databricks Notification Alerts, SQL dashboards, and Genie Spaces to improve monitoring, self-service analytics, and business user adoption.
• Monitor, troubleshoot, and improve data pipeline reliability and quality.
• Collaborate with stakeholders to deliver business-focused data solutions.
• Mentor junior engineers and promote data engineering best practices.

Skills

Databricks

ADF

Python

PySpark

SQL

Azure Synapse

Delta Lake

Unity Catalog

Kafka

CI/CD

Tools

Azure DevOps

GitHub

Genie Spaces

DAB

Lakeflow Connect

Job description

Role Overview

We are looking for a Senior Azure Data Engineer with strong expertise in Databricks and Azure data services. The role involves designing, building, and optimizing scalable, secure, and high-performance data platforms, leveraging modern Lakehouse architecture and real-time data processing capabilities.

Role Overview

We are looking for a Senior Azure Data Engineer with strong expertise in Databricks and Azure data services. The role involves designing, building, and optimizing scalable, secure, and high-performance data platforms, leveraging modern Lakehouse architecture and real-time data processing capabilities.

Key Skills

• Databricks : Notebooks (Python, SQL), SQL Editor, Unity Catalog, Delta Lake, Databricks Notification Alerts, Genie Spaces, and performance optimization.
• Azure Data Factory (ADF) for ETL/ELT orchestration.
• Advanced Python, PySpark, and SQL skills with strong understanding of Spark performance tuning.
• Azure Ecosystem : Azure Data Lake Storage (ADLS Gen2), Azure Synapse Analytics, Azure SQL Database, and Azure Event Hub.
• Real-time data ingestion : Structured Streaming, Kafka, Event Hubs, and Auto Loader.
• Architecture : Spark Declarative Pipelines (SDP), Delta Live Tables (DLT), and Medallion Architecture.
• Governance : Data modeling, data warehousing, governance, lineage, and access control.
• DevOps : GitHub, GitHub Actions, Azure DevOps, and CI/CD practices.
• Familiarity with Databricks Asset Bundles (DAB) and Lakeflow Connect.

Responsibilities

• Design and develop scalable data pipelines using Databricks and ADF.
• Build and maintain Delta Lake tables and Lakehouse solutions for analytics and reporting.
• Implement governance and security using Unity Catalog.
• Optimize Databricks jobs, Spark workloads, and SQL queries for performance and cost efficiency.
• Develop and support real-time and batch data processing solutions.
• Configure Databricks Notification Alerts, SQL dashboards, and Genie Spaces to improve monitoring, self-service analytics, and business user adoption.
• Monitor, troubleshoot, and improve data pipeline reliability and quality.
• Collaborate with stakeholders to deliver business-focused data solutions.
• Mentor junior engineers and promote data engineering best practices.

Preferred Qualifications

• 5+ years of Azure Data Engineering experience.
• Experience with large-scale distributed data processing systems.
• Databricks and/or Azure Data Engineering certifications preferred.
• Strong communication, collaboration, and problem-solving skills.

(ref:hirist.tech)