USEReady Technology - Senior Data Engineer - Databricks/Azure

USEReady; Kolkata District; On-site; INR 3,500,000 - 6,000,000; Full time

Job summary

USEReady is seeking a Senior Azure Data Engineer to design, build, and optimize scalable data platforms using Databricks and Azure data services. You will implement Lakehouse architectures, real-time processing, and governance across data layers, ensuring high performance and cost-efficiency.

You will mentor junior engineers, collaborate with stakeholders, and drive best practices in data engineering, security, and CI/CD integration. A strong background in Spark, Python, and SQL is required.

Qualifications

• 5+ years of Azure Data Engineering experience.
• Experience with large-scale distributed data processing systems.
• Databricks and/or Azure Data Engineering certifications preferred.
• Strong communication, collaboration, and problem-solving skills.

Responsibilities

• Design and develop scalable data pipelines using Databricks and ADF.
• Build and maintain Delta Lake tables and Lakehouse solutions for analytics and reporting.
• Implement governance and security using Unity Catalog.
• Optimize Databricks jobs, Spark workloads, and SQL queries for performance and cost efficiency.
• Develop and support real-time and batch data processing solutions.
• Configure Databricks Notification Alerts, SQL dashboards, and Genie Spaces to improve monitoring, self-service analytics, and business user adoption.
• Monitor, troubleshoot, and improve data pipeline reliability and quality.
• Collaborate with stakeholders to deliver business-focused data solutions.
• Mentor junior engineers and promote data engineering best practices.

Skills

Databricks

Python

PySpark

SQL

Spark tuning

Unity Catalog

Delta Lake

Lakehouse

Real-time processing

ADF (Azure Data Factory)

Azure ecosystem

Data governance

DevOps

CI/CD

Tools

Azure Data Factory

GitHub

GitHub Actions

Azure DevOps

Job description

Role Overview

We are looking for a Senior Azure Data Engineer with strong expertise in Databricks and Azure data services. The role involves designing, building, and optimizing scalable, secure, and high-performance data platforms, leveraging modern Lakehouse architecture and real-time data processing capabilities.

Key Skills

• Databricks : Notebooks (Python, SQL), SQL Editor, Unity Catalog, Delta Lake, Databricks Notification Alerts, Genie Spaces, and performance optimization.
• Azure Data Factory (ADF) for ETL/ELT orchestration.
• Advanced Python, PySpark, and SQL skills with strong understanding of Spark performance tuning.
• Azure Ecosystem : Azure Data Lake Storage (ADLS Gen2), Azure Synapse Analytics, Azure SQL Database, and Azure Event Hub.
• Real-time data ingestion : Structured Streaming, Kafka, Event Hubs, and Auto Loader.
• Architecture : Spark Declarative Pipelines (SDP), Delta Live Tables (DLT), and Medallion Architecture.
• Governance : Data modeling, data warehousing, governance, lineage, and access control.
• DevOps : GitHub, GitHub Actions, Azure DevOps, and CI/CD practices.
• Familiarity with Databricks Asset Bundles (DAB) and Lakeflow Connect.

Responsibilities

• Design and develop scalable data pipelines using Databricks and ADF.
• Build and maintain Delta Lake tables and Lakehouse solutions for analytics and reporting.
• Implement governance and security using Unity Catalog.
• Optimize Databricks jobs, Spark workloads, and SQL queries for performance and cost efficiency.
• Develop and support real-time and batch data processing solutions.
• Configure Databricks Notification Alerts, SQL dashboards, and Genie Spaces to improve monitoring, self-service analytics, and business user adoption.
• Monitor, troubleshoot, and improve data pipeline reliability and quality.
• Collaborate with stakeholders to deliver business-focused data solutions.
• Mentor junior engineers and promote data engineering best practices.

Preferred Qualifications

• 5+ years of Azure Data Engineering experience.
• Experience with large-scale distributed data processing systems.
• Databricks and/or Azure Data Engineering certifications preferred.
• Strong communication, collaboration, and problem-solving skills.

(ref:hirist.tech)