Senior Data Bricks Engineer

Databeat; Telangana; Presencial; INR 3.500.000 - 7.000.000; Jornada completa

Descripción de la vacante

Databeat is seeking a Senior Databricks Engineer in Hyderabad to lead end-to-end enterprise data platforms on Azure and AWS. You will architect Lakehouse solutions, build scalable PySpark/Delta Lake pipelines, and drive client engagements while mentoring teams.

The ideal candidate has 8+ years in data engineering, deep Spark knowledge, and strong governance experience with Unity Catalog and ISO 27001. This is a full-time/permanent role requiring hands-on leadership.

Formación

• 8+ years of total experience in Data Engineering and Big Data Analytics.
• 5+ years hands-on with Databricks, Apache Spark, PySpark, SQL and Delta Lake.
• Deep understanding of Spark internals, partitioning, and Medallion Architecture.
• Proven experience with Azure (ADF/Synapse) or AWS (S3/Glue/Lambda).
• Solid experience with Unity Catalog, CI/CD DevOps practices, and data governance.

Responsabilidades

• Lead end-to-end design and deployment of enterprise data platforms using Databricks, PySpark, SQL and Delta Lake.
• Engage with clients and stakeholders to gather requirements and present architectures.
• Design scalable Lakehouse/Medallion architectures and optimize Spark workloads to balance cost and speed.
• Implement Unity Catalog for data governance and ensure ISO 27001/ISMS compliance.
• Develop batch and real-time ETL pipelines integrating Databricks with Azure and AWS, and mentor teams.

Conocimientos

Databricks

PySpark

Apache Spark

Delta Lake

Python

SQL

Azure Data Factory

AWS Glue

Unity Catalog

Medallion Architecture

Lakehouse Architecture

CI/CD

ISO 27001

Educación

Bachelor’s or Master’s degree in Computer Science/IT

Herramientas

Azure Synapse

Azure Data Factory (ADF)

S3

AWS Glue

Lambda

Databricks

GitLab

GitHub

Descripción del empleo

Role - Senior Databricks Engineer

Experience Required: 8+ Years
Location: Hyderabad
Employment Type: Full Time, Permanent

Job Overview

We are seeking a Senior Databricks Engineer (8+ Years) to lead end-to-end enterprise data platform solutions across Azure and AWS environments. You will architect high-performance Lakehouse platforms, build scalable ETL/ELT pipelines using PySpark and Delta Lake, drive client/stakeholder engagements, and mentor technical teams.

Key Responsibilities

• Technical Leadership & Delivery: Lead the end-to-end design, development, and deployment of enterprise data platforms using Databricks, PySpark, SQL, and Delta Lake.
• Stakeholder Management: Conduct technical discussions, architecture reviews, effort estimations, and requirement-gathering workshops directly with clients and business stakeholders.
• Architecture & Optimization: Design scalable Lakehouse/Medallion architectures; optimize Spark applications, Databricks Workflows, and Job Clusters to minimize infrastructure costs and maximize speed.
• Governance & Security: Implement Unity Catalog for access control, metadata management, and data lineage while upholding ISO 27001 / ISMS compliance standards.
• Cloud & Pipeline Engineering: Build robust batch and real-time ETL pipelines integrating Databricks with Azure (ADF, Synapse) and/or AWS (S3, Glue, Lambda).
• DevOps & Best Practices: Drive automation via CI/CD pipelines (GitLab/GitHub), enforce coding standards, and mentor junior/mid-level data engineers.

Key Requirements

• 8+ years of total experience in Data Engineering & Big Data Analytics.
• 5+ years of hands-on expertise with Databricks, Apache Spark, PySpark, SQL, and Delta Lake.
• Deep understanding of Spark internals, cluster tuning, partitioning strategies, and Medallion Architecture.
• Proven hands-on experience with Azure (ADF/Synapse) or AWS (S3/Glue/Lambda).
• Solid experience with Unity Catalog, CI/CD DevOps practices, and data governance frameworks.
• Strong client-facing, communication, and technical team leadership capabilities.

• Education: Bachelors or Master’s degree in Computer Science, IT, or a related field.

Key Skills - Databricks PySpark Apache Spark Delta Lake Python SQL Azure Data Factory (ADF) AWS Glue Unity Catalog Medallion Architecture Data Engineering Lakehouse Architecture CI/CD ISO 27001