Senior Databricks Engineer

DataBeat; Hyderabad; On-site; INR 2,500,000 - 4,200,000; Full time

Job summary

DataBeat seeks an experienced Senior Databricks Developer in Hyderabad to lead enterprise data engineering initiatives. You will own design, development and deployment of scalable data platforms using Delta Lake, PySpark, SQL and Spark, across Azure and AWS environments.

You will mentor junior engineers, engage with clients and stakeholders, drive governance and CI/CD practices, and ensure high performance, security and reliability of data pipelines and lakehouse architectures.

Qualifications

• 8+ years in data engineering, big data and analytics platforms.
• Hands-on with Databricks, Spark, PySpark and SQL.
• Experience designing lakehouse/data warehouse solutions.
• Experience with Azure/AWS data services and governance.

Responsibilities

• Lead end-to-end data engineering projects using Databricks, PySpark and Delta Lake.
• Engage clients and stakeholders; translate requirements into scalable solutions.
• Mentor development teams and drive architectural reviews.
• Design scalable lakehouse architectures across Azure and AWS.
• Develop high-performance ETL/ELT pipelines for batch and streaming data.
• Enforce coding standards, governance, and CI/CD practices.
• Optimize Databricks workloads; manage clusters and Spark applications.
• Implement Unity Catalog governance, access controls and lineage.
• Collaborate with data scientists and BI teams on analytics use cases.
• Lead IaC/DevOps automation and establish observability frameworks.

Skills

Databricks

Apache Spark

PySpark

SQL

Delta Lake

Azure

AWS

Data Governance

Education

Bachelor's or Master's in Computer Science/Engineering/IT

Tools

Azure Data Factory

Azure Synapse Analytics

AWS S3

Glue

Lambda

Unity Catalog

Databricks Job Clusters

CI/CD Pipelines

GitLab / GitHub

Job description

We are seeking an experienced Senior Databricks Developer with 8+ years of experience in Data Engineering, Big Data, and Cloud Analytics solutions, including extensive hands-onexpertise in Databricks, Apache Spark, PySpark, SQL, and Delta Lake. The ideal candidate will be responsible for leading end-to-end data engineering projects, engaging directly with clients and business stakeholders, mentoring development teams, and delivering scalable, secure, and high-performance data platforms across Azure and AWS environments.

This role requires strong technical leadership, solution design capabilities, stakeholder management skills, and the ability to drive data transformation initiatives from requirements gathering through deployment and production support.

Technical Expertise

Data Factory (ADF) ,Azure Synapse Analytics ,AWS S3, Glue, Lambda , Unity Catalog

MLflow ,Databricks Job Clusters ,CI/CD Pipelines ,GitLab / GitHub ,Data Governance & Data

Quality Frameworks, Power BI / Tableau , Data Cataloging ,Lakehouse Architecture ,Medallion

Architecture, Performance Tuning & Optimization

Key Responsibilities

• Lead the design, development, and implementation of enterprise-scale data engineering solutions using Databricks, PySpark, SQL, and Delta Lake. development, testing, deployment, and production support.
• Collaborate directly with clients, business stakeholders, architects, and product owners to understand business requirements and translate them into scalable technical solutions.
• Conduct client discussions, solution workshops, effort estimations, technical presentations, and architecture reviews.
• Design and implement scalable Lakehouse architectures and data platforms across Azure and AWS cloud environments.
• Lead the development of high-performance ETL/ELT pipelines supporting both batch and
• Drive best practices around coding standards, architecture governance, version control, CI/CD implementation, and operational excellence.
• Architect and optimize Databricks Workflows, Job Clusters, Delta Tables, and Spark applications to maximize performance and minimize infrastructure costs.
• Implement data governance frameworks using Unity Catalog, ensuring proper access controls, lineage tracking, metadata management, and compliance standards.
• Mentor and guide junior and mid-level data engineers through code reviews, technical coaching, and knowledge-sharing initiatives.
• Lead technical teams and coordinate project activities to ensure timely and successful project delivery.
• Collaborate with Data Scientists, BI Teams, and Analytics stakeholders to enable advanced analytics and machine learning use cases.
• Drive automation initiatives through Infrastructure as Code (IaC), DevOps practices, and
• Establish monitoring, observability, and performance tracking frameworks for data platforms and pipelines.
• Ensure security, compliance, data quality, and operational reliability across enterprise data ecosystems.
• Prepare and maintain technical architecture documents, design specifications, implementation guides, and operational runbooks.

Requirements

• 8+ years of experience in Data Engineering, Big Data, and Analytics platforms.
• Minimum 5+ years of hands-on experience working with Databricks, Apache Spark, PySpark, and SQL.
• Strong expertise in designing and implementing enterprise-scale Data Lake, Lakehouse, and Data Warehouse solutions.
• Deep understanding of Spark internals, cluster management, performance tuning, partitioning strategies, and resource optimization.
• Extensive experience working with Delta Lake, schema evolution, ACID transactions, and
• Hands-on experience integrating Databricks with Azure (ADF, Synapse) and/or AWS (S3, Glue, Lambda) services.
• Proven experience handling end-to-end project delivery and managing technical engagements with clients and stakeholders.
• Experience leading development teams and mentoring engineers in enterprise environments.
• Strong understanding of data modeling, dimensional modeling, Medallion Architecture, and
• Expertise in implementing CI/CD pipelines, DevOps practices, and automated deployment strategies.
• Experience with Unity Catalog or equivalent governance platforms.
• Excellent communication, stakeholder management, presentation, and client-facing skills.
• Ability to lead technical discussions, architecture reviews, and solution design workshops.
• Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or a related field.

Information Security & ISO 27001 Compliance

Security is at the heart of everything we do. In this role, you will be strictly required to uphold our Information Security Management System (ISMS) policies in alignment with ISO/IEC 27001 standards. Responsibilities include safeguarding sensitive asset data, completing mandatory security awareness training, reporting potential security incidents or vulnerabilities immediately, and ensuring that daily operations comply with our rigorous data protection protocols.