Senior Databricks Engineer/Architect

Databeat; Hyderabad; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

Databeat is seeking a Senior Databricks Engineer with deep hands-on expertise in Data Engineering, Big Data, Cloud Analytics, and enterprise data platforms. The candidate should be proficient with Databricks, Apache Spark, PySpark, SQL, Delta Lake, and cloud platforms (Azure/AWS).

You will design scalable data solutions, lead end-to-end data projects from requirements to deployment, and collaborate with clients, architects, and stakeholders to deliver high-performance Lakehouse architectures and

Qualifications

• 8+ years of experience in Data Engineering and Big Data technologies.
• Hands-on with Databricks, Spark, PySpark, and SQL in cloud environments.
• Experience with Azure/AWS platforms and enterprise-scale data solutions.
• Strong understanding of Spark optimization and Delta Lake.

Responsibilities

• Design and develop scalable data engineering solutions using Databricks, Spark, and Delta Lake.
• Lead end-to-end data projects from requirement gathering to deployment and production support.
• Work closely with clients, architects, and business stakeholders to deliver solutions.
• Design and optimize Lakehouse architectures and high-performance ETL pipelines.
• Implement data governance, security, monitoring, and performance optimization.
• Mentor and guide data engineering teams through technical leadership and code reviews.

Skills

Databricks

PySpark

Apache Spark

SQL

Delta Lake

Azure

AWS

ETL Pipelines

Data Modeling

CI/CD

Data Governance

Python

Power BI

Education

Bachelor's/Master's in CS or related

Tools

Azure Data Factory

Azure Synapse

AWS Glue

Lambda

Databricks Workflows

Job description

Job Title: Senior Databricks Engineer
Experience: 8+ Years
Location: Hyderabad
Employment Type: Full-Time

Job Summary:
We are looking for a Senior Databricks Engineer with strong experience in Data Engineering, Big Data, Cloud Analytics, and enterprise data platforms. The candidate should have hands-on expertise in Databricks, Apache Spark, PySpark, SQL, Delta Lake, and cloud platforms (Azure/AWS).

Key Skills Required

• Databricks, PySpark, Apache Spark, SQL, Delta Lake


• Python, Azure Data Factory (ADF), Azure Synapse


• AWS S3, Glue, Lambda


• Unity Catalog, MLflow, Databricks Workflows & Job Clusters


• Lakehouse Architecture, Medallion Architecture


• Data Modeling, ETL/ELT Pipelines, Performance Optimization


• CI/CD, GitHub/GitLab, DevOps Practices


• Data Governance, Data Quality, Power BI/Tableau

Responsibilities

• Design and develop scalable data engineering solutions using Databricks, Spark, and Delta Lake.


• Lead end-to-end data projects from requirement gathering to deployment and production support.


• Work closely with clients, architects, and business stakeholders to deliver solutions.


• Design and optimize Lakehouse architectures and high-performance ETL pipelines.


• Implement data governance, security, monitoring, and performance optimization.


• Mentor and guide data engineering teams through technical leadership and code reviews.

Required Experience

• 8+ years of experience in Data Engineering and Big Data technologies.


• 5+ years of hands-on experience with Databricks, Spark, PySpark, and SQL.


• Experience with Azure/AWS cloud data platforms and enterprise-scale data solutions.


• Strong understanding of Spark optimization, Delta Lake, data architecture, and CI/CD practices.


• Excellent communication and stakeholder management skills.

Education

Bachelors/Master's degree in Computer Science, Engineering, IT, or related field.