Databricks Engineer

DATAECONOMY Inc; Hyderabad; On-site; INR 1,800,000 - 3,200,000; Full time

Benefits offered by this job

Health insurance

Group Personal Accident Insurance

Group Term Life Insurance

Hybrid work options

21 days annual leave + holidays

Break-out area with refreshments

Job summary

DATAECONOMY Inc in Hyderabad is seeking a Databricks Engineer to design, develop, and optimize scalable data solutions on the Databricks platform. The role focuses on building robust data pipelines, Delta Lake, and Lakehouse architectures.

The ideal candidate will manage Databricks clusters, implement Medallion Architecture, and ensure data quality while collaborating with data architects and analysts in a hybrid work setting.

Qualifications

• 5-10 years of experience in Data Engineering or Big Data technologies.
• Strong understanding of Databricks architecture and core platform capabilities.
• Hands-on experience in setting up and managing Databricks clusters.
• Strong expertise in Apache Spark (PySpark/Scala).
• Experience developing and deploying ETL pipelines in Databricks.
• Experience implementing Medallion Architecture within Databricks.
• Hands-on experience with Delta Live Tables (DLT).
• Strong understanding and implementation experience with Unity Catalog.
• Experience with Delta Lake and Lakehouse architecture principles.
• Knowledge of data modeling, data quality, and performance tuning techniques.
• Familiarity with Git and version control.

Responsibilities

• Design, develop, and maintain scalable data engineering solutions using Databricks.
• Create and deploy custom ETL/ELT pipelines for large data volumes.
• Configure, manage, and optimize Databricks clusters for performance and cost.
• Implement Lakehouse architectures using Databricks best practices.
• Develop data workflows with Delta Lake and Delta Live Tables.
• Design Medallion Architecture for reliable data processing.
• Govern data assets with Unity Catalog ensuring security and access control.
• Collaborate with data architects and analysts to translate requirements.
• Monitor pipelines and troubleshoot performance or data quality issues.
• Optimize Spark jobs and CI/CD deployment.

Skills

Databricks

Apache Spark

Delta Lake

DLT (Delta Live Tables)

Unity Catalog

Medallion Architecture

ETL pipelines

Data modeling

Git

CI/CD

Education

BTech/MTech/MCA

Tools

Databricks clusters

Delta Lake

Unity Catalog

DLT

Spark (PySpark/Scala)

Job description

Job Title: Databricks Engineer Experience Level: 5 - 10 years Location: Hyderabad / Pune

Qualification: BTech/MTech/MCA Mode of Work: Hybrid

We are seeking a skilled and motivated Databricks Engineer to design, develop, and optimize scalable data solutions on the Databricks platform. The ideal candidate will have strong expertise in Apache Spark, Databricks architecture, ETL development, and modern data lakehouse concepts. This role will be responsible for building robust data pipelines, implementing best practices in data engineering, and supporting analytics and machine learning workloads. Key Responsibilities:

• Design, develop, and maintain scalable data engineering solutions using Databricks and Apache Spark.
• Create and deploy custom ETL/ELT pipelines to ingest, transform, and process large volumes of structured and unstructured data.
• Configure, manage, and optimize Databricks clusters for performance, scalability, and cost efficiency.
• Implement and manage Lakehouse architectures using Databricks best practices.
• Develop and maintain data workflows utilizing Delta Lake and Delta Live Tables (DLT).
• Design and implement Medallion Architecture (Bronze, Silver, Gold layers) for reliable and scalable data processing.
• Configure and govern data assets using Unity Catalog, ensuring data security, access control, and compliance.
• Collaborate with data architects, analysts, and business stakeholders to translate business requirements into technical solutions.
• Monitor data pipelines and troubleshoot performance or data quality issues.
• Optimize Spark jobs and workflows to improve processing efficiency and reduce execution time.
• Follow DevOps and CI/CD best practices for deployment and release management.

Requirements

• 5-10 years of experience in Data Engineering or Big Data technologies.
• Strong understanding of Databricks architecture and core platform capabilities.
• Hands-on experience in setting up and managing Databricks clusters.
• Strong expertise in Apache Spark (PySpark/Scala Spark).
• Experience developing and deploying custom ETL pipelines in Databricks.
• Experience implementing Medallion Architecture within Databricks.
• Hands-on experience with Delta Live Tables (DLT).
• Strong understanding and implementation experience with Unity Catalog.
• Experience working with Delta Lake and Lakehouse architecture principles.
• Knowledge of data modeling, data quality, and performance tuning techniques.
• Familiarity with version control tools such as Git.

Healthinsurance of INR 5.0 Lakhs for you and your family (up to 6members), ensuring complete peace of mind.

RobustProtection Plans:

GroupPersonal Accident Insurance and Group Term Life Insurance tosafeguard you and your loved ones.

PF andGratuity provided as per standard government regulations.

FlexibleWork Options:

Enjoy hybridwork arrangements & flexible working hours

GenerousLeave Policy:

21 days ofannual leave, in addition to 10 company-declared holidays.

Access to adedicated break-out area with round-the-clock refreshmentsfor relaxation and rejuvenation.