Databricks Engineer

DATAECONOMY Inc; Maharashtra; Hybrid; INR 2,500,000 - 4,000,000; Full time

Job summary

DATAECONOMY Inc. is seeking a Lead/Senior Databricks Engineer in Pune/Hyderabad (Hybrid). The role requires 8+ years of experience in data engineering, Databricks, PySpark, and Azure Cloud.

You will design and optimize scalable data pipelines and modern data platforms. Responsibilities include building ETL/ELT pipelines, integrating data sources with ADF, ADLS Gen2, Delta Lake, and Lakehouse architectures, and ensuring data quality and governance.

Qualifications

• Bachelor's degree in CS/IT or related field required.
• 8+ years of data engineering experience with Databricks and Azure.
• Azure/Databricks certifications preferred.

Responsibilities

• Design and build scalable ETL/ELT pipelines using Databricks and PySpark.
• Create data ingestion frameworks from multiple sources.
• Develop pipelines with Azure Data Factory (ADF).
• Work with ADLS Gen2 for large datasets storage and processing.
• Implement Delta Lake and Lakehouse concepts.
• Optimize Spark jobs for performance and scale.
• Transform data using SQL and PySpark.
• Collaborate with data architects, analysts, and business stakeholders.
• Implement CI/CD pipelines and deployment automation.
• Ensure data quality, security, and governance.

Skills

Databricks

PySpark

Spark SQL

SQL Server

Azure Cloud

ADF

ADLS

Delta Lake

Azure Synapse

Git

CI/CD

Performance tuning

Data Modeling

Education

Bachelor's degree in CS/IT

Azure or Databricks certifications

Tools

SSIS

Essbase

SQL Server

Data Warehouse

Stored Procedures

Views

Triggers

Job description

Job Description – Lead/Senior Databricks Engineer

Experience: 8+ Years

Location: Pune / Hyderabad (Hybrid)

Employment Type: Full-Time

Role Summary

We are looking for a skilled Data Engineer with strong experience in Databricks and Azure Cloud to design, develop, and optimize scalable data pipelines and modern data platforms. The ideal candidate should have expertise in big data technologies, ETL development, and cloud-based data engineering.

Key Responsibilities

• Design and develop scalable ETL/ELT pipelines using Databricks and PySpark.
• Build and optimize data ingestion frameworks from multiple data sources.
• Develop data pipelines using Azure Data Factory (ADF).
• Work with Azure Data Lake Storage (ADLS Gen2) for storing and processing large datasets.
• Implement Delta Lake and Lakehouse architecture concepts.
• Optimize Spark jobs for performance and scalability.
• Develop data transformations using SQL and PySpark.
• Collaborate with data architects, analysts, and business stakeholders.
• Implement CI/CD pipelines and deployment automation.
• Ensure data quality, security, and governance standards.

Required Skills

• Strong hands-on experience with Databricks
• Proficiency in PySpark and Spark SQL
• SSIS, Essbase, SQL Server, Data Warehouse, Stored Procedure, Views Triggers
• Experience with Microsoft Azure
• Good knowledge of:
• Azure Data Factory (ADF)
• Azure Data Lake Storage (ADLS)
• Delta Lake
• Azure Synapse Analytics
• Strong SQL and data modeling skills
• Experience with Git and CI/CD processes
• Knowledge of performance tuning and optimization

Preferred Skills

• Experience with:
• Streaming using Spark Structured Streaming
• Kafka/Event Hub
• Data Warehousing concepts
• DevOps and Infrastructure as Code
• Knowledge of:
• Data Governance
• Data Quality frameworks
• Medallion Architecture (Bronze, Silver, Gold)

Educational Qualification

• Bachelor's degree in Computer Science, Information Technology, or related field.
• Relevant Azure or Databricks certifications are preferred.

Nice to Have Certifications

• Databricks Certified Data Engineer Associate
• Microsoft Certified Azure Data Engineer Associate

Requirements

Required Skills

• Strong hands-on experience with Databricks
• Proficiency in PySpark and Spark SQL
• SSIS, Essbase, SQL Server, Data Warehouse, Stored Procedure, Views Triggers
• Experience with Microsoft Azure
• Good knowledge of:
• Azure Data Factory (ADF)
• Azure Data Lake Storage (ADLS)
• Delta Lake
• Azure Synapse Analytics
• Strong SQL and data modeling skills
• Experience with Git and CI/CD processes
• Knowledge of performance tuning and optimization