Databricks Data Engineer

Unique Erp Solutions; Hyderabad; Hybrid; INR 1,800,000 - 3,800,000; Full time

Job summary

Unique Erp Solutions is seeking an experienced Databricks Data Engineer to design, develop, and maintain scalable data pipelines and cloud-based platforms for enterprise analytics.

You will work with Databricks, PySpark, Delta Lake, and cloud services (Azure/AWS) to enable reporting, BI tools, and data governance across the organization. The role requires strong data modeling, CI/CD for data platforms, and collaboration with architects and analysts.

Qualifications

• 5+ years of Data Engineering experience.
• Strong hands-on experience with Databricks.
• Expertise in PySpark, Spark SQL, Python, and SQL.
• Experience with Delta Lake, Delta Live Tables (DLT), Unity Catalog.
• Strong understanding of Data Lake, Lakehouse, Medallion Architecture (Bronze, Silver, Gold).
• Experience with Azure Databricks or AWS Databricks.
• Experience with Azure Data Factory (ADF), ADLS Gen2, or equivalent AWS services.
• Strong knowledge of data modeling, ETL/ELT concepts, and performance tuning.
• Experience with Git, CI/CD pipelines, and Agile methodologies.
• Excellent problem-solving and communication skills.

Responsibilities

• Design, build, and optimize scalable ETL/ELT pipelines using Databricks and PySpark.
• Develop and maintain data ingestion frameworks for batch and real-time data processing.
• Implement and manage Delta Lake, Delta Live Tables, and Lakehouse architecture solutions.
• Work with Azure/AWS cloud services for data storage, processing, and orchestration.
• Transform and integrate structured and unstructured data from multiple enterprise sources.
• Develop reusable data models and frameworks to support enterprise analytics.
• Optimize Spark jobs, cluster configurations, and query performance.
• Implement data quality checks, monitoring, and governance controls.
• Collaborate with business stakeholders, architects, and analysts to deliver data solutions.
• Ensure data security, compliance, and best practices across the data ecosystem.
• Support CI/CD processes and automation for data platform deployments.

Skills

Databricks

PySpark

Spark SQL

Python

SQL

Delta Lake

Delta Live Tables

Unity Catalog

Azure Databricks

AWS Databricks

Education

Bachelor's or Master's degree in Computer Science/IT/Engineering

Tools

Azure Databricks

AWS Databricks

Azure Data Factory

ADLS Gen2

Git

CI/CD

Job description

About the Role

Client is seeking an experienced Databricks Data Engineer to design, develop, and maintain scalable data pipelines and cloud-based data platforms. The ideal candidate will have expertise in Databricks, PySpark, Delta Lake, and modern data engineering practices to support enterprise analytics, reporting, and business intelligence initiatives.

Key Responsibilities

• Design, build, and optimize scalable ETL/ELT pipelines using Databricks and PySpark.
• Develop and maintain data ingestion frameworks for batch and real-time data processing.
• Implement and manage Delta Lake, Delta Live Tables, and Lakehouse architecture solutions.
• Work with Azure/AWS cloud services for data storage, processing, and orchestration.
• Transform and integrate structured and unstructured data from multiple enterprise sources.
• Develop reusable data models and frameworks to support enterprise analytics.
• Optimize Spark jobs, cluster configurations, and query performance.
• Implement data quality checks, monitoring, and governance controls.
• Collaborate with business stakeholders, architects, and analysts to deliver data solutions.
• Ensure data security, compliance, and best practices across the data ecosystem.
• Support CI/CD processes and automation for data platform deployments.

Required Skills

• 5+ years of Data Engineering experience.
• Strong hands-on experience with Databricks.
• Expertise in PySpark, Spark SQL, Python, and SQL.
• Experience with Delta Lake, Delta Live Tables (DLT), Unity Catalog.
• Strong understanding of Data Lake, Lakehouse, Medallion Architecture (Bronze, Silver, Gold).
• Experience with Azure Databricks or AWS Databricks.
• Experience with Azure Data Factory (ADF), ADLS Gen2, or equivalent AWS services.
• Strong knowledge of data modeling, ETL/ELT concepts, and performance tuning.
• Experience with Git, CI/CD pipelines, and Agile methodologies.
• Excellent problem-solving and communication skills.

Preferred Skills

• Experience with Snowflake, Synapse Analytics, Redshift, or Fabric.
• Knowledge of Kafka, Event Hubs, or other streaming technologies.
• Exposure to Unity Catalog and data governance frameworks.
• Experience with Power BI, Tableau, or other BI tools.
• Databricks certifications are highly preferred.
• Manufacturing, Banking, Retail, or Enterprise Applications domain experience.

Nice to Have

• Databricks Certified Data Engineer Associate/Professional.
• Knowledge of Data Governance, Data Lineage, and Data Quality frameworks.
• Experience in cloud migration and modernization projects.
• Exposure to AI/ML data pipelines and MLOps.

Education

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or related field.

Keywords for Sourcing

Databricks, PySpark, Spark SQL, Delta Lake, Delta Live Tables, DLT, Unity Catalog, Azure Databricks, AWS Databricks, ADF, ADLS, Synapse, Snowflake, Data Lake, Lakehouse, ETL, ELT, Data Engineering, Python, SQL.