Data Architect with Databricks

Unique Erp Technologies; Hyderabad; On-site; INR 1,800,000 - 4,000,000; Full time

Job summary

Unique Erp Technologies is seeking an experienced Data Architect to design, develop, and maintain scalable data pipelines using Databricks and Delta Lake in a cloud-based environment. The role focuses on enterprise analytics, BI initiatives, and robust data governance.

The ideal candidate will have 5+ years of data engineering experience with hands-on Databricks, PySpark, and SQL, plus strong knowledge of Delta Lake, DLT, Unity Catalog, and data modeling.

Qualifications

• 5+ years of Data Engineering experience.
• Hands-on Databricks proficiency.
• Strong PySpark, Spark SQL, Python, and SQL skills.
• Experience with Delta Lake, DLT, Unity Catalog.
• Knowledge of Data Lake, Lakehouse, and Medallion Architecture.
• Experience with Azure Databricks or AWS Databricks.
• Familiarity with Azure Data Factory and ADLS Gen2.
• Understanding of data modeling and ETL/ELT concepts.
• CI/CD pipelines and Agile methodologies experience.
• Excellent problem-solving and communication skills.

Responsibilities

• Design, build, and optimize scalable ETL/ELT pipelines using Databricks and PySpark.
• Develop and maintain data ingestion frameworks for batch and real-time data processing.
• Implement and manage Delta Lake, Delta Live Tables, and Lakehouse architectures.
• Work with Azure/AWS cloud services for data storage, processing, and orchestration.
• Transform and integrate structured and unstructured data from multiple enterprise sources.
• Develop reusable data models and frameworks to support analytics.
• Optimize Spark jobs, cluster configurations, and query performance.
• Implement data quality checks, monitoring, and governance controls.
• Collaborate with stakeholders, architects, and analysts to deliver data solutions.
• Ensure data security, compliance, and best practices across the data ecosystem.
• Support CI/CD processes and automation for data platform deployments.

Skills

Databricks

PySpark

Spark SQL

Python

SQL

Delta Lake

Delta Live Tables

Unity Catalog

Azure Databricks

AWS Databricks

Azure Data Factory

ADLS Gen2

Data Modeling

ETL/ELT

CI/CD

Agile

Communication

Education

Bachelor's degree

Master's degree

Job description

About the Role

Client is seeking an experienced Data Architect with Databricks to design, develop, and maintain scalable data pipelines and cloud-based data platforms. The ideal candidate will have expertise in Databricks, PySpark, Delta Lake, and modern data engineering practices to support enterprise analytics, reporting, and business intelligence initiatives.

Key Responsibilities

• Design, build, and optimize scalable ETL/ELT pipelines using Databricks and PySpark.
• Develop and maintain data ingestion frameworks for batch and real-time data processing.
• Implement and manage Delta Lake, Delta Live Tables, and Lakehouse architecture solutions.
• Work with Azure/AWS cloud services for data storage, processing, and orchestration.
• Transform and integrate structured and unstructured data from multiple enterprise sources.
• Develop reusable data models and frameworks to support enterprise analytics.
• Optimize Spark jobs, cluster configurations, and query performance.
• Implement data quality checks, monitoring, and governance controls.
• Collaborate with business stakeholders, architects, and analysts to deliver data solutions.
• Ensure data security, compliance, and best practices across the data ecosystem.
• Support CI/CD processes and automation for data platform deployments.

Required Skills

• 5+ years of Data Engineering experience.
• Strong hands-on experience with Databricks.
• Expertise in PySpark, Spark SQL, Python, and SQL.
• Experience with Delta Lake, Delta Live Tables (DLT), Unity Catalog.
• Strong understanding of Data Lake, Lakehouse, Medallion Architecture (Bronze, Silver, Gold).
• Experience with Azure Databricks or AWS Databricks.
• Experience with Azure Data Factory (ADF), ADLS Gen2, or equivalent AWS services.
• Strong knowledge of data modeling, ETL/ELT concepts, and performance tuning.
• Experience with Git, CI/CD pipelines, and Agile methodologies.
• Excellent problem-solving and communication skills.

Preferred Skills

• Experience with Snowflake, Synapse Analytics, Redshift, or Fabric.
• Knowledge of Kafka, Event Hubs, or other streaming technologies.
• Exposure to Unity Catalog and data governance frameworks.
• Experience with Power BI, Tableau, or other BI tools.
• Databricks certifications are highly preferred.
• Manufacturing, Banking, Retail, or Enterprise Applications domain experience.

Nice to Have

• Databricks Certified Data Engineer Associate/Professional.
• Knowledge of Data Governance, Data Lineage, and Data Quality frameworks.
• Experience in cloud migration and modernization projects.
• Exposure to AI/ML data pipelines and MLOps.

Education

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or related field.

Keywords for Sourcing

Databricks, PySpark, Spark SQL, Delta Lake, Delta Live Tables, DLT, Unity Catalog, Azure Databricks, AWS Databricks, ADF, ADLS, Synapse, Snowflake, Data Lake, Lakehouse, ETL, ELT, Data Engineering, Python, SQL.