Databricks Architect

Experion Technologies Pvt. Ltd.; Ernakulam; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

Experion Technologies Pvt. Ltd. is seeking a Databricks Lead / Architect to drive architecture, design and delivery of enterprise-scale data platforms across Azure, AWS or GCP.

You will provide hands-on technical leadership for Databricks, Lakehouse architectures, data integration, migration and analytics workloads. You will guide data engineers, establish engineering standards and oversee modernization initiatives while collaborating with clients and enterprise architects to translate

Qualifications

• 10+ years in Data Engineering / Data Architecture.
• Hands-on Databricks and Lakehouse architecture experience.
• Strong knowledge of Spark, PySpark, Python and SQL.
• Experience with Delta Lake and Medallion Architecture.
• Experience designing scalable batch and real-time data solutions.
• Strong knowledge of at least one cloud platform: Azure, AWS or GCP.
• Understanding of cloud data services and integration patterns.
• CI/CD, Git and DevOps practices.

Responsibilities

• Design and implement enterprise-scale Databricks Lakehouse architectures.
• Define architecture across Azure, AWS and GCP based on requirements.
• Design data ingestion, transformation and processing using Databricks, Spark and Python.
• Establish Lakehouse architecture including Delta Lake and Unity Catalog.
• Design batch and real-time data processing and integration solutions.
• Lead migration and modernization of legacy data platforms to Databricks.
• Establish CI/CD, governance and deployment standards for data engineering.
• Provide technical leadership through design reviews and code reviews.
• Mentor Data Engineers and technical teams.
• Engage with stakeholders to translate requirements into technical solutions.

Skills

Databricks

Lakehouse architecture

Apache Spark

PySpark

Python

SQL

Cloud platforms

CI/CD

DevOps practices

Data governance

Tools

Delta Lake

Unity Catalog

Medallion Architecture

Databricks

Job description

Lead the architecture, design and delivery of enterprise-scale Databricks data platforms across Azure, AWS or GCP. The role will provide technical leadership for modern data engineering, Lakehouse architecture, data integration, data migration, analytics and AI/ML workloads while ensuring scalable, secure, performant and cost-effective solutions.

Job Description

The Databricks Lead / Architect will own end-to-end solution design and provide hands‑on technical leadership across Databricks and cloud data platforms. The role involves translating business and data requirements into scalable architectures, establishing engineering standards, guiding development teams, driving modernization and migration initiatives, and working closely with clients, architects and technology leadership.

Key Responsibilities

• Design and implement enterprise-scale Databricks Lakehouse architectures.
• Define architecture across Azure, AWS and GCP based on business and technical requirements.
• Design data ingestion, transformation and processing using Databricks, Spark, PySpark, SQL and Python.
• Establish Lakehouse architecture, including Delta Lake, Medallion Architecture and data modelling.
• Design batch and real-time data processing and integration solutions.
• Architect and implement Databricks Workflows, Jobs, Unity Catalog and governanc capabilities.
• Define data security, access control, lineage, quality, monitoring and governance standards.
• Design scalable and optimized solutions for performance and cloud cost.
• Lead migration and modernization of legacy data platforms to Databricks.
• Integrate Databricks with cloud-native services across Azure / AWS / GCP.
• Establish CI/CD, Git, DevOps and deployment standards for data engineering.
• Provide technical leadership through design reviews, code reviews and engineering standards.
• Guide and mentor Data Engineers and technical teams.
• Conduct POCs and evaluate new Databricks/cloud capabilities.
• Work with business stakeholders, project managers and enterprise architects to translate requirements into technical solutions.
• Support estimation, solution proposals, technical presentations and client discussions.
• Adhere to the Information Security Management policies and procedures.

Job Specification

Skills

Mandatory Skills

• 10+ years of experience in Data Engineering / Data Architecture.
• Strong hands-on experience with Databricks and Lakehouse architecture.
• Strong knowledge of Apache Spark, PySpark, Python and SQL.
• Experience with Delta Lake and Medallion Architecture.
• Strong experience in data modelling, ETL/ELT and data integration.
• Experience designing scalable batch and real-time data solutions.
• Strong knowledge of at least one major cloud platform: Azure, AWS or GCP.
• Understanding of cloud data services and integration patterns.
• Experience with Databricks Workflows/Jobs and Unity Catalog.
• Strong understanding of data security, governance, quality and lineage.
• Experience with CI/CD, Git and DevOps practices.
• Strong solution architecture and technical leadership skills.

Good to Have

• Snowflake