Databricks Architect

Code1; India; Remote; INR 900,000 - 1,500,000; Full time

Job summary

Code1 Tech in India is seeking an experienced Databricks Architect with strong hands‑on coding in Apache Spark, PySpark, and the Databricks Lakehouse Platform. You will design enterprise data architectures, optimize Spark workloads, and lead large migration initiatives. 8+ years of experience is required.

The role focuses on governance, security, and scalable data pipelines, with exposure to Lakeflow, DLT, and Unity Catalog. You will mentor teams and drive architectural decisions across projects.

Qualifications

• 8+ years of experience in data engineering, data architecture, or distributed data processing.
• Strong practical experience with Databricks, Apache Spark, PySpark, Spark SQL, and Delta Lake.
• Expert‑level hands‑on coding and advanced troubleshooting capabilities.
• Demonstrated experience delivering 3–4 end‑to‑end data platform migrations.

Responsibilities

• Design and implement enterprise‑grade data architectures using Databricks, Apache Spark, Delta Lake, and PySpark.
• Optimize Spark workloads, cluster configurations, memory utilization, joins, and execution plans.
• Architect batch and streaming ingestion pipelines using Auto Loader, Lakeflow Declarative Pipelines (LDP), and Delta Live Tables (DLT).
• Implement Delta Lake optimization strategies, CDC, SCD, and advanced table management.
• Design secure data governance using Unity Catalog, RBAC, ABAC, and access controls.
• Lead end‑to‑end data platform migrations, including architecture, execution, reconciliation, validation, and cost analysis.
• Implement IaC, platform administration, automation, and operational best practices.
• Evaluate and implement emerging Databricks capabilities, including Lakeflow, Databricks Genie, AI/BI tools, and Lakehouse Federation.
• Lead architecture reviews, resolve production issues, and mentor engineering teams.

Skills

Spark internals

Performance optimization

PySpark

Databricks Lakehouse

Leadership

Stakeholder management

Tools

Auto Loader

Lakeflow

Delta Live Tables (DLT)

Delta Lake

Unity Catalog

Terraform

Job description

At Code1 Tech, we drive innovations that shape the future of enterprise technology. Our expertise spans Data Engineering, AI/ML, Cloud Solutions, and Full-Stack Development. We empower businesses with cutting-edge technology solutions, enabling digital transformation at scale. Join us to build impactful products with a passionate team of engineers and innovators.

About The Role

We are looking for an experienced Databricks Architect with strong hands‑on coding expertise in Apache Spark, PySpark, and the Databricks Lakehouse Platform. The ideal candidate should have deep knowledge of Spark internals, advanced performance optimization, Delta Lake architecture, data ingestion and streaming, Unity Catalog governance, and enterprise‑scale data platform migrations.

The candidate must have successfully delivered 3 - 4 end-to-end data platform migrations and demonstrate expertise in designing scalable, secure, cost‑efficient, and high‑performance data architectures.

Exp: 8+ Years

Key Responsibilities:

• Design and implement enterprise‑grade data architectures using Databricks, Apache Spark, Delta Lake, and PySpark.
• Optimize Spark workloads, cluster configurations, memory utilization, joins, and execution plans.
• Architect batch and streaming ingestion pipelines using Auto Loader, Lakeflow Declarative Pipelines (LDP), and Delta Live Tables (DLT).
• Implement Delta Lake optimization strategies, Change Data Capture (CDC), Slowly Changing Dimensions (SCD), and advanced table management.
• Design secure data governance frameworks using Unity Catalog, RBAC, ABAC, and access‑control policies.
• Lead end‑to‑end data platform migrations, including migration architecture, execution, reconciliation, validation, and cost analysis.
• Implement Infrastructure as Code (IaC), platform administration, automation, and operational best practices.
• Evaluate and implement emerging Databricks capabilities, including Lakeflow, Databricks Genie, AI/BI, AgentBricks, Lakebase, and Lakehouse Federation.
• Lead architecture reviews, resolve complex production issues, and mentor engineering teams.

Required Technical Skills:

1. Apache Spark Core and Architecture:

• Deep understanding of Spark architecture, execution plans, DAG scheduling, stages, tasks, and Spark internals.
• Advanced performance optimization and troubleshooting.
• Speculative execution, join strategies, shuffle optimization, and memory spill management.
• Cluster sizing, resource utilization, heap memory, garbage collection, and executor tuning.
• Photon engine, vectorized execution, and serverless architecture.

2. Delta Lake and Data Ingestion:

• Delta Lake architecture, file layout optimization, partitioning, compaction, and data skipping.
• Delta architecture limitations and design trade‑offs.
• Slowly Changing Dimensions (SCD) and Change Data Capture (CDC) strategies.
• UniForm and Apache Iceberg interoperability.
• Deletion vectors and table maintenance.
• Auto Loader architecture, schema evolution, and scalable ingestion patterns.

3. Streaming and Lakeflow Declarative Pipelines:

• Lakeflow Declarative Pipelines (LDP), including DLT-based pipeline development and migration from legacy implementations.
• Multi‑source ingestion patterns and materialized views.
• Stream‑stream joins, stateful versus stateless processing, and event‑time windows.
• Watermarking, state management, and state clearing.
• Pipeline reliability, monitoring, incremental processing, and fault tolerance.
• Minimum 3–4 end‑to‑end data platform or ETL migration projects.
• Migration assessment, target architecture, and phased migration planning.
• In‑flight migration strategies and workload cutover.
• Migration tooling, data reconciliation, validation, and parallel‑run strategies.
• Migration Total Cost of Ownership (TCO), performance benchmarking, and optimization.

5. Platform Architecture, Governance, and Administration:

• Unity Catalog architecture, governance, lineage, and centralized access management.
• Role‑Based Access Control (RBAC) and Attribute‑Based Access Control (ABAC).
• Workspace administration, permissions, compute policies, and platform security.
• Infrastructure as Code using Terraform or equivalent tooling.
• Databricks console administration, monitoring, troubleshooting, and cost governance.
• Environment management and deployment automation.

6. Advanced Troubleshooting and Hands‑on Coding:

• Diagnose performance degradation, inefficient query plans, skew, and resource bottlenecks.
• Troubleshoot Auto Loader scaling and ingestion failures.
• Strong hands‑on expertise in PySpark, Spark SQL, Python, and SQL.
• Solve complex coding and debugging scenarios involving API‑based UDFs, partition repair, and PySpark row manipulation.
• Optimize reusable code, transformations, joins, and large‑scale data processing workloads.

7. Next‑Generation Databricks and AI Capabilities:

• Lakeflow orchestration and workflow automation.
• Unity Catalog metric views and semantic‑layer design.
• Agentic AI and AgentBricks.
• Lakebase architecture and Lakehouse Federation.
• DBT integration with Databricks for analytics engineering and transformation workflows.

Mandatory Requirements:

• 8+ years of experience in data engineering, data architecture, or distributed data processing.
• Strong practical experience with Databricks, Apache Spark, PySpark, Spark SQL, and Delta Lake.
• Expert‑level hands‑on coding and advanced troubleshooting capabilities.
• Demonstrated experience delivering 3 - 4 end‑to‑end data platform migrations.
• Strong understanding of Spark performance tuning, data ingestion, streaming, and platform governance.
• Hands‑on experience with Unity Catalog and enterprise data security.
• Experience with DBT, infrastructure automation, and modern data platform architecture.
• Excellent communication, architectural decision‑making, and stakeholder management skills.

Preferred Qualifications:

• Exposure to modern Databricks AI, BI, and data application capabilities.
• Experience with cloud data platforms on AWS, Microsoft Azure, or Google Cloud.
• Ability to lead technical discussions, conduct architecture reviews, and guide engineering teams.