Databricks Solution Engineer

CloudTech Innovations; Bengaluru; On-site; INR 3,500,000 - 5,500,000; Full time

Job summary

CloudTech Innovations seeks a senior Solution Engineer with deep Databricks expertise to design and scale cloud-native data platforms. You will lead end-to-end Lakehouse implementations, emphasizing Unity Catalog governance and Delta Lake-based pipelines.

Candidates should demonstrate 10+ years in enterprise data engineering, experience with Spark, Delta Lake, and Medallion Architecture, and strong collaboration across cross-functional teams.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related field.
• 10+ years of enterprise software, cloud architecture, or data engineering experience.
• Hands-on with Databricks, Apache Spark, Delta Lake, and Lakehouse platform design.
• Experience administering Unity Catalog for governance, lineage, and access control.
• Experience designing Medallion Architecture for analytics and engineering workloads.
• Hands-on with AWS/Azure/GCP cloud platforms and data services.
• Experience with streaming technologies such as Kafka, Kinesis, or Pub/Sub.
• Strong knowledge of data modeling, orchestration (Airflow, Databricks Workflows, dbt), and pipeline automation.
• Familiarity with Scala-based Spark workloads in addition to PySpark and SQL pipelines.
• Proven ability to communicate with leadership and stakeholders.
• Certifications in Databricks, cloud architecture, or TOGAF are a plus.

Responsibilities

• Design and lead end-to-end Databricks Lakehouse Platforms using Delta Lake and MLflow.
• Architect Medallion Architecture for various data workloads.
• Implement governed Lakehouse patterns with Unity Catalog for access control, lineage, and data classification.
• Build scalable ETL/ELT pipelines using Databricks tools and Spark-based transformations.
• Develop real-time streaming pipelines with Auto Loader and Structured Streaming.
• Integrate Databricks with cloud-native services like AWS Glue, Azure Data Factory, GCP Dataform.
• Define distributed integration patterns using REST APIs and microservices.
• Enforce data governance, RBAC/ABAC, encryption, and secret management.
• Optimize Delta Lake tables and Spark workloads; manage cluster configurations.

Skills

Databricks

Apache Spark

Delta Lake

Lakehouse architecture

Data governance

RBAC/ABAC

Stakeholder communication

Cloud data platforms

Education

Bachelor’s or Master’s in CS/Data Eng/related

Tools

Unity Catalog

Medallion Architecture

Delta Live Tables

MLflow

Airflow

dbt

Kafka

Kinesis

Pub/Sub

Job description

We are looking for a highly experienced Solution Engineer with a strong background in building and scaling cloud-native data platforms. This role is Databricks-heavy, focusing on Unity Catalog, Medallion Architecture, Delta Lake, governance, and modern data engineering patterns. You will collaborate with cross-functional teams to translate business needs into well-architected solutions and guide delivery across enterprise environments.

Key Responsibilities

• Design and lead implementation of end-to-end Databricks Lakehouse Platforms using Delta Lake, Delta Live Tables, and MLflow.
• Architect Medallion Architecture (Bronze/Silver/Gold) for structured, semi-structured, and streaming workloads.
• Implement governed Lakehouse patterns using Unity Catalog for access control, lineage, data classification, and secure sharing.
• Build scalable ETL/ELT pipelines using Databricks Notebooks, Workflows, SQL Warehouses, and Spark-based transformations.
• Develop real-time streaming pipelines with Auto Loader, Structured Streaming, and event-driven platforms (Kafka, Kinesis, Pub/Sub).
• Integrate Databricks with cloud-native services such as AWS Glue, Azure Data Factory, and GCP Dataform.
• Define distributed integration patterns using REST APIs, microservices, and event-driven architectures.
• Enforce data governance, RBAC/ABAC, encryption, secret management, and compliance controls.
• Optimize Delta Lake tables, Spark workloads, and cluster configurations using Photon and autoscaling patterns.
• Drive cloud cost optimization across storage, compute, and workflow orchestration.
• Participate in architecture reviews, set standards, and support engineering teams throughout execution.
• Stay current on Databricks capabilities including Unity Catalog updates, Lakehouse Federation, serverless compute, and AI/ML features.

Required Qualifications

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related field.
• 10+ years of experience in enterprise software, cloud architecture, or data engineering roles.
• Strong hands-on experience with Databricks, Apache Spark, Delta Lake, and Lakehouse platform design.
• Experience implementing and administering Unity Catalog for governance, lineage, and fine-grained access control.
• Experience designing Medallion Architecture for analytics and engineering workloads.
• Hands-on experience with cloud platforms such as AWS, Azure, or GCP, including storage, compute, and networking services.
• Experience with streaming technologies such as Kafka, Kinesis, or Pub/Sub.
• Strong understanding of data modeling, workflow orchestration (Airflow, Databricks Workflows, dbt), and pipeline automation.
• Familiarity with Scala-based Spark workloads in addition to PySpark and SQL pipelines.
• Skilled in performance tuning, Spark optimization, cluster policies, and cloud cost management.
• Excellent communication skills for technical leadership and stakeholder collaboration.
• Certifications in Databricks, AWS/GCP/Azure Solution Architecture, or TOGAF are a plus.