Databricks Forward Deployed Engineer

Dream Jobs Staffing Solution; Pune District, Chennai, Bengaluru; On-site; INR 3,000,000 - 6,000,000; Full time

Job summary

Dream Jobs Staffing Solution is seeking a Senior Databricks Forward Deployed Engineer (FDE) to design and deploy scalable data platforms on Databricks for enterprise clients. You will work with PySpark, Spark SQL, Delta Lake on Azure, and contribute to GenAI initiatives.

The role emphasizes production-grade data pipelines, data governance, and client-facing architecture decisions. Strong problem solving and communication skills are essential in a fast-paced environment.

Qualifications

• 5+ years of experience in data engineering.
• 2+ years of hands-on experience with Databricks or Apache Spark.
• Strong knowledge of PySpark, Spark SQL, Delta Lake, and performance tuning.
• Experience implementing enterprise Lakehouse and Medallion architectures.
• Hands-on experience with CI/CD pipelines and multi-environment deployments.
• Understanding of data governance, access controls, and production monitoring.
• Excellent communication, analytical thinking, and stakeholder management skills.
• Prior experience in client-facing, consulting, or enterprise delivery environments.

Responsibilities

• Design scalable Lakehouse architectures using Databricks and Medallion Architecture (Bronze, Silver, Gold).
• Develop and optimize production-grade data pipelines using PySpark, Spark SQL, and Delta Lake.
• Implement batch and streaming data ingestion using Auto Loader, Kafka, and event-driven architectures.
• Implement CDC and SCD patterns.
• Optimize Spark jobs, Delta Lake tables, cluster configurations, and overall platform performance.
• Configure Unity Catalog, data governance, access controls, and data lineage.
• Build and manage CI/CD pipelines using DABs, GitHub Actions, or Azure DevOps.
• Troubleshoot Spark performance issues and memory-related errors.
• Support GenAI solutions involving RAG pipelines, vector stores, embeddings, and LLM evaluation.
• Engage with clients to understand requirements and communicate architecture decisions.
• Establish data quality checks, monitoring, production alerting, and SLA frameworks.

Skills

Python

PySpark

Spark SQL

Databricks

Delta Lake

Azure

React

GenAI

CI/CD

SQL

Tools

Databricks

GitHub Actions

Azure DevOps

Kafka

Delta Live Tables

Job description

Job Title: Databricks Forward Deployed Engineer (FDE)

Experience: 58 Years

Location: PAN India

Open Positions: 10

Function: Data & AI Engineering

Level: Senior (G7)

Job Description

We are looking for experienced Databricks Forward Deployed Engineers (FDEs) with strong expertise in data engineering, Apache Spark, PySpark, Azure, CI/CD, and exposure to Generative AI. The ideal candidate will work closely with enterprise clients to design, develop, optimize, and deploy scalable, production-grade data platforms on Databricks.

The role requires strong technical problem-solving, solution design, performance optimization, and client-facing communication skills.

Key Responsibilities

• Design and implement scalable Lakehouse architectures using Databricks and Medallion Architecture (Bronze, Silver, Gold).
• Develop and optimize production-grade data pipelines using PySpark, Spark SQL, and Delta Lake.
• Implement batch and streaming data ingestion using Auto Loader, Kafka, and event-driven architectures.
• Implement CDC and SCD Type 1, Type 2, and Type 3 patterns.
• Optimize Spark jobs, Delta Lake tables, cluster configurations, and overall platform performance.
• Configure Unity Catalog, data governance, access controls, and data lineage.
• Build and manage CI/CD pipelines using Databricks Asset Bundles (DABs), GitHub Actions, or Azure DevOps.
• Troubleshoot Spark performance issues, including data skew, shuffle overhead, and memory-related errors.
• Support GenAI solutions involving RAG pipelines, vector databases, embeddings, and LLM evaluation.
• Engage with clients to understand business requirements, recommend technical solutions, and communicate architecture decisions.
• Establish data quality checks, monitoring, production alerting, and SLA frameworks.

Mandatory Skills

• Python and PySpark
• Apache Spark and Spark SQL
• Databricks and Delta Lake
• Microsoft Azure
• React
• Generative AI (GenAI)
• CI/CD and deployment automation
• Strong SQL and data pipeline development

Required Qualifications

• 5+ years of experience in data engineering.
• 2+ years of hands-on experience with Databricks or Apache Spark.
• Strong knowledge of PySpark, Spark SQL, Delta Lake, and performance tuning.
• Experience implementing enterprise Lakehouse and Medallion architectures.
• Hands-on experience with CI/CD pipelines and multi-environment deployments.
• Understanding of data governance, access controls, and production monitoring.
• Excellent communication, analytical thinking, and stakeholder management skills.
• Prior experience in client-facing, consulting, or enterprise delivery environments.

Good to Have

• Databricks Certified Data Engineer Associate or Professional certification.
• Experience with Structured Streaming, Delta Live Tables, or Kafka.
• Exposure to Mosaic AI, Databricks Genie, RAG, vector stores, and LLM evaluation.
• Experience with Azure DevOps, GitHub Actions, or Databricks Asset Bundles.
• Exposure to AWS or GCP cloud platforms.
• Experience in consulting, GSI, or enterprise client delivery environments.