Databricks Engineer

Doist; Milpitas (CA); On-site; USD 120,000 - 180,000; Full time

Job summary

Doist is seeking a skilled Databricks Engineer to design, develop, and optimize data pipelines on the Databricks Lakehouse Platform. The role focuses on building scalable ETL/ELT workflows, leveraging PySpark, Spark SQL, and Delta Lake to deliver high-performance data solutions.

Ideal candidates will implement Medallion Architecture (Bronze, Silver, Gold), integrate data from diverse sources, and optimize Spark jobs for performance and cost.

Qualifications

• Bachelors or Masters in CS/IT/Engineering or related field.
• Strong experience with Databricks Lakehouse Platform and Spark.
• Proficient in PySpark, Spark SQL, and Delta Lake.
• Experience with cloud data platforms (Azure, AWS, GCP) and CI/CD.

Responsibilities

• Design, develop, and maintain scalable data pipelines on Databricks.
• Build ETL/ELT workflows for batch and streaming data processing.
• Develop solutions using PySpark, Spark SQL, and Delta Lake.
• Implement Medallion Architecture (Bronze, Silver, Gold).
• Integrate data from relational databases, APIs, cloud storage, and streams.
• Optimize Spark jobs for performance, scalability, and cost.
• Collaborate with Data Architects, Data Scientists, BI developers, and stakeholders.
• Implement CI/CD pipelines and deployment automation for Databricks workloads.
• Ensure data quality, governance, and security compliance.
• Monitor, troubleshoot, and optimize production data pipelines.
• Document technical solutions and follow engineering best practices.

Skills

Databricks Lakehouse Platform

Apache Spark

PySpark

Spark SQL

Delta Lake

Python

SQL

Cloud Platforms

CI/CD

ETL/ELT

Data Engineering

Education

Bachelor's or Master’s degree in CS/IT/Engineering

Tools

Databricks

Unity Catalog

Delta Live Tables

Terraform

Azure Data Factory

Azure Synapse

Job description

Role

Databricks Engineer

Experience

4–8 years

Location

As per business requirement

Employment Type

Full-time

Job Summary

We are looking for a skilled Databricks Engineer with strong expertise in designing, developing, and optimizing modern data engineering solutions on the Databricks Lakehouse Platform. The ideal candidate should have experience building scalable ETL/ELT pipelines, working with large-scale data, and leveraging Apache Spark to deliver high-performance data solutions.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks.
• Build ETL/ELT workflows for batch and streaming data processing.
• Develop solutions using PySpark, Spark SQL, and Delta Lake.
• Implement Medallion Architecture (Bronze, Silver, Gold) for data transformation.
• Integrate data from various sources including relational databases, APIs, cloud storage, and streaming platforms.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Collaborate with Data Architects, Data Scientists, BI developers, and business stakeholders.
• Implement CI/CD pipelines and deployment automation for Databricks workloads.
• Ensure data quality, security, governance, and compliance.
• Monitor, troubleshoot, and optimize production data pipelines.
• Document technical solutions and follow engineering best practices.

Required Skills

• Core Technologies Databricks Lakehouse Platform Apache Spark PySpark Spark SQL Delta Lake
• Python
• SQL
• Cloud Platforms (one or more) Microsoft Azure (preferred) AWS Google Cloud Platform
• Azure Technologies (Preferred) Azure Data Factory (ADF) Azure Data Lake Storage (ADLS Gen2) Azure Synapse Analytics Azure Key Vault Azure DevOps
• Data Engineering
• Data Warehousing
• Data Modeling
• ETL/ELT Development
• Batch Processing
• Streaming (Kafka/Event Hubs)
• Data Lake Architecture
• DevOps & Version Control
• Git
• Azure DevOps / GitHub CI/CD Pipelines

Preferred Qualifications

• Experience with Unity Catalog.
• Knowledge of Databricks Workflows and Jobs.
• Hands-on experience with Delta Live Tables (DLT).
• Exposure to MLflow is an added advantage.
• Experience with data governance and security best practices.
• Familiarity with Infrastructure as Code (Terraform) is a plus.

Educational Qualification

Bachelor\'s or Master\'s degree in Computer Science, Information Technology, Engineering, or a related field.

Preferred Certifications

• Databricks Certified Data Engineer Associate
• Databricks Certified Data Engineer Professional
• Microsoft Certified: Azure Data Engineer Associate (DP-203)
• Azure Fundamentals (AZ-900)

Nice to Have

• Experience with real-time analytics.
• Knowledge of Lakehouse architecture.
• Experience with Agile/Scrum methodologies.
• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management abilities.

Mandatory Skills

• Databricks
• PySpark
• Spark SQL
• Delta Lake
• Python
• SQL
• Azure/AWS/GCP (at least one cloud platform)
• ETL/ELT Development
• Data Lake Architecture