Databricks Developer

Vservice It Solutions; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

VService IT Solutions seeks a Senior Databricks Developer to design and deliver enterprise-grade Lakehouse solutions using Databricks on cloud platforms. You will lead end-to-end data engineering projects, building scalable ETL/ELT pipelines and optimizing Spark workloads.

The ideal candidate has 5+ years in IT with 4+ years in data engineering/big data, strong hands-on experience with Databricks, PySpark, Spark, and Delta Lake, and a solid background in CI/CD, Git, and Terraform.

Qualifications

• 5+ years in IT with 4+ years in Data Engineering/Big Data.
• Hands-on Databricks, PySpark, Apache Spark, and Delta Lake.
• Experience with Unity Catalog, Auto Loader, Databricks Workflows, DLT/Lakehouse, Databricks SQL, and Serverless.
• Strong understanding of Spark internals (DAG, Driver/Executors, partitioning, shuffle, AQE, Catalyst).
• Experience with ETL/ELT, Python, SQL, data modeling, and batch/streaming pipelines.
• Cloud experience across AWS, Azure, or GCP; familiarity with another cloud is a plus.
• Databricks certifications preferred (Data Engineer Professional).
• Delivered multiple end-to-end Databricks implementations.

Responsibilities

• Design and develop Lakehouse architectures using Azure Databricks, Delta Lake and Unity Catalog.
• Build and optimize ETL/ELT pipelines with PySpark, SQL, Python; utilize Auto Loader and Delta Live Tables.
• Develop batch and streaming data processing solutions and optimize Spark workloads.
• Implement Databricks Workflows, Serverless computing and Databricks SQL solutions.
• Manage data governance, security, access controls and metadata with Unity Catalog.
• Collaborate with customers and stakeholders on requirements, architecture design and delivery.

Skills

Databricks

PySpark

Apache Spark

Delta Lake

Unity Catalog

Auto Loader

DLT/Lakehouse

Databricks SQL

Serverless

ETL/ELT

Tools

Git

CI/CD

Terraform

Databricks Asset Bundles

MLflow

Job description

We are seeking a skilled Senior Databricks Developer with strong expertise in Databricks, PySpark, Apache Spark, and cloud-based data engineering. The ideal candidate should have experience designing and implementing enterprise-grade Lakehouse solutions and delivering end-to-end data engineering projects.

Key Responsibilities

• Design and develop scalable Lakehouse architectures using Azure Databricks, Delta Lake, and Unity Catalog.
• Build and optimize ETL/ELT pipelines using PySpark, SQL, Python, Auto Loader, and Delta Live Tables (DLT).
• Develop batch and streaming data processing solutions.
• Optimize Spark workloads through partitioning, caching, AQE, Catalyst Optimizer, and performance tuning.
• Implement Databricks Workflows, Serverless computing, and Databricks SQL solutions.
• Manage data governance, security, access controls, and metadata using Unity Catalog.
• Implement CI/CD pipelines using Git, Azure DevOps, GitHub, Jenkins, or GitLab.
• Work with Terraform and Databricks Asset Bundles for infrastructure and deployment automation.
• Support MLOps practices using MLflow, Model Registry, and model deployment.
• Collaborate with customers and stakeholders on requirements, architecture design, technical presentations, and solution delivery.

Mandatory Skills

• 5+ years of overall IT experience with 4+ years in Data Engineering / Big Data.
• Strong hands‑on experience with Databricks, PySpark, Apache Spark, and Delta Lake.
• Experience with Unity Catalog, Auto Loader, Databricks Workflows, DLT/Lakeflow, Databricks SQL, and Serverless.
• Strong understanding of Spark internals, including DAG, Driver/Executors, partitioning, shuffle, AQE, and Catalyst Optimizer.
• Experience with ETL/ELT, Python, SQL, data modeling, and batch/streaming pipelines.
• Strong expertise in AWS, Azure, or GCP, with working knowledge of another cloud platform.
• Experience in Spark performance tuning, Photon, cluster sizing, autoscaling, and cost optimization.
• Knowledge of Git, CI/CD, Terraform, and Databricks Asset Bundles.
• Experience with MLflow and MLOps practices.
• Databricks Certified Data Engineer Professional certification preferred.

Additional Requirements

• Must have delivered 34 end-to-end Databricks implementations.
• Strong customer‑facing communication and stakeholder management skills.
• Experience in enterprise data architecture, migration, and modernization is an advantage.