Senior Databricks Lakehouse Architect (PySpark/ETL)

VService IT Solutions; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

VService IT Solutions seeks a Senior Databricks Developer to design and deliver enterprise-grade Lakehouse solutions using Databricks on cloud platforms. You will lead end-to-end data engineering projects, building scalable ETL/ELT pipelines and optimizing Spark workloads.

The ideal candidate has 5+ years in IT with 4+ years in data engineering/big data, strong hands-on experience with Databricks, PySpark, Spark, and Delta Lake, and a solid background in CI/CD, Git, and Terraform.

Qualifications

• 5+ years in IT with 4+ years in Data Engineering/Big Data.
• Hands-on Databricks, PySpark, Apache Spark, and Delta Lake.
• Experience with Unity Catalog, Auto Loader, Databricks Workflows, DLT/Lakehouse, Databricks SQL, and Serverless.
• Strong understanding of Spark internals (DAG, Driver/Executors, partitioning, shuffle, AQE, Catalyst).
• Experience with ETL/ELT, Python, SQL, data modeling, and batch/streaming pipelines.
• Cloud experience across AWS, Azure, or GCP; familiarity with another cloud is a plus.
• Databricks certifications preferred (Data Engineer Professional).
• Delivered multiple end-to-end Databricks implementations.

Responsibilities

• Design and develop Lakehouse architectures using Azure Databricks, Delta Lake and Unity Catalog.
• Build and optimize ETL/ELT pipelines with PySpark, SQL, Python; utilize Auto Loader and Delta Live Tables.
• Develop batch and streaming data processing solutions and optimize Spark workloads.
• Implement Databricks Workflows, Serverless computing and Databricks SQL solutions.
• Manage data governance, security, access controls and metadata with Unity Catalog.
• Collaborate with customers and stakeholders on requirements, architecture design and delivery.

Skills

Databricks

PySpark

Apache Spark

Delta Lake

Unity Catalog

Auto Loader

DLT/Lakehouse

Databricks SQL

Serverless

ETL/ELT

Tools

Git

CI/CD

Terraform

Databricks Asset Bundles

MLflow

Job description

VService IT Solutions seeks a Senior Databricks Developer to design and deliver enterprise-grade Lakehouse solutions using Databricks on cloud platforms. You will lead end-to-end data engineering projects, building scalable ETL/ELT pipelines and optimizing Spark workloads.

The ideal candidate has 5+ years in IT with 4+ years in data engineering/big data, strong hands-on experience with Databricks, PySpark, Spark, and Delta Lake, and a solid background in CI/CD, Git, and Terraform.