Databricks Data Engineer - Spark, ETL & Lakehouse

CloudBigD

Quincy (MA)

On-site

USD 77,000 - 110,000

Full time

38 hours ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

CloudBigD in Quincy, MA is seeking a Data Engineer to design and optimize Spark-based data pipelines on the Azure Databricks platform. You will implement lakehouse architectures and streaming jobs while collaborating with product teams.

The role emphasizes Spark, PySpark/Scala, SQL tuning, and Databricks expertise, with Docker/Kubernetes familiarity as a plus for production readiness.

Qualifications

  • Experience building Spark-based data pipelines and transformations (ETL/ELT).
  • Proficient in PySpark/Scala and SQL performance tuning.
  • Hands-on with Azure Databricks, Databricks Lakehouse, and Unity Catalog.
  • Familiarity with Docker and Kubernetes in data engineering contexts.

Responsibilities

  • Develop and optimize Spark jobs and data processing workflows.
  • Implement streaming and batch transformations in Azure Databricks.
  • Design data warehouse/lakehouse solutions and monitor live jobs.
  • Troubleshoot DevOps pipelines and cloud services in production.

Skills

Spark/DataFrames
ETL/ELT development
Scala/PySpark/SQL
Streaming processing
Data pipelines
Cloud data engineering
DevOps pipelines
Cluster sizing
Spark SQL optimization
Big data concepts

Tools

Databricks
Azure Databricks
Databricks Unity Catalog
Auto Loader
Lakehouse architecture
Databricks Serverless
Docker
Kubernetes

Job description

CloudBigD in Quincy, MA is seeking a Data Engineer to design and optimize Spark-based data pipelines on the Azure Databricks platform. You will implement lakehouse architectures and streaming jobs while collaborating with product teams.

The role emphasizes Spark, PySpark/Scala, SQL tuning, and Databricks expertise, with Docker/Kubernetes familiarity as a plus for production readiness.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Databricks & Spark Data Engineer | Azure Lakehouse
Databricks & Spark Data Engineer | Azure Lakehouse

Ethereum Technologies LLC • Quincy (MA)

On-site
USD 77,000 - 110,000
Databricks Data Engineer: Spark, Lakehouse & Streaming
Databricks Data Engineer: Spark, Lakehouse & Streaming

Valiant Technologies LLC • Quincy (MA)

On-site
USD 77,000 - 110,000
Databricks Data Engineer: Lakehouse & Spark Architect
Databricks Data Engineer: Lakehouse & Spark Architect

CloudBigD • Atlanta (GA)

On-site
USD 86,000 - 123,000
Data Engineer: Spark, Databricks & Lakehouse Architect
Data Engineer: Spark, Databricks & Lakehouse Architect

Quantum Technologies LLC • Houston (TX)

On-site
USD 110,000 - 170,000
Data Engineer - Databricks & Spark on Azure
Data Engineer - Databricks & Spark on Azure

Quantum Technologies LLC • Quincy (MA)

On-site
USD 80,000 - 107,000
Databricks Data Engineer: Lakehouse & Spark Expert
Databricks Data Engineer: Lakehouse & Spark Expert

CloudBigD • Louisville (KY)

On-site
USD 120,000 - 180,000
Databricks Data Engineer: Lakehouse & Streaming Specialist
Databricks Data Engineer: Lakehouse & Streaming Specialist

CloudBigD • Houston (TX)

On-site
USD 120,000 - 180,000
Data Engineer – Azure Databricks & Lakehouse (14‑Month)
Data Engineer – Azure Databricks & Lakehouse (14‑Month)

Robotic Process Automation LLC • Quincy (MA)

On-site
USD 80,000 - 106,000
Databricks Engineer - Lakehouse Data Pipelines & Spark
Databricks Engineer - Lakehouse Data Pipelines & Spark

Quantum Technologies LLC • Austin (TX)

On-site
USD 120,000 - 190,000
Azure Databricks Data Engineer — Lakehouse & Spark Expert
Azure Databricks Data Engineer — Lakehouse & Spark Expert

Quantum Technologies LLC • Dallas (TX)

On-site
USD 120,000 - 180,000