Data Engineer - Databricks & Spark on Azure

Quantum Technologies. LLC

Quincy (MA)

On-site

USD 80,000 - 107,000

Full time

2 hours ago

Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Quantum Technologies. LLC in Quincy, MA seeks a Data Engineer to design and optimize Spark-based data pipelines on Azure Databricks, building scalable lakehouse architectures for large volumes of data.

The role focuses on streaming and batch transformations, Databricks job orchestration, and SQL tuning. Collaboration with analytics teams is essential; Docker/Kubernetes knowledge is a plus.

Qualifications

Experience developing Spark jobs and data pipelines.
Strong SQL skills and performance tuning experience.
Experience with Azure Databricks and lakehouse architectures.

Responsibilities

Develop and optimize Spark jobs, data transformations, and processing workflows.
Implement complex data streaming and transformation solutions using Azure Databricks.
Design and implement data warehouse/lakehouse solutions.
Monitor, debug, and resolve issues in live Databricks jobs.
Troubleshoot DevOps pipelines, Azure, and AKS services.
Build data ingestion, streaming, and transformation solutions.

Skills

Apache Spark

Spark DataFrames

ETL/ELT

Azure Databricks

Databricks

Databricks Lakehouse

Structured Streaming

Data pipelines

Tools

Unity Catalog

Auto Loader

Databricks Serverless

Cluster Sizing

Job Orchestration

Job description

The role focuses on streaming and batch transformations, Databricks job orchestration, and SQL tuning. Collaboration with analytics teams is essential; Docker/Kubernetes knowledge is a plus.

Get your free, confidential resume review.

or drag and drop your file here.