Senior Azure Databricks Data Engineer – Onsite in Quincy

SFE; Quincy (MA); On-site; USD 120,000 - 135,000; Full time

Job summary

SFE in Quincy, MA is seeking an experienced Data Engineer to design, build, and optimize large-scale data pipelines on Azure Databricks. You will work with Spark, Scala, PySpark, and SQL to deliver reliable data ingestion and transformations.

The role focuses on Delta Lake, Lakehouse architectures, Unity Catalog, Autoloader, and real-time streaming using Spark Structured Streaming. You will collaborate with DevOps to troubleshoot pipelines and production workloads on Azure and AKS.

Qualifications

• 8+ years in data engineering with Databricks/ETL pipelines.
• Hands-on experience with Scala, Spark, and notebooks/SQL.
• Experience building batch and streaming pipelines on Azure.
• Knowledge of Delta Lake, Unity Catalog, Autoloader, Lakehouse/Medallion architectures.
• Strong troubleshooting of DevOps pipelines, Azure services, and AKS.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines on Azure Databricks.
• Implement Delta Lake, Unity Catalog, Autoloader, and Lakehouse patterns.
• Build real-time streaming pipelines with Spark Structured Streaming.
• Tune Spark jobs and optimize data transformations for large volumes.
• Collaborate with DevOps to manage production Databricks workloads.

Skills

Azure Databricks

Apache Spark

Scala

PySpark

SQL

Azure

Delta Lake

Lakehouse architectures

Tools

Databricks Notebooks

Spark Jobs

AKS

Job description

SFE in Quincy, MA is seeking an experienced Data Engineer to design, build, and optimize large-scale data pipelines on Azure Databricks. You will work with Spark, Scala, PySpark, and SQL to deliver reliable data ingestion and transformations.

The role focuses on Delta Lake, Lakehouse architectures, Unity Catalog, Autoloader, and real-time streaming using Spark Structured Streaming. You will collaborate with DevOps to troubleshoot pipelines and production workloads on Azure and AKS.