Senior Databricks Data Engineer: Spark & Lakehouse

Northern Base; Quincy (MA); On-site; USD 140,000 - 170,000; Full time

Job summary

Northern Base seeks a senior data engineer with deep Databricks expertise to design and optimize data pipelines, streaming workloads, and data transformations. You will implement scalable solutions using Spark, Scala and PySpark, and work with Delta Lake and Lakehouse patterns.

The role emphasizes performance tuning, large-scale data processing, and collaboration with DevOps teams to manage pipelines on Azure Databricks.

Qualifications

• 8+ years of data engineering experience with Databricks and ETL/ELT pipelines.
• Hands-on development in Scala, Spark, Spark SQL and PySpark.
• Experience in large-scale streaming and data transformations.
• Strong knowledge of Delta Lake, Unity Catalog and Lakehouse concepts.

Skills

Data engineering

ETL/ELT

SQL

Spark

Scala

PySpark

Data pipelines

Performance tuning

Tools

Databricks

Azure Databricks

Docker

Kubernetes

AKS

CI/CD pipelines

Job description

Northern Base seeks a senior data engineer with deep Databricks expertise to design and optimize data pipelines, streaming workloads, and data transformations. You will implement scalable solutions using Spark, Scala and PySpark, and work with Delta Lake and Lakehouse patterns.

The role emphasizes performance tuning, large-scale data processing, and collaboration with DevOps teams to manage pipelines on Azure Databricks.