Data Engineer

Northern Base; Quincy (MA); On-site; USD 140,000 - 170,000; Full time

Job summary

Northern Base seeks a senior data engineer with deep Databricks expertise to design and optimize data pipelines, streaming workloads, and data transformations. You will implement scalable solutions using Spark, Scala and PySpark, and work with Delta Lake and Lakehouse patterns.

The role emphasizes performance tuning, large-scale data processing, and collaboration with DevOps teams to manage pipelines on Azure Databricks.

Qualifications

• 8+ years of data engineering experience with Databricks and ETL/ELT pipelines.
• Hands-on development in Scala, Spark, Spark SQL and PySpark.
• Experience in large-scale streaming and data transformations.
• Strong knowledge of Delta Lake, Unity Catalog and Lakehouse concepts.

Skills

Data engineering

ETL/ELT

SQL

Spark

Scala

PySpark

Data pipelines

Performance tuning

Tools

Databricks

Azure Databricks

Docker

Kubernetes

AKS

CI/CD pipelines

Job description

Must Have Technical/Functional Skills

• Must have atleast 8+ years of extensive Data engineering experience in Databricks, ETL/ELT using data pipelines, SQL/Procesurr experience
• Must have 8 years of hands on development proficiency in implementing Databricks solutions using scala and spark with data frames and notebooks/sql
• Must have 8+ years of hands on development and performance tuning/ enhancement experience in scala/pyspark and sql
• Must have strong experience with Databricks, including developing and optimizing spark jobs, data transformations, and data processing workflows
• Must have experience with complex and large volume data streaming and data transformations using Azure Databricks
• Must have extensive experience in datawarehouse / data lakehouse implementation
• Must have experience with monitoring, debugging, and resolving issues in kive Databricks jobs Must have strong hands on expertise in troubleshooting devops pipelines, azure , and AKS services The candidate shd be able to provide data ingestion, streaming and transformations solutions and do development work Experience with apache big data implementation with programming proficiency like pyspark, R or Java prior to Databricks is advantageous Good to have experience with containerization technologies such as docker and Kubernetes

Must have experience and a good understanding on the below topics

• Databricks delta lake storage
• Unity catalog
• Autoloader
• Lakehouse architecture
• Medallion architecture
• Databricks serverless options
• Dataset and dataframe
• Databricks clusters configuration and sizing
• Databricks job orchestration
• Spark structured streaming