Data Engineer

SFE; Quincy (MA); On-site; USD 120,000 - 135,000; Full time

Job summary

SFE in Quincy, MA is seeking an experienced Data Engineer to design, build, and optimize large-scale data pipelines on Azure Databricks. You will work with Spark, Scala, PySpark, and SQL to deliver reliable data ingestion and transformations.

The role focuses on Delta Lake, Lakehouse architectures, Unity Catalog, Autoloader, and real-time streaming using Spark Structured Streaming. You will collaborate with DevOps to troubleshoot pipelines and production workloads on Azure and AKS.

Qualifications

• 8+ years in data engineering with Databricks/ETL pipelines.
• Hands-on experience with Scala, Spark, and notebooks/SQL.
• Experience building batch and streaming pipelines on Azure.
• Knowledge of Delta Lake, Unity Catalog, Autoloader, Lakehouse/Medallion architectures.
• Strong troubleshooting of DevOps pipelines, Azure services, and AKS.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines on Azure Databricks.
• Implement Delta Lake, Unity Catalog, Autoloader, and Lakehouse patterns.
• Build real-time streaming pipelines with Spark Structured Streaming.
• Tune Spark jobs and optimize data transformations for large volumes.
• Collaborate with DevOps to manage production Databricks workloads.

Skills

Azure Databricks

Apache Spark

Scala

PySpark

SQL

Azure

Delta Lake

Lakehouse architectures

Tools

Databricks Notebooks

Spark Jobs

AKS

Job description

Job Title

Data Engineer

Job Type

Full Time

Client Location

Quincy,MA

Work Arrangement

Onsite

Duration

Full Time

Pay Rate / Salary

$120-$135k/yr.

Job Summary

• We are seeking an experienced Azure Databricks Data Engineer with 8+ years of hands-on experience in data engineering, ETL/ELT, distributed data processing, and large-scale data pipeline development.
• The ideal candidate will have strong expertise in Databricks,Apache Spark, Scala, PySpark, SQL, Azure, and data lakehousearchitectures, with proven experience designing and optimizing batch and streaming data solutions.
• The role involves developing scalable data ingestion, transformation, and processing pipelines using Azure Databricks, implementing Delta Lake and Medallion architectures, optimizing Spark workloads, and supporting high-volume streaming environments.
• The candidate will also work closely with DevOps teams to troubleshoot pipelines, Azure services, AKS environments, and production Databricks workloads.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Databricks, Spark, Scala, PySpark, and SQL.
• Develop complex data ingestion, transformation, aggregation, and processing workflows for large-volume structured and unstructured datasets.
• Build and optimize Databricks notebooks, Spark jobs, DataFrames, Datasets, and SQL-based solutions.
• Implement Delta Lake, Unity Catalog, Auto Loader, Lakehouse, and Medallion architecture patterns.
• Develop real-time and near-real-time data pipelines using Spark Structured Streaming.
• Design streaming solutions capable of processing high-volume data with appropriate performance, scalability, fault tolerance, and data-quality controls.
• Configure and optimize Databricks clusters, including sizing, autoscaling, compute configuration, and workload optimization.
• Develop and manage Databricks job orchestration, scheduling, dependencies, retries, alerts, and production workflows.
• Perform Spark, Scala/PySpark, and SQL performance tuning, including optimization of transformations, joins, partitioning, caching, file formats, and query execution.
• Design and implement enterprise data lake and data warehouse/lakehouse solutions using Azure and Databricks.
• Develop data pipelines using Databricks Serverless capabilities where appropriate.
• Monitor, debug, troubleshoot, and resolve issues in live/production Databricks jobs and data pipelines.

Required Qualifications

• Must have atleast 8+ years of extensive Data engineering experience in Databricks, ETL/ELT using data pipelines, SQL/Procesurr experience
• Must have 8 years of hands on development proficiency in implementing Databricks solutions using scala and spark with data frames and notebooks/sql
• Must have 8+ years of hands on development and performance tuning/ enhancement experience in scala/pyspark and sql
• Must have strong experience with Databricks, including developing and optimizing spark jobs, data transformations, and data processing workflows
• Must have experience and a good understanding on the below topics
• Databricks delta lake storage
• Unity catalog
• Autoloader
• Lakehouse architecture
• Medallion architecture
• Databricks serverless options
• Dataset and dataframe
• Databricks clusters configuration and sizing
• Databricks job orchestration
• Spark structured streaming
• Must have experience with complex and large volume data streaming and data transformations using Azure Databricks
• Must have extensive experience in datawarehouse / data lakehouse implementation
• Must have experience with monitoring, debugging, and resolving issues in kive Databricks jobs
• Must have strong hands on expertise in troubleshooting devops pipelines, azure , and AKS services
• The candidate shd be able to provide data ingestion, streaming and transformations solutions and do development work
• Experience with apache big data implementation with programming proficiency like pyspark, R or Java prior to Databricks is advantageous
• Good to have experience with containerization technologies such as docker and Kubernetes