Databricks Data Engineer: Spark, Lakehouse & Streaming

Valiant Technologies LLC

Quincy (MA)

On-site

USD 77,000 - 110,000

Full time

4 hours ago

Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Valiant Technologies LLC is seeking a Data Engineer in Quincy, MA, to develop and optimize Spark-based data processing and streaming pipelines on Azure Databricks. The role involves design and implementation of lakehouse architectures and ongoing production support for Databricks jobs.

Ideal candidates have strong PySpark/Scala skills, SQL tuning experience, and familiarity with DevOps and cloud services. Experience with Docker/Kubernetes is a plus for deployment and scalability.

Qualifications

Experience with Spark-based data processing and ETL pipelines.
Implement streaming data solutions using Databricks.
Design data warehouse/lakehouse architectures.
Monitor and debug Databricks jobs in production.
Familiarity with DevOps pipelines and cloud services.

Responsibilities

Develop and optimize Spark jobs, data transformations, and processing workflows.
Implement complex data streaming and transformation solutions using Azure Databricks.
Design and implement data warehouse/lakehouse solutions.
Monitor, debug, and resolve issues in live Databricks jobs.
Troubleshoot DevOps pipelines, Azure, and AKS services.
Build data ingestion, streaming, and transformation solutions.

Skills

Spark DataFrames

ETL/ELT pipelines

Scala/PySpark

SQL performance tuning

Streaming data

Tools

Azure Databricks

Databricks Unity Catalog

Databricks Lakehouse

Databricks Job Orchestration

Docker

Kubernetes

Job description

Ideal candidates have strong PySpark/Scala skills, SQL tuning experience, and familiarity with DevOps and cloud services. Experience with Docker/Kubernetes is a plus for deployment and scalability.

Get your free, confidential resume review.

or drag and drop your file here.