Senior Data Engineer — Databricks & Spark Pipelines

Prosum; Glendale (CA); On-site; USD 150,000 - 210,000; Full time

Job summary

Prosum in Glendale, CA is seeking a Senior Data Engineer to own the Core Data platform on Databricks, delivering batch and streaming Spark pipelines, platform governance, and solution architecture across AWS, Kubernetes and Airflow. You will collaborate with technologists to turn data into actionable insights that drive growth.

You will design and implement data pipelines using PySpark/Scala, work with stakeholders, maintain Unity Catalog governance, and ensure pipeline reliability with CI/CD

Qualifications

• 5+ years of data engineering experience developing and operating large-scale data pipelines.
• Deep hands-on experience with Databricks and Apache Spark (batch and streaming).
• Proficiency with Databricks platform tooling (API, SDK, CLI) for automation and governance.
• Proficient in SQL with advanced performance tuning.
• Hands-on production experience with Airflow (MWAA) for orchestrating data pipelines.

Responsibilities

• Design, write, test, and deploy data pipelines using PySpark, Scala, SQL, Python.
• Meet with stakeholders to gather requirements and translate them into scalable data platform solutions.
• Understand Databricks platform and tooling to diagnose errors, audit activity, and automate updates.

Skills

Databricks

Advanced SQL

Data pipelines

Education

Bachelor's degree

Tools

Airflow (MWAA)

Docker

Kubernetes

Snowflake

Job description

Prosum in Glendale, CA is seeking a Senior Data Engineer to own the Core Data platform on Databricks, delivering batch and streaming Spark pipelines, platform governance, and solution architecture across AWS, Kubernetes and Airflow. You will collaborate with technologists to turn data into actionable insights that drive growth.

You will design and implement data pipelines using PySpark/Scala, work with stakeholders, maintain Unity Catalog governance, and ensure pipeline reliability with CI/CD