Senior Data Engineer: Databricks & Streaming Pipelines

Creative Solutions Services, LLC; Glendale (CA); Hybrid; USD 124,000 - 128,000; Full time

Job summary

Software Resources is seeking a Senior Data Engineer in Glendale, CA to design, build, and optimize scalable data solutions. You will own the Core Data platform on Databricks, delivering reliable batch and streaming Spark pipelines and governance across AWS, Kubernetes, and Airflow.

You will collaborate with cross-functional teams to modernize our data platform and drive data-driven decision making. The role requires 5+ years of data engineering experience, strong Databricks/Spark expertise, and

Qualifications

• 5+ years of data engineering experience developing and operating large-scale data pipelines.
• Deep hands-on experience with Databricks and Apache Spark (batch and streaming).
• Proficient in PySpark and Scala; strong SQL skills.
• Experience with Databricks platform tooling for automation, auditing, governance, and troubleshooting.
• Experience with Airflow (MWAA) for orchestrating pipelines.
• Proficiency with Python and at least one additional language (Scala/Kotlin).
• Experience designing and optimizing scalable ETL/ELT pipelines; cloud experience with AWS.

Responsibilities

• Manage Databricks governance including Unity Catalog, ACLs, lineage, and data discovery tooling.
• Design, write, test, and deploy batch and streaming data pipelines using PySpark, Scala, SQL, Python, Kotlin.
• Translate stakeholder requirements into scalable data platform solutions.
• Explain Spark architecture and diagnose issues for stakeholders.
• Provide solution architecture across AWS, Databricks, Kubernetes, and Airflow (MWAA).
• Build and maintain Kubernetes containers and utilities for data platform services.
• Administer platform access, monitor health, and optimize costs.
• Collaborate with engineers and product teams in agile ceremonies.
• Document platform changes, standards, and pipeline configurations.
• Develop and maintain scalable data engineering solutions and pipelines.

Skills

Databricks

Python

Terraform

Spark

SQL

Airflow

Kubernetes

Docker

MWAA

AWS

PySpark

Scala

Kotlin

Education

BA/BS in Computer Science, Information Systems, or related field

Master's Degree is a plus

Tools

Databricks CLI

MWAA

Job description

Software Resources is seeking a Senior Data Engineer in Glendale, CA to design, build, and optimize scalable data solutions. You will own the Core Data platform on Databricks, delivering reliable batch and streaming Spark pipelines and governance across AWS, Kubernetes, and Airflow.

You will collaborate with cross-functional teams to modernize our data platform and drive data-driven decision making. The role requires 5+ years of data engineering experience, strong Databricks/Spark expertise, and