Senior Data Engineer

Creative Solutions Services, LLC; Glendale (CA); Hybrid; USD 124,000 - 128,000; Full time

Job summary

Software Resources is seeking a Senior Data Engineer in Glendale, CA to design, build, and optimize scalable data solutions. You will own the Core Data platform on Databricks, delivering reliable batch and streaming Spark pipelines and governance across AWS, Kubernetes, and Airflow.

You will collaborate with cross-functional teams to modernize our data platform and drive data-driven decision making. The role requires 5+ years of data engineering experience, strong Databricks/Spark expertise, and

Qualifications

• 5+ years of data engineering experience developing and operating large-scale data pipelines.
• Deep hands-on experience with Databricks and Apache Spark (batch and streaming).
• Proficient in PySpark and Scala; strong SQL skills.
• Experience with Databricks platform tooling for automation, auditing, governance, and troubleshooting.
• Experience with Airflow (MWAA) for orchestrating pipelines.
• Proficiency with Python and at least one additional language (Scala/Kotlin).
• Experience designing and optimizing scalable ETL/ELT pipelines; cloud experience with AWS.

Responsibilities

• Manage Databricks governance including Unity Catalog, ACLs, lineage, and data discovery tooling.
• Design, write, test, and deploy batch and streaming data pipelines using PySpark, Scala, SQL, Python, Kotlin.
• Translate stakeholder requirements into scalable data platform solutions.
• Explain Spark architecture and diagnose issues for stakeholders.
• Provide solution architecture across AWS, Databricks, Kubernetes, and Airflow (MWAA).
• Build and maintain Kubernetes containers and utilities for data platform services.
• Administer platform access, monitor health, and optimize costs.
• Collaborate with engineers and product teams in agile ceremonies.
• Document platform changes, standards, and pipeline configurations.
• Develop and maintain scalable data engineering solutions and pipelines.

Skills

Databricks

Python

Terraform

Spark

SQL

Airflow

Kubernetes

Docker

MWAA

AWS

PySpark

Scala

Kotlin

Education

BA/BS in Computer Science, Information Systems, or related field

Master's Degree is a plus

Tools

Databricks CLI

MWAA

Job description

Software Resources has an immediate job opportunity for a Senior Data Engineer with a major corporation in Glendale, CA. Hybrid, 2 days onsite (candidate may select onsite days)

Duration: 12+ months

Pay Rate: $90 -$93/hr DOE

Position Overview:

Seeking an experienced Sr. Data Engineer to support enterprise data engineering initiatives. This individual will design, build, and optimize scalable data solutions while working closely with cross-functional teams to support data platform modernization efforts.

Must-Haves:

Databricks, Python, Terraform

Nice-to-Haves:

AWS

Description:

As a Senior Data Engineer at The Studios, you will be pivotal in transforming data into actionable insights. Collaborate with our dynamic team of technologists to develop cutting‑edge data solutions that drive innovation and fuel business growth. You will own and operate the Core Data platform on Databricks, delivering reliable batch and streaming Spark pipelines, platform governance, and solution architecture across AWS, Kubernetes, and Airflow. Your expertise will be essential in explaining Spark architecture, recommending best‑fit Databricks solutions to stakeholders, and optimizing our data‑driven decision‑making processes. If you're passionate about leveraging data to make a tangible impact, we welcome you to join us in shaping the future of our organization.

Key Responsibilities:

• Manage Databricks platform governance, including Unity Catalog, ACLs, lineage, and data discovery and privacy tooling
• Design, write, test, and deploy batch and streaming data pipelines using PySpark, Scala, SQL, Python, Kotlin
• Meet with stakeholders to gather requirements and translate them into scalable data platform solutions
• Understanding of Databricks platform and developer tooling to diagnose errors, audit platform activity, and automate updates across pipelines, objects, and integrations
• Ability to explain Spark architecture and pipeline behavior to stakeholders to diagnose root causes and recommend solutions
• Provide solution architecture across AWS, Databricks, Kubernetes, and Airflow (MWAA), including cross‑platform integrations
• Build and maintain Kubernetes containers and containerized utilities supporting deployed data platform services
• Apply networking knowledge to troubleshoot connectivity and integration errors across platform components
• Perform platform administration: provision and remove access, assess resource utilization, monitor platform health and cost, and evaluate stakeholder requests
• Collaborate with engineers, architects, and product managers to drive Core Data platform success; participate in agile/scrum ceremonies
• Maintain documentation of platform changes, standards, and pipeline configurations to support data quality and governance
• Develop and maintain scalable data engineering solutions
• Build and enhance data pipelines and workflows
• Support cloud‑based data platform initiatives
• Collaborate with engineering teams on system architecture and design
• Implement infrastructure automation and deployment standards
• Troubleshoot and optimize data processing environments

Qualifications:

• 5+ years of data engineering experience developing and operating large‑scale data pipelines
• Deep hands‑on experience with Databricks and Apache Spark (batch and streaming), including pipeline development in PySpark and/or Scala
• Strong understanding of Spark architecture—executors, stages, partitioning, shuffle, and performance tuning—with ability to explain tradeoffs to technical and non‑technical stakeholders
• Proficiency with Databricks platform tooling (API, SDK, CLI) for automation, auditing, governance, and operational troubleshooting
• Proficient in SQL with advanced performance tuning capabilities
• Hands‑on production experience with Airflow (MWAA) for orchestrating data pipelines
• Experience managing Databricks platform governance: ACLs, Unity Catalog, lineage, and access provisioning
• Proficiency in Python and at least one additional language (Scala, Kotlin, or SQL‑driven pipeline tooling)
• Experience designing and optimizing scalable ETL/ELT pipelines integrating diverse structured and unstructured data sources
• AWS‑primary experience (compute, storage, networking, IAM); experience with other cloud providers is transferable
• Proficiency with Docker and Kubernetes for building and maintaining containerized data platform services
• Working knowledge of networking concepts to diagnose cross‑platform integration and connectivity issues
• Familiarity with Snowflake and comparable tooling relative to the Databricks ecosystem
• Experience designing and implementing CI/CD and DevOps practices (Git‑based workflows)
• Experience implementing data quality checks, monitoring, and logging for pipeline reliability
• Self‑starting problem solver with strong analytical and communication skills; willingness to learn new tooling and trends
• Familiar with Scrum and Agile methodologies
• Bachelor's Degree in Computer Science, Information Systems, or a related field, or equivalent work experience, Master's Degree is a plus

Required Qualifications:

• Strong experience with Databricks
• Advanced Python development skills
• Experience with Terraform and Infrastructure as Code
• 5+ years of relevant data engineering experience
• Ability to participate in architecture and system design discussions
• Strong problem‑solving and data platform implementation experience

Preferred Qualifications:

• Experience with AWS
• Cloud‑based data platform experience
• Data pipeline optimization and automation expertise

Required Education:

BA/BS

About Software Resources

Software Resources, founded more than 3 decades ago, is a trusted staffing partner specializing in Technology (IT, Creative, & Marketing), Finance, & Accounting placements. We work with companies nationwide that need skilled professionals and connect them with candidates like you looking for the right opportunity.

We are always looking for professionals at all experience levels and in various specialties to meet our clients' hiring needs. As a member of our team, you will have access to benefits including medical, dental, and vision coverage, a 401(k) with company match, short‑term disability, life insurance with AD&D.