Data Engineer

Tavant; Glendale (CA); On-site; USD 140,000 - 190,000; Full time

Job summary

Tavant in Glendale, CA seeks a Senior Data Engineer to own and operate the Core Data platform on Databricks, delivering reliable batch and streaming Spark pipelines and governance across AWS, Kubernetes, and Airflow.

You will collaborate with technologists to translate requirements into scalable data solutions, explain Spark architecture to stakeholders, and optimize data-driven decision-making processes. This role is onsite in Glendale with hybrid collaboration.

Qualifications

• Databricks experience required.
• Advanced SQL with performance tuning.
• Experience building and maintaining data pipelines and workflows.

Responsibilities

• Design, write, test, and deploy data pipelines using PySpark, Scala, SQL, Python.
• Meet with stakeholders to translate requirements into scalable data platform solutions.
• Understand Databricks platform and tooling to diagnose errors and automate updates.
• Explain Spark architecture and pipeline behavior to stakeholders.
• Provide solution architecture across AWS, Databricks, Kubernetes, and Airflow.

Skills

Databricks

SQL

Pipelines & workflows

Spark architecture

Communication

Education

Bachelor’s Degree in CS/IS or related

Tools

Databricks tooling

Airflow (MWAA)

AWS

Kubernetes

Docker

Snowflake

Job description

As a Senior Data Engineer, you will be pivotal in transforming data into actionable insights. Collaborate with our dynamic team of technologists to develop cutting-edge data solutions that drive innovation and fuel business growth. You will own and operate the Core Data platform on Databricks, delivering reliable batch and streaming Spark pipelines, platform governance, and solution architecture across AWS, Kubernetes, and Airflow. Your expertise will be essential in explaining Spark architecture, recommending best-fit Databricks solutions to stakeholders, and optimizing our data-driven decision-making processes. If you're passionate about leveraging data to make a tangible impact, we welcome you to join us in shaping the future of our organization.

LOCATION IS GLENDALE (GC3) 2-3 days onsite.

Key Responsibilities:

• Design, write, test, and deploy data pipelines using PySpark, Scala, SQL, Python
• Meet with stakeholders to gather requirements and translate them into scalable data platform solutions
• Understanding of Databricks platform and developer tooling to diagnose errors, audit platform activity, and automate updates across pipelines, objects, and integrations
• Ability to explain Spark architecture and pipeline behavior to stakeholders to diagnose root causes and recommend solutions
• Provide solution architecture across AWS, Databricks, Kubernetes, and Airflow (MWAA), including cross-platform integrations
• Manage Databricks platform governance, including Unity Catalog, ACLs, lineage, and data discovery and privacy tooling
• Build and maintain Kubernetes containers and containerized utilities supporting deployed data platform services
• Apply networking knowledge to troubleshoot connectivity and integration errors across platform components
• Perform platform administration: provision and remove access, assess resource utilization, monitor platform health and cost, and evaluate stakeholder requests
• Collaborate with engineers, architects, and product managers to drive Core Data platform success; participate in agile/scrum ceremonies
• Maintain documentation of platform changes, standards, and pipeline configurations to support data quality and governance

Qualifications:

• 5+ years of data engineering experience developing and operating large-scale data pipelines
• Deep hands-on experience with Databricks and Apache Spark (batch and streaming), including pipeline development in PySpark and/or Scala
• Strong understanding of Spark architecture—executors, stages, partitioning, shuffle, and performance tuning—with ability to explain tradeoffs to technical and non-technical stakeholders
• Proficiency with Databricks platform tooling (API, SDK, CLI) for automation, auditing, governance, and operational troubleshooting
• Proficient in SQL with advanced performance tuning capabilities
• Hands-on production experience with Airflow (MWAA) for orchestrating data pipelines
• Experience managing Databricks platform governance: ACLs, Unity Catalog, lineage, and access provisioning
• Proficiency in Python and at least one additional language (Scala, Kotlin, or SQL-driven pipeline tooling)
• Experience designing and optimizing scalable ETL/ELT pipelines integrating diverse structured and unstructured data sources
• AWS-primary experience (compute, storage, networking, IAM); experience with other cloud providers is transferable
• Proficiency with Docker and Kubernetes for building and maintaining containerized data platform services
• Working knowledge of networking concepts to diagnose cross-platform integration and connectivity issues
• Familiarity with Snowflake and comparable tooling relative to the Databricks ecosystem
• Experience designing and implementing CI/CD and DevOps practices (Git-based workflows)
• Experience implementing data quality checks, monitoring, and logging for pipeline reliability
• Self-starting problem solver with strong analytical and communication skills; willingness to learn new tooling and trends
• Familiar with Scrum and Agile methodologies
• Experience with Snowflake is a plus
• Bachelor’s Degree in Computer Science, Information Systems, or a related field, or equivalent work experience, Master’s Degree is a plus

Required Skills (Must-Have)

• Databricks experience (primary requirement)
• Advanced SQL skills
• Experience building and maintaining data pipelines and workflows

Preferred Skills (Nice-to-Have)

• Additional cloud-based data platform experience
• Candidate must be local to the Glendale area and able to work onsite 2-4 days per week.
• Team operates in a collaborative hybrid environment.

Interview Process

• Expected to follow the team's recent hiring process for similar data engineering positions.
• Typically consists of:
  • Technical interview
  • Final interview (potentially onsite)
• Total process generally spans 2-3 interview rounds.

Candidate Profile

• Demonstrate strong hands-on experience with Databricks-based solutions.
• Have expertise in SQL and workflow orchestration tools such as Airflow.
• Be capable of designing and supporting scalable data engineering architectures.

Work effectively in a collaborative team environment and hybrid workplace setting.