Data Engineer - Fleet Operations

Hireart; Foster City (CA); On-site; USD 78,000 - 110,000; Full time

Benefits offered by this job

Pre‑tax commuter benefits

Healthcare benefits

FSA for healthcare costs

Disability and life insurance covered

401k package

Job summary

HireArt is assisting an on-demand ride-hailing company in Foster City, CA to hire a Data Engineer - Fleet Operations. You will design, build, and maintain data pipelines and models powering the Manufacturing Intelligence data lakehouse and collaborate with cross-functional teams across data engineering, analytics, and operations.

The role emphasizes 80% hands-on data engineering with 20% on CI/CD, AWS ingestion infrastructure, Databricks configuration, and Redshift-to-Databricks migration,

Qualifications

• Bachelor's degree in computer science, information systems, engineering, or related discipline, or equivalent experience
• 6+ years of data engineering experience
• Strong proficiency in SQL, Python, and PySpark
• Hands-on production experience with Databricks, Delta Lake, Unity Catalog permissions, Workflows, and Spark tuning
• Production experience with dbt, including incremental modeling and testing
• Strong understanding of dimensional modeling, medallion architecture, CDC, and incremental data loading
• Experience deploying data code via Git-based workflows and CI/CD tools such as GitHub Actions or Databricks Asset Bundles
• Working knowledge of AWS data services (S3, IAM, DMS, Kinesis) and IaC tools like Terraform

Responsibilities

• Build and maintain batch and streaming ingestion pipelines into Delta Lake
• Design and maintain bronze, silver, and gold data layers following MOPS concepts
• Develop incremental dbt models on Databricks with MERGE, upsert, and SCD Type 2 patterns
• Model manufacturing data to support analytics, dashboards, and reporting
• Process large-scale datasets using PySpark and Databricks SQL, optimizing performance and costs
• Maintain data quality and pipeline reliability with tests, expectations, and monitoring
• Orchestrate data workloads with Databricks Workflows or Airflow and support migration from Redshift to Databricks
• Collaborate with analysts and manufacturing teams to define data requirements and document lineage
• Build and maintain CI/CD pipelines using Databricks Asset Bundles and GitHub Actions
• Manage Databricks configurations and governance in Unity Catalog
• Configure AWS ingestion infrastructure and coordinate Terraform provisioning
• Monitor pipeline health, performance, and DBU costs; maintain docs and incident response

Skills

SQL

Python

PySpark

Databricks

Delta Lake

Unity Catalog

dbt

CI/CD

AWS

Terraform

Education

Bachelor's degree in Computer Science, Information Systems, Engineering

Tools

Databricks

Delta Lake

Unity Catalog

dbt

GitHub Actions

Terraform

AWS

DMS

Kinesis

Job description

Expected compensation: 68.20 USD Per Hour

HireArt is helping an on-demand, autonomous ride-hailing company find a Data Engineer - Fleet Operations to design, build, and maintain data pipelines and models that power its Manufacturing Intelligence data lakehouse.

In this role, you'll integrate manufacturing data from on-premises databases, SAP MES, SAP HANA, and testing and diagnostic systems into the Manufacturing Operations (MOPS) lakehouse, transforming it into reliable datasets for reporting, dashboards, and AI applications. Approximately 80% of your work will focus on hands‑on data engineering, with the remaining 20% dedicated to CI/CD, AWS ingestion infrastructure, Databricks configuration, and pipeline reliability. You'll also support the migration from Redshift to Databricks while collaborating with data engineering, analytics, manufacturing operations, and platform teams.

The ideal candidate has extensive experience building production‑grade data pipelines and models, with strong expertise in Databricks, SQL, Python, and cloud‑based data infrastructure. They're detail‑oriented, proactive, and skilled at optimizing data performance, maintaining data quality, and collaborating across technical and operational teams.

As a Data Engineer - Fleet Operations, You'll

• Build and maintain batch and streaming ingestion pipelines into Delta Lake using Auto Loader, Lakeflow Declarative Pipelines, Lakeflow Connect, or Structured Streaming, including change data capture (CDC) feeds from AWS DMS and SAP SLT.
• Design and maintain bronze, silver, and gold data layers following MOPS naming conventions and data dictionary standards.
• Develop incremental dbt models on Databricks using MERGE, upsert, and Slowly Changing Dimension (SCD) Type 2 patterns, including handling late or out-of-order records.
• Model manufacturing data, including unit-level traceability (SFC), work orders, nonconformances, station results, and first-pass yield, to support analytics, dashboards, and semantic reporting.
• Process large-scale and semi-structured datasets using PySpark and Databricks SQL, optimizing performance and cost through clustering, partitioning, OPTIMIZE, VACUUM, file sizing, and compute configuration.
• Maintain data quality and pipeline reliability using dbt tests, Lakeflow expectations, Lakehouse Monitoring, freshness SLAs, and automated alerts.
• Orchestrate data workloads using Databricks Workflows or Airflow and support Redshift-to-Databricks migration through SQL conversion, parallel validation, reconciliation, and cutover.
• Collaborate with analysts and manufacturing teams to define data requirements and document lineage, data contracts, and table definitions in Unity Catalog.
• Build and maintain CI/CD pipelines using Databricks Asset Bundles and GitHub Actions across development, staging, and production environments.
• Manage MOPS-specific Databricks configurations, including Unity Catalog permissions, compute resources, SQL warehouses, service principals, secrets, and OAuth authentication, in alignment with central platform governance.
• Configure and troubleshoot AWS ingestion infrastructure, including DMS, Kinesis, S3, and IAM, while coordinating Terraform-based provisioning with infrastructure teams.
• Monitor pipeline health, performance, and Databricks Unit (DBU) costs, maintain operational documentation, and support incident response and root cause analysis.

Requirements

• A bachelor's degree in computer science, information systems, engineering, or a related discipline, or equivalent practical experience
• 6+ years of data engineering experience
• Strong proficiency in SQL, Python, and PySpark
• Hands‑on production experience with Databricks, including Delta Lake, Unity Catalog permissions, Workflows, and Spark performance tuning
• Production experience with dbt, including incremental modeling and testing
• Strong understanding of dimensional modeling, medallion architecture, CDC, and incremental data loading
• Experience deploying data code through Git‑based workflows and CI/CD tools such as GitHub Actions or Databricks Asset Bundles
• Working knowledge of AWS data services, including S3, IAM, DMS, or Kinesis, and infrastructure‑as‑code tools such as Terraform

Preferred Qualifications

• Databricks Certified Data Engineer Associate or Professional certification; AWS certification, such as Solutions Architect Associate, is a plus
• Experience preparing data for Databricks AI/BI dashboards, Genie, machine learning, or large language model applications
• Experience with manufacturing or industrial data, including quality, yield, traceability, or manufacturing execution systems (MES)
• Understanding of data governance and compliance requirements related to data pipelines

Benefits

• Pre‑tax commuter benefits
• Employer (HireArt) subsidized healthcare benefits (eligibility begins on the first of the month following 60 days of service)
• Flexible Spending Account for healthcare‑related costs
• HireArt covers all costs for short‑term and long‑term disability and life insurance
• 401k package

Commitment

This is a full‑time, 12‑month contract position staffed via HireArt. It will be onsite and available to candidates who are local to the Foster City, CA area.

HireArt values diversity and is an Equal Opportunity Employer. We are interested in every qualified candidate who is eligible to work in the United States. Unfortunately, we are not able to sponsor visas or employ corp‑to‑corp.