Data Engineer - Fleet Operations

An Autonomous Mobility Company

Foster City (CA)

Sur place

USD 83 000 - 107 000

Plein temps

Il y a 2 jours
Soyez parmi les premiers à postuler

Obtenez une réponse de cet employeur

Votre CV et votre lettre de motivation, adaptés exactement à ce que l’employeur recherche.

Passez les filtres ATS

Avantages offerts par ce poste

Pre‑tax commuter benefits
Healthcare benefits (HireArt)
Flexible Spending Account
Disability and life insurance
401k package

Résumé du poste

An Autonomous Mobility Company in Foster City, CA is seeking a senior data engineer to design, build, and maintain batch and streaming ingestion pipelines for manufacturing intelligence and fleet operations. You will develop incremental dbt models on Databricks and implement SCD Type 2 patterns to handle late or out-of-order records.

The role requires 6+ years in data engineering, strong SQL/Python, and hands-on Databricks experience.

Qualifications

  • Bachelor’s degree in computer science, information systems, engineering, or related discipline or equivalent experience.
  • 6+ years of data engineering experience.
  • Strong proficiency in SQL, Python, and PySpark.
  • Hands-on production experience with Databricks, Delta Lake, Unity Catalog permissions, Workflows, and Spark tuning.
  • Production experience with dbt, including incremental modeling and testing.
  • Strong understanding of dimensional modeling, medallion architecture, CDC, and incremental data loading.
  • Experience with Git-based workflows and CI/CD tools like GitHub Actions or Databricks Asset Bundles.
  • Working knowledge of AWS data services (S3, IAM, DMS, Kinesis) and Terraform.

Responsabilités

  • Design, build, and maintain batch and streaming ingestion pipelines into Delta Lake.
  • Implement CDC feeds from AWS DMS and SAP SLT.
  • Set up bronze, silver, and gold data layers following MOPS naming conventions and data dictionary standards.
  • Develop incremental dbt models on Databricks using MERGE and upsert patterns.
  • Apply SCD Type 2 patterns in dbt, including late/out-of-order records support.
  • Model manufacturing datasets to support analytics: unit-level traceability, work orders, nonconformances, station results, first-pass yield.
  • Process large-scale data with PySpark and Databricks SQL; optimize performance and cost.

Connaissances

SQL proficiency
Python proficiency
Data engineering

Formation

Bachelor’s degree in CS/IS/Engineering

Outils

Databricks
Delta Lake
Unity Catalog
Airflow
GitHub Actions
Terraform
DMS
Kinesis
S3
IAM

Description du poste

Build and operate the data foundation for manufacturing intelligence and fleet operations using modern lakehouse and streaming technologies.

Responsibilities
  • Design, build, and maintain batch and streaming ingestion pipelines into Delta Lake using Auto Loader, Lakeflow Declarative Pipelines, Lakeflow Connect, or Structured Streaming.
  • Implement change data capture (CDC) feeds sourced from AWS DMS and SAP SLT.
  • Set up and maintain bronze, silver, and gold data layers following MOPS naming conventions and data dictionary standards.
  • Develop incremental dbt models on Databricks, using MERGE and upsert patterns.
  • Apply Slowly Changing Dimension (SCD) Type 2 patterns in dbt, including support for late or out-of-order records.
  • Model manufacturing datasets to support analytics and reporting, including unit-level traceability (SFC), work orders, nonconformances, station results, and first-pass yield.
  • Process large-scale and semi-structured data using PySpark and Databricks SQL.
  • Optimize performance and cost using clustering, partitioning, OPTIMIZE, VACUUM, file sizing, and compute configuration.
  • Improve reliability and data quality with dbt tests, Lakeflow expectations, and Lakehouse Monitoring.
  • Set and manage freshness SLAs and implement automated alerts.
  • Orchestrate pipelines using Databricks Workflows or Airflow.
  • Support Redshift-to-Databricks migration via SQL conversion, parallel validation, reconciliation, and cutover.
  • Collaborate with analysts and manufacturing teams to define data requirements, document lineage, and publish data contracts and table definitions in Unity Catalog.
  • Maintain CI/CD using Databricks Asset Bundles and GitHub Actions across development, staging, and production.
  • Manage MOPS-specific Databricks configuration, including Unity Catalog permissions, compute resources, SQL warehouses, service principals, secrets, and OAuth authentication under central platform governance.
  • Configure and troubleshoot AWS ingestion components including DMS, Kinesis, S3, and IAM, coordinating Terraform-based provisioning with infrastructure teams.
  • Monitor pipeline health, performance, and Databricks Unit (DBU) costs; maintain operational documentation and support incident response and root cause analysis.
Requirements
  • Bachelor’s degree in computer science, information systems, engineering, or a related discipline (or equivalent practical experience).
  • 6+ years of data engineering experience.
  • Strong proficiency in SQL, Python, and PySpark.
  • Hands‑on production experience with Databricks, including Delta Lake, Unity Catalog permissions, Workflows, and Spark performance tuning.
  • Production experience with dbt, including incremental modeling and testing.
  • Strong understanding of dimensional modeling, medallion architecture, CDC, and incremental data loading.
  • Experience deploying data code through Git‑based workflows and CI/CD tools such as GitHub Actions or Databricks Asset Bundles.
  • Working knowledge of AWS data services including S3, IAM, DMS, or Kinesis, and infrastructure‑as‑code tools such as Terraform.
Technologies
  • Databricks, Delta Lake, Auto Loader, Lakeflow Declarative Pipelines, Lakeflow Connect, Structured Streaming
  • AWS DMS, SAP SLT, dbt, MERGE, upsert, Slowly Changing Dimension (SCD) Type 2
  • PySpark, Databricks SQL, OPTIMIZE, VACUUM, Lakehouse Monitoring
  • Databricks Workflows, Airflow, Redshift, Unity Catalog
  • Databricks Asset Bundles, GitHub Actions, Unity Catalog permissions, SQL warehouses
  • service principals, OAuth authentication, AWS Kinesis, AWS S3, AWS IAM, Terraform, Databricks Unit (DBU)
  • Git‑based workflows, SAP MES, SAP HANA, SFC
Benefits
  • Pre‑tax commuter benefits
  • Employer (HireArt) subsidized healthcare benefits (eligibility begins on the first of the month following 60 days of service)
  • Flexible Spending Account for healthcare‑related costs
  • HireArt covers all costs for short‑and long‑term disability and life insurance
  • 401k package
Preferred Qualifications
  • Databricks Certified Data Engineer Associate or Professional certification; AWS certification such as Solutions Architect Associate is a plus
  • Experience preparing data for Databricks AI/BI dashboards, Genie, machine learning, or large language model applications
  • Experience with manufacturing or industrial data, including quality, yield, traceability, or manufacturing execution systems (MES)
  • Understanding of data governance and compliance requirements related to data pipelines
Job Details
  • Location: Foster City, CA (onsite)
  • Work type: contract
  • Salary: USD 68 per hour
  • Commitment: full‑time, 12‑month contract position staffed via HireArt
  • Availability: onsite role for candidates local to the Foster City, CA area
Obtenez votre examen gratuit et confidentiel de votre CV.

ou faites glisser et déposez votre fichier ici.

Similar jobs

Postes similaires à comparer

Data Engineer - Fleet Operations
Data Engineer - Fleet Operations

HireArt, Inc. • Foster City (CA)

Sur place
USD 150 000 - 210 000
Employer subsidized healthcare
Flexible Spending Account
Disability and life insurance coverage
+1
Data Engineer - Fleet Operations
Data Engineer - Fleet Operations

Hireart • Foster City (CA)

Sur place
USD 78 000 - 110 000
Pre‑tax commuter benefits
Healthcare benefits
FSA for healthcare costs
+2
Senior Data Engineer – Databricks & AWS
Senior Data Engineer – Databricks & AWS

Robotics Technologies • Sunnyvale (CA)

Sur place
USD 76 000 - 108 000
Manufacturing Data Engineer
Manufacturing Data Engineer

DeWinter BH • Foster City (CA)

Sur place
USD 140 000 - 190 000
Senior Databricks Data Engineer
Senior Databricks Data Engineer

CGI Group, Inc. • Miami (FL)

Hybride
USD 92 000 - 204 000
Competitive compensation
Comprehensive insurance options
401(k) match and share purchase plan
+3
Senior Data Engineer (Databricks) - Contractor
Senior Data Engineer (Databricks) - Contractor

Jobgether • États-Unis

À distance
USD 96 000 - 165 000
Remote work
Contract 6 months
Extension possibility
Manufacturing Data Engineer
Manufacturing Data Engineer

Ursus, Inc. • Foster City (CA)

Sur place
USD 94 000 - 107 000
Medical benefits
Dental benefits
Vision benefits
+2
Senior Data Engineer
Senior Data Engineer

Hillwood Development Company LLC • Dallas (TX)

Sur place
USD 140 000 - 180 000
Medical Insurance
Dental Insurance
Vision Insurance
+12
Senior Data Engineer - Databricks
Senior Data Engineer - Databricks

Capgemini • New York (NY)

Sur place
USD 89 000 - 166 000
Paid time off
Health benefits
401(k) plan
+1
Senior Azure Data Engineer
Senior Azure Data Engineer

Yoh, A Day & Zimmermann Company • Plano (TX)

Sur place
USD 105 000 - 150 000
Medical, Prescription, Dental & Vision
Health Savings Account (HSA)
Life & Disability Insurance
+6