Data Engineer (Manufacturing and Operations)

Comrise; Foster City (CA); On-site; USD 78,000 - 110,000; Full time

Job summary

Comrise is seeking a Data Engineer to design and maintain scalable data pipelines for manufacturing and operations. You will integrate SAP MES/HANA, ERP data, and diagnostics into a Databricks lakehouse, enabling dashboards, analytics, and AI-driven apps.

The role emphasizes hands-on data engineering (80%) with CI/CD, ETL pipeline reliability, and cloud ingestion (AWS) (Onsite in Foster City). MI, analytics, and manufacturing operations teams collaborate to migrate pipelines and models from

Qualifications

• Bachelor's degree in CS, IS, engineering, or equivalent experience.
• 5+ years in data engineering with Databricks, SQL, and Python.
• Experience building data pipelines, dbt models, and CI/CD pipelines.

Responsibilities

• Design, develop, and maintain batch and streaming data pipelines into Delta Lake.
• Build bronze/silver/gold data layers with medallion architecture and governance.
• Develop incremental dbt models and implement SCD Type 2 patterns.
• Migrate pipelines/data from Redshift to Databricks and validate results.
• Configure CI/CD workflows with GitHub Actions and Databricks asset bundles.
• Manage AWS ingestion services (S3, DMS, Kinesis) and Unity Catalog permissions.
• Monitor pipeline health, costs, and DBU consumption; maintain docs.

Skills

SQL

Python

PySpark

Databricks

Delta Lake

Unity Catalog

dbt

CI/CD

GitHub Actions

Terraform

S3 / AWS data services

Education

Bachelor's degree

Tools

Databricks

GitHub Actions

Git

Airflow

Terraform

DMS / Kinesis

S3

Job description

Data Engineer (Manufacturing and Operations)

Location: Foster City, CA 94404 (Onsite)

Contract: 12 Months (Potential extension or conversion)

Pay Rate: $68.20/hour (W2 Only)

Industry: Automotive Technology / Autonomous Mobility

Note: No C2C (Corp-to-Corp) or 1099. No visa sponsorship. Candidates must be legally authorized to work in the United States for the duration of the assignment.

Job Overview

We are seeking an experienced Data Engineer to design, develop, and maintain scalable data pipelines and data models supporting manufacturing intelligence and operational analytics.

In this role, you will integrate data from manufacturing systems, enterprise databases, SAP MES, SAP HANA, and testing and diagnostic platforms into a centralized Databricks lakehouse environment. Your work will help transform complex manufacturing and operational data into reliable, high-quality datasets that support reporting, dashboards, analytics, and AI-driven applications.

Approximately 80% of the role involves hands-on data engineering, including pipeline development, data transformation, and modeling. The remaining 20% focuses on CI/CD, AWS ingestion infrastructure, Databricks configuration, and pipeline reliability.

You will also play an important role in supporting a data platform migration from Amazon Redshift to Databricks while collaborating with data engineering, analytics, manufacturing operations, and infrastructure teams.

Key Responsibilities

• Design, develop, and maintain batch and streaming data pipelines into Delta Lake using Databricks Auto Loader, Lakeflow, Structured Streaming, and related ingestion technologies.
• Build and manage bronze, silver, and gold data layers using medallion architecture and established data governance standards.
• Develop incremental dbt models using SQL, MERGE, upsert operations, and Slowly Changing Dimension (SCD) Type 2 patterns.
• Integrate and transform manufacturing data, including work orders, production records, nonconformances, station results, unit-level traceability, and first-pass yield.
• Process large-scale and semi-structured datasets using Python, PySpark, and Databricks SQL.
• Optimize data pipeline performance and cost through clustering, partitioning, file sizing, OPTIMIZE, VACUUM, and compute configuration.
• Implement data quality checks, pipeline monitoring, freshness SLAs, automated alerts, and reliability controls.
• Orchestrate data workflows using Databricks Workflows or Apache Airflow.
• Support the migration of existing data pipelines and models from Amazon Redshift to Databricks, including SQL conversion, data validation, reconciliation, and production cutover.
• Build and maintain CI/CD workflows using GitHub Actions, Databricks Asset Bundles, and Git-based deployment practices.
• Manage Databricks configurations, including Unity Catalog permissions, compute resources, SQL warehouses, service principals, secrets, and authentication.
• Configure and troubleshoot AWS ingestion services, including S3, DMS, Kinesis, and IAM, in collaboration with infrastructure teams.
• Monitor pipeline health, performance, and Databricks Unit (DBU) consumption while maintaining technical documentation and supporting incident resolution.

Required Qualifications

• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field, or equivalent practical experience.
• 5+ years of professional experience in data engineering or a closely related discipline.
• Strong programming skills in SQL and Python, including hands-on PySpark experience.
• Production-level experience with Databricks, including Delta Lake, Unity Catalog, Databricks Workflows or Jobs, and Spark performance optimization.
• Hands-on experience developing and deploying dbt models, including incremental models and automated testing.
• Strong understanding of dimensional data modeling, medallion architecture, change data capture (CDC), and incremental data processing.
• Experience implementing CI/CD pipelines and Git-based development and deployment workflows.
• Working knowledge of AWS data services, particularly S3, IAM, DMS, and/or Kinesis.
• Familiarity with infrastructure-as-code tools such as Terraform.

Preferred Qualifications

• Databricks Certified Data Engineer Associate or Professional certification.
• AWS certification, such as AWS Solutions Architect Associate.
• Experience working with manufacturing, industrial, or production-related datasets, particularly MES, quality, yield, and traceability data.
• Experience supporting Redshift-to-Databricks or similar data platform migrations.
• Familiarity with Databricks AI/BI Dashboards, Genie, machine learning, or LLM-related data use cases.
• Understanding of data governance, data lineage, and compliance requirements within enterprise data environments.