This Data Engineer role supports manufacturing-operations analytics by building and operating a batch and streaming data platform. The work includes developing CDC pipelines from SAP and industrial sources, implementing a Medallion lakehouse architecture, and delivering semantic layers for analytics and downstream applications.
Responsibilities
- Design and build batch and streaming Change Data Capture (CDC) pipelines from scratch to ingest manufacturing operations data from SAP (MES, HANA, SLT), on-premises databases, AWS DMS, and industrial sources such as PLCs.
- Develop and maintain Medallion lakehouse layers (Bronze, Silver, Gold) using dimensional models tailored to manufacturing operations topics including unit traceability, work orders, non-conformance reporting, assembly line station results, logistics, and first-pass yield.
- Build incremental SQL and dbt models, including MERGE and upsert patterns, SCD Type 2, and late-arriving data handling; process semi-structured test and diagnostic payloads using Python and PySpark.
- Monitor data pipeline health, data quality, and freshness SLAs using AWS CloudWatch alarms, log monitoring, Grafana dashboards, and automated data quality tests.
- Orchestrate production workflows with Apache Airflow and manage AWS ingestion infrastructure including S3, IAM, DMS, and Kinesis using Git-based CI/CD.
- Collaborate with cross-functional teams to build semantic layers that support business intelligence dashboards and downstream analytical applications.
Required Qualifications
- 5 to 10 years of hands-on data engineering experience.
- Proven experience building data pipelines in manufacturing, industrial, automotive, or semiconductor operations.
- Direct, hands-on experience with SAP (ERP, MES, HANA, or SLT) and strong familiarity with factory-floor concepts such as non-conformances, production testing or diagnostics, yield, work orders, and supply chain logistics.
- Deep, hands-on experience building CDC pipelines from scratch across relational databases and operational or industrial sources.
- Strong proficiency in dimensional modeling, Medallion architecture (Bronze/Silver/Gold), and building semantic layers for analytics.
- Advanced programming skills in Python, PySpark, and SQL, plus production experience in AWS services including S3, DMS, Kinesis, CloudWatch, and IAM, along with workflow orchestration using Airflow or similar tools.
Preferred Qualifications
- Production experience with dbt, Amazon Redshift, Databricks, or Grafana.
- Experience structuring lakehouse data to support AI and agentic applications.
Key Technologies
Change Data Capture (CDC), SAP, MES, HANA, SLT, AWS DMS, AWS CloudWatch, Grafana, Apache Airflow, S3, IAM, Kinesis, Git-based CI/CD, Python, PySpark, SQL, dbt, MERGE, SCD Type 2.
Work Authorization and Location
- Applicants must be authorized to work for ANY employer in the U.S. The employer is unable to sponsor or take over sponsorship of an employment Visa at this time.
- Location: Foster City, CA (onsite), #LI-Onsite.
Compensation
USD 69 - 76 per hour.