Senior Data Engineer

Principal Mineral Holdings; Camden (SC); In loco; USD 120.000 - 180.000; Tempo pieno

Descrizione del lavoro

Principal Mineral Holdings in Camden, SC is seeking a Senior Data Engineer to design and deliver a Databricks lakehouse, robust ELT/ETL pipelines, and trusted data models for KPIs and ML workflows. You’ll work onsite with cross-functional teams to ingest data from ERP systems and external feeds, build central transformation logic with dbt, and ensure governance and security for government contracts.

The role emphasizes data quality, collaboration with ML/platform engineers, and scalable,

Competenze

• 4+ years of professional data engineering experience building production pipelines.
• Strong SQL and Python; experience with Spark or a comparable distributed engine.
• Hands-on experience with a lakehouse or cloud warehouse (Databricks preferred; Snowflake or BigQuery considered).
• Experience with dbt or a comparable transformation and testing framework.
• Experience with Git-based workflows, CI, and orchestration (Databricks Workflows, Airflow, or similar).
• Comfort working directly with non-technical business owners to define metrics and resolve data issues.
• Ability to work as a U.S. person and support government contract data requirements.

Mansioni

• Build production ELT/ETL pipelines that are scalable, reliable, and maintainable.
• Ingest data from internal ERP systems and external market, trade, and commodity API feeds.
• Develop central transformation logic using dbt to support analytics, data science, optimization, and ML workflows.
• Standardize operational KPIs in a unified semantic layer (yield, inventory, margins, component SKUs).
• Improve end-to-end lakehouse architecture with cost and scalability optimizations.
• Run QA checks and data quality validation; monitor and alert to ensure trusted data for downstream consumers.
• Collaborate with ML and platform engineers on model training, scoring, and deployment needs.

Conoscenze

SQL

Python

Spark

Strumenti

Databricks

dbt

Snowflake

BigQuery

Git

Airflow

CI

Databricks Workflows

Unity Catalog

Delta Lake

Descrizione del lavoro

Principal Mineral Holdings is building production-grade data capabilities to power analytics, automation, predictions (ML/AI), and daily decisions across global supply chains and manufacturing plants. As a Senior Data Engineer on an onsite team in Camden, SC, you will help design and deliver a Databricks lakehouse, robust ELT/ETL pipelines, and trusted data models that support operational KPIs and downstream machine learning workflows.

What you’ll be building

• Production batch and near-real-time ELT/ETL pipelines that are scalable, reliable, and maintainable
• Ingestion and curation of data from disparate sources, including internal ERP systems and external market, trade, and commodity API feeds
• Central transformation logic using dbt (or similar) to support analytics, data science, optimization, and agentic AI workflows
• Operational KPI standardization in a unified semantic layer (yield, inventory, margins, component SKUs)
• End-to-end lakehouse improvements, balancing cost and scalability through architecture and performance optimization

How the work supports trusted analytics and ML

• Run QA checks against source systems and business reports, reconciling discrepancies with data owners
• Implement data quality validation, monitoring, and alerting so downstream consumers can rely on the data
• Collaborate with ML and platform engineers on model training, scoring, and deployment needs, including feature pipeline support

Engineering standards, security, and governance

• Establish and follow best practices for data modeling, naming, version control, and documentation
• Ensure solutions meet security, privacy, lineage, and governance standards, including requirements associated with PMC’s U.S. government contracts
• Implement and review role-based access control and security-as-code with the DevOps/MLOps Engineer

Required qualifications

• 4+ years of professional data engineering experience building production pipelines
• Strong SQL and Python; experience with Spark or a comparable distributed engine
• Hands-on experience with a lakehouse or cloud warehouse (Databricks preferred; Snowflake or BigQuery considered)
• Experience with dbt or a comparable transformation and testing framework
• Experience with Git-based workflows, CI, and orchestration (Databricks Workflows, Airflow, or similar)
• Comfort working directly with non-technical business owners to define metrics and resolve data issues
• Ability to work as a U.S. person and support government contract data requirements

Technologies you may use

Databricks, Databricks lakehouse, dbt, SQL, Python, Spark, Snowflake, BigQuery, Git, CI, Databricks Workflows, Airflow, SAP, SAP ECC, SAP S/4HANA, Unity Catalog, Delta Lake

Preferred qualifications

• Experience extracting data from SAP (ECC or S/4HANA) or other ERP systems
• Experience in manufacturing, supply chain, or industrial data
• Experience with Unity Catalog, Delta Lake, and data quality tooling
• Experience supporting ML feature pipelines