Senior Data Engineer

Camden Copper/Metis Endeavor; Camden (SC); In loco; USD 120.000 - 160.000; Tempo pieno

Descrizione del lavoro

Camden Copper/Metis Endeavor is hiring a Senior Data Engineer to build and scale our Databricks lakehouse, turning fragmented data into reliable assets that empower analytics, AI, and decision-making across global operations.

You will architect batch and near-real-time pipelines, curate data from ERP and external feeds, and partner with business owners to define KPIs in a unified semantic layer, ensuring security, lineage, and governance for government contracts.

Competenze

• 4+ years of professional data engineering experience building production pipelines.
• Strong SQL and Python; experience with Spark or a comparable distributed engine.
• Hands-on experience with a lakehouse or cloud warehouse (Databricks preferred; Snowflake or BigQuery considered).
• Experience with dbt or a comparable transformation and testing framework.
• Experience with Git-based workflows, CI, and orchestration (Databricks Workflows, Airflow, or similar).
• Comfort working directly with non-technical business owners to define metrics and resolve data issues.
• Ability to work as a U.S. person and support government contract data requirements.

Mansioni

• Architect, build, and maintain batch and near-real-time ELT/ETL pipelines that are scalable, reliable, and maintainable.
• Ingest and curate data from disparate systems including ERP systems and external feeds.
• Design, build, and maintain central data models in dbt to support analytics, data science, optimization, and AI workflows.
• Partner with business owners to define and standardize KPIs in a unified semantic layer.
• Perform QA checks against source systems and reconcile discrepancies with data owners.
• Implement data quality validation, monitoring, and alerting for downstream trust.
• Collaborate with ML and platform engineers on model training, scoring, and deployment.

Conoscenze

SQL

Python

Spark

Strumenti

Databricks

dbt

Git-based workflows

Airflow

Databricks Workflows

Descrizione del lavoro

THE ROLE

Camden Copper/Metis Endeavor, through Principal Mineral, brings together data from operating companies and trusted external sources to understand operations and optimize decisions across global supply chains and manufacturing plants.

As a Senior Data Engineer, you'll be a foundational builder of our Databricks lakehouse, turning fragmented information into production-grade pipelines and reliable data assets that support analytics, automation, predictions (ML/AI), and day-to-day decision-making.

You'll work across the data lifecycle: extracting raw data from legacy and ERP systems (including SAP and custom SQL databases), transforming and modeling data into clean, standardized, business-ready formats using dbt (or similar), and validating that numbers match what teams see in the business. You'll collaborate with business owners to define, validate, and improve the KPIs your data powers.

ESSENTIAL FUNCTIONS

Core ingestion pipelines

• Architect, build, and maintain batch and near-real-time ELT/ETL pipelines that are scalable, reliable, and maintainable.

• Ingest and curate data from disparate systems, including internal ERP systems and external market, trade, and commodity API feeds.

Data quality and transformation

• Design, build, and maintain central data models in dbt (or similar) that support analytics, data science, optimization, and agentic AI workflows.

• Partner with business owners and subject matter experts to define and standardize operational KPIs (yield, inventory, margins, component SKUs) in a unified semantic layer.

• Perform QA checks against source systems and business reports; reconcile discrepancies with data owners.

• Implement data quality validation, monitoring, and alerting so downstream consumers can trust the data.

• Help establish and follow best practices for data modeling, naming, version control, and documentation.

Platform, security, and collaboration

• Optimize lakehouse architecture and pipeline performance while balancing cost and scalability.

• Collaborate with ML and platform engineers on model training, scoring, and deployment needs.

• Ensure solutions meet standards for security, privacy, lineage, and governance, including requirements tied to PMC's U.S. government contracts.

• Implement and review role-based access control and security-as-code with the DevOps/MLOps Engineer.

QUALIFICATIONS

Required

• 4+ years of professional data engineering experience building production pipelines.

• Strong SQL and Python; experience with Spark or a comparable distributed engine.

• Hands-on experience with a lakehouse or cloud warehouse (Databricks preferred; Snowflake or BigQuery considered).

• Experience with dbt or a comparable transformation and testing framework.

• Experience with Git-based workflows, CI, and orchestration (Databricks Workflows, Airflow, or similar).

• Comfort working directly with non-technical business owners to define metrics and resolve data issues.

• Ability to work as a U.S. person and support government contract data requirements.

Preferred

• Experience extracting data from SAP (ECC or S/4HANA) or other ERP systems.

• Experience in manufacturing, supply chain, or industrial data.

• Experience with Unity Catalog, Delta Lake, and data quality tooling.

• Experience supporting ML feature pipelines.

Principal Mineral Corporation is an equal opportunity employer.