Senior Data Engineer - Remote (Databricks & PySpark)

Optiveum; United States; Remote; USD 53,000 - 89,000; Full time

Job summary

Optiveum is seeking a Senior Data Engineer to design and operate the Sustainability Data Foundation built on Databricks, Delta Lake, and PySpark. You will develop scalable data products and pipelines supporting regulatory reporting and advanced analytics.

The role emphasizes data quality, governance, CI/CD, IaC, and close collaboration with sustainability experts and analysts. Fluent English and proactive problem-solving are essential.

Qualifications

• 5–7 years of experience designing, developing, and operating data platforms and pipelines.
• Extensive hands-on experience with Databricks (including Azure Databricks and Databricks Workflows) in production environments.
• Expert-level proficiency in PySpark, Python, Spark SQL, Data Modelling, and Data Pipeline Design.
• Experience implementing CI/CD pipelines for data engineering workloads, Git-based development, and version control.
• Solid understanding of data lineage, metadata management, governance, auditing, and validation frameworks.
• Fluent English (both written and spoken).
• Proactive, self-driven mindset with strong troubleshooting and root-cause analysis skills.

Responsibilities

• Data Engineering & Product Development: Design and implement scalable data products using Databricks, Delta Lake, and PySpark.
• Build and maintain data pipelines processing complex datasets from ERP, procurement, sustainability, and external systems.
• Performance Optimization: Optimize workloads for performance, scalability, and cost efficiency, building reusable engineering patterns.
• Data Quality & Governance: Implement automated data quality controls throughout the data lifecycle.
• Platform Engineering & DevOps: Implement CI/CD pipelines, automated deployments, testing frameworks, and IaC.
• Stakeholder Collaboration: Partner with sustainability experts, business analysts, and reporting teams to support requirements gathering and production releases.

Skills

Databricks

PySpark

Python

Spark SQL

Data Modeling

Data Pipeline Design

CI/CD

Git

Data Governance

Data Lineage

Metadata Management

English

Tools

Azure Databricks

Databricks Workflows

Delta Lake

Infrastructure as Code

Job description

Optiveum is seeking a Senior Data Engineer to design and operate the Sustainability Data Foundation built on Databricks, Delta Lake, and PySpark. You will develop scalable data products and pipelines supporting regulatory reporting and advanced analytics.

The role emphasizes data quality, governance, CI/CD, IaC, and close collaboration with sustainability experts and analysts. Fluent English and proactive problem-solving are essential.