Senior Data Engineer - Databricks Lakehouse (Remote)

Stermedia; United States; Remote; USD 57,000 - 85,000; Full time

Benefits offered by this job

Flexible working hours

Remote work possible

Self-development budget

Knowledge-sharing workshops

Support for personal branding

Job summary

Stermedia .ai is seeking an experienced Senior Data Engineer to join a global data platform for a leading pharmaceutical organization. You will design and build scalable data pipelines on Databricks, integrating data from laboratories, manufacturing, logistics, and operations.

You will work with international teams, implement Delta Lake bronze-silver-gold architectures, ensure data quality and governance with Unity Catalog, and contribute to production deployments and monitoring.

Qualifications

• Design, develop, and maintain data pipelines in Databricks using Python, PySpark, and SQL.
• Build Delta Lake tables and transformation layers following bronze, silver, and gold architecture.
• Integrate data from multiple source systems into consistent, analytics-ready datasets.
• Implement incremental processing, change data capture, and schema evolution where required.
• Develop and orchestrate production workflows, including dependencies, scheduling, retries, and monitoring.
• Optimize Spark workloads, SQL queries, and compute usage to improve performance and manage costs.
• Implement automated data quality checks, validation, and pipeline tests.
• Support data access management, governance, and lineage using Unity Catalog.
• Maintain reusable code, technical documentation, and Git-based development workflows.
• Collaborate with business stakeholders, data engineers, analysts, and machine learning specialists to translate requirements into reliable data solutions.

Responsibilities

• Work directly on Databricks-based data pipelines for a global analytics environment.

Skills

Databricks

Python

PySpark

SQL

Spark

Delta Lake

Unity Catalog

CI/CD

Git

DBT

Tools

Git

DBT

Terraform

Job description

Stermedia .ai is seeking an experienced Senior Data Engineer to join a global data platform for a leading pharmaceutical organization. You will design and build scalable data pipelines on Databricks, integrating data from laboratories, manufacturing, logistics, and operations.

You will work with international teams, implement Delta Lake bronze-silver-gold architectures, ensure data quality and governance with Unity Catalog, and contribute to production deployments and monitoring.