Senior Databricks Data Engineer – Scalable Pipelines

Xebia Group; United States; Remote; USD 110,000 - 170,000; Full time

Job summary

Xebia is seeking a Data Engineer to design and implement scalable data pipelines on Databricks, standardize and migrate legacy pipelines, and develop data transformations using dbt and Python. You will onboard new datasets across regions (US, APAC) and collaborate with data architects and analytics teams to enable data-driven solutions.

The role requires strong experience with cloud platforms (Azure/AWS/GCP), data modeling, and CI/CD practices for data pipelines, with ownership mindset and

Qualifications

• Strong experience as a Data Engineer.
• Hands-on experience with Databricks (production-grade pipelines).
• Very good knowledge of Python for data processing.
• Experience with dbt and data transformation layers.
• Solid understanding of data pipeline design, orchestration, and scalability.
• Experience working with legacy systems and pipeline migration.
• Experience with cloud-based data platforms (Azure / AWS / GCP).
• Knowledge of data modeling (medallion architecture, dimensional models).
• Experience working with financial or market data.
• Familiarity with CI/CD practices for data pipelines
• Ability to work independently and take ownership of data solutions.
• Strong communication skills and team collaboration mindset.

Responsibilities

• Design scalable data pipelines on Databricks.
• Standardize existing pipelines (gold standard approach).
• Migrate and refactor legacy data pipelines.
• Develop data transformations using dbt and Python.
• Support onboarding of new datasets (US, APAC regions).
• Collaborate with data architects and analytics teams.
• Contribute to future Data Science and MLOps foundations.
• Work under capacity constraints and optimize performance and resource usage.

Skills

Databricks

Python

dbt

Data pipelines

Cloud platforms

CI/CD for data

Data modeling

Legacy migration

Communication skills

Tools

Azure

AWS

GCP

dbt

Job description

Xebia is seeking a Data Engineer to design and implement scalable data pipelines on Databricks, standardize and migrate legacy pipelines, and develop data transformations using dbt and Python. You will onboard new datasets across regions (US, APAC) and collaborate with data architects and analytics teams to enable data-driven solutions.

The role requires strong experience with cloud platforms (Azure/AWS/GCP), data modeling, and CI/CD practices for data pipelines, with ownership mindset and