Senior Data Engineer (Databricks on Azure)

Devsu

United States

Remote

USD 140,000 - 190,000

Full time

11 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Stable long-term contract
Private health insurance
Remote-friendly culture
Continuous training & mentorship
Free AI training resources
Flexible PTO & holidays

Job summary

Devsu is seeking a Senior Data Engineer with advanced Databricks on Azure experience to design, build, modernise, and operate scalable data pipelines for Tax technology platforms.

You will migrate Spark and Python workloads to cloud-native platforms, build ingestion pipelines, and enforce data quality, observability, and security. Collaborate with architects, SMEs, and platform teams to improve maintainability and performance.

Qualifications

  • 5+ years of experience in data engineering or equivalent.
  • Strong Python development experience with Spark-based workloads.
  • Advanced, hands-on Databricks on Azure experience in production.
  • Strong SQL and data modelling skills.
  • Experience building, testing, deploying, and troubleshooting ETL/ELT pipelines.
  • Experience with Git-based source control and CI/CD pipelines (e.g., Azure DevOps).
  • Experience consuming REST APIs for data ingestion with error handling.

Responsibilities

  • Design, build, and support scalable data pipelines using Python, Spark, PySpark, and Databricks on Azure.
  • Modernise and migrate legacy data processing workloads to secure, cloud-native platforms.
  • Build and maintain batch data ingestion pipelines from structured and unstructured sources.
  • Integrate data from REST APIs, SharePoint, document repositories, and cloud platforms.
  • Implement data quality, monitoring, observability, and operational controls.
  • Optimise data workloads for performance, scalability, reliability, and cost efficiency.
  • Develop document extraction, classification, and automation pipelines.
  • Apply software engineering practices including Git, CI/CD, automated testing, and code reviews.
  • Build, deploy, troubleshoot, and maintain production ETL/ELT pipelines.
  • Collaborate with architects, developers, tax SMEs, and platform teams.
  • Analyse codebases to identify gaps and practical improvements.

Skills

Python development
Apache Spark / PySpark
Databricks on Azure
SQL / data modeling
ETL/ELT pipelines
REST APIs data ingestion
Automated tests for data pipelines
CI/CD (Git, Azure DevOps)
Data quality & observability
Problem solving & collaboration
AI-assisted development tools

Tools

Azure DevOps
Git
Delta Lake
Unity Catalog
ADLS Gen2
Azure Data Factory
CI/CD tooling

Job description

We are looking for a Senior Data Engineer with advanced, hands‑on Databricks on Azure experience to design, build, modernise and operate scalable data pipelines supporting Tax technology platforms. You will migrate and enhance existing Spark and Python workloads on Azure Databricks and build scalable ingestion and document‑processing capabilities. We need someone who has owned production Databricks on Azure pipelines end to end, understands the wider system, surfaces assumptions and trade‑offs, and proposes better approaches.

Responsibilities
  • Design, build, and support scalable data pipelines using Python, Apache Spark, PySpark, and Databricks on Azure.
  • Modernise and migrate legacy data processing workloads to secure, cloud‑native platforms.
  • Build and maintain batch data ingestion pipelines from structured and unstructured sources.
  • Integrate data from REST APIs, SharePoint, document repositories, enterprise applications, and cloud platforms.
  • Implement data quality, monitoring, observability, and operational controls.
  • Optimise data workloads for performance, scalability, reliability, and cost efficiency.
  • Develop document extraction, classification, metadata enrichment, and automation pipelines.
  • Apply software engineering practices, including Git, CI/CD, automated testing, and code reviews.
  • Build, deploy, troubleshoot, and maintain production ETL/ELT pipelines.
  • Collaborate with architects, developers, tax subject matter experts, and platform teams.
  • Analyse existing codebases and identify opportunities to improve maintainability, security, and reliability.
  • Use AI‑assisted development tools to build, review, test, and maintain data pipeline code.
Must Have
  • 5+ years of experience in data engineering.
  • Strong Python development experience.
  • Strong Apache Spark and PySpark experience.
  • Advanced, hands‑on Databricks on Azure experience in production (Delta Lake, Unity Catalogue, Workflows/Jobs, ADLS Gen2, Azure Data Factory).
  • Strong SQL and data modelling skills.
  • Experience building, testing, deploying, and troubleshooting production ETL/ELT pipelines.
  • Experience with Git‑based source control and CI/CD pipelines (e.g., Azure DevOps).
  • Experience consuming REST APIs for data ingestion, including authentication, pagination, and failure handling.
  • Experience writing automated tests for data pipelines.
  • Strong understanding of data quality, reliability, and data engineering best practices.
  • Strong problem‑solving, communication, and collaboration skills, with sound judgement to analyse unfamiliar codebases, identify gaps and trade‑offs, challenge requirements and propose practical improvements.
Nice to have
  • Experience modernising legacy data pipelines.
  • Experience with document processing, intelligent document extraction, or unstructured data.
  • Familiarity with AI‑assisted development tools such as Codex, GitHub Copilot, or similar.
  • Experience with streaming data pipelines or large‑scale data repositories.

A stable, long‑term contract. Continuous Training. Private Health insurance. Flexible schedule. Work with some of the most talented software engineers in Latin America and the US, doing challenging work and world‑class software for clients in the US and around the world.

At Devsu, we believe in creating an environment where you can thrive both personally and professionally. By joining our team, you’ll enjoy:

  • A stable, long‑term contract with opportunities for career growth
  • Private health insurance
  • A remote‑friendly culture that promotes work‑life balance
  • Continuous training, mentorship, and learning programs to keep you at the forefront of the industry
  • Free access to AI training resources and state‑of‑the‑art AI tools to elevate your daily work
  • A flexible Paid Time Off (PTO) policy as well as paid holiday days
  • Challenging, world‑class software projects for clients in the US and LatAm
  • Collaboration with some of the most talented software engineers in Latin America and the US, in a diverse work environment

Join Devsu and discover a workplace that values your growth, supports your well‑being, and empowers you to make a global impact.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer (AWS, Python & SQL)
Senior Data Engineer (AWS, Python & SQL)

Devsu • United States

Remote
USD 120,000 - 180,000
Private health insurance
Flexible schedule
Remote-friendly culture
+1
Semi Senior / Senior Data Engineer (AWS, Python & SQL)
Semi Senior / Senior Data Engineer (AWS, Python & SQL)

Devsu • United States

Remote
USD 140,000 - 200,000
Private health insurance
Remote-friendly culture
Continuous training and learning
Data Engineer (Azure Databricks + Python) - Remote Work | REF#295244
Data Engineer (Azure Databricks + Python) - Remote Work | REF#295244

BairesDev • United States

Remote
USD 120,000 - 180,000
100% remote
Competitive USD pay
Home office setup
+5
Data Management and BI Senior Executive, GOC
Data Management and BI Senior Executive, GOC

wppmedia • United States

Remote
USD 120,000 - 180,000
Senior Databricks Engineer on Azure — Remote
Senior Databricks Engineer on Azure — Remote

Devsu • United States

Remote
USD 140,000 - 190,000
Stable long-term contract
Private health insurance
Remote-friendly culture
+3
Senior Data Engineer (Databricks) - Remote Work | REF#302633
Senior Data Engineer (Databricks) - Remote Work | REF#302633

BairesDev • United States

Remote
MXN 1,677,000 - 2,608,000
100% remote
Competitive USD salary
Home office setup
Data Engineer, Azure - Remote, Latin America
Data Engineer, Azure - Remote, Latin America

Remote • San Francisco (CA)

Remote
USD 120,000 - 160,000
Competitive salary and bonuses
Generous paid-time-off
Flexible working hours
+3
Azure Data Engineer - Remote, Latin America
Azure Data Engineer - Remote, Latin America

Remote • San Francisco (CA)

Remote
USD 135,000 - 185,000
Bonuses
PTO (Paid Time Off)
Flexible hours
+2
Senior Data Engineer
Senior Data Engineer

Writing • United States

Remote
USD 33,000 - 50,000
Data Engineer (Azure) - Remote Work | REF#295527
Data Engineer (Azure) - Remote Work | REF#295527

BairesDev • United States

Remote
USD 120,000 - 180,000
Remote work
Competitive USD salary
Hardware provided
+4