Senior Data Engineer - Azure Databricks Pipelines (Remote)

Devsu; United States; Remote; USD 140,000 - 180,000; Full time

Benefits offered by this job

Private health insurance

Remote-friendly culture

Continous training and AI resources

Flexible PTO and holidays

Job summary

Devsu is seeking a Senior Data Engineer to design, build, modernise, and operate scalable data pipelines supporting Tax technology platforms. You will migrate Spark/Python workloads to cloud-native Databricks on Azure, building ingestion and document-processing capabilities.

You will own production Databricks pipelines end-to-end, collaborate with architects and tax SMEs, and apply CI/CD and testing practices to improve data quality, reliability, and performance for US and LatAm clients.

Qualifications

• 5+ years of data engineering experience.
• Strong Python development experience.
• Strong Spark and PySpark experience.
• Advanced, hands-on Databricks on Azure experience in production (Delta Lake, Unity Catalogue, Workflows/Jobs, ADLS Gen2, Azure Data Factory).
• Strong SQL and data modelling skills.
• Experience building, testing, deploying, and troubleshooting production ETL/ELT pipelines.
• Experience with Git-based source control and CI/CD pipelines (e.g., Azure DevOps).
• Experience consuming REST APIs for data ingestion, including authentication, pagination, and failure handling.
• Experience writing automated tests for data pipelines.
• Strong understanding of data quality, reliability, and data engineering best practices.
• Strong problem-solving, communication, and collaboration skills, with sound judgement to analyse unfamiliar codebases, identify gaps and trade-offs, challenge requirements and propose practical improvements.

Responsibilities

• Design, build, and support scalable data pipelines using Python, Apache Spark, PySpark, and Databricks on Azure.
• Modernise and migrate legacy data processing workloads to secure, cloud-native platforms.
• Build and maintain batch data ingestion pipelines from structured and unstructured sources.
• Integrate data from REST APIs, SharePoint, document repositories, enterprise applications, and cloud platforms.
• Implement data quality, monitoring, observability, and operational controls.
• Optimise data workloads for performance, scalability, reliability, and cost efficiency.
• Develop document extraction, classification, metadata enrichment, and automation pipelines.
• Apply software engineering practices, including Git, CI/CD, automated testing, and code reviews.
• Build, deploy, troubleshoot, and maintain production ETL/ELT pipelines.
• Collaborate with architects, developers, tax subject matter experts, and platform teams.
• Analyse existing codebases and identify opportunities to improve maintainability, security, and reliability.
• Use AI-assisted development tools to build, review, test, and maintain data pipeline code.

Skills

Python

Apache Spark

PySpark

Databricks on Azure

SQL

CI/CD

REST APIs

Automated tests

Data quality

Problem solving

Collaboration

Tools

Git

Azure DevOps

CI/CD pipelines

Job description

Devsu is seeking a Senior Data Engineer to design, build, modernise, and operate scalable data pipelines supporting Tax technology platforms. You will migrate Spark/Python workloads to cloud-native Databricks on Azure, building ingestion and document-processing capabilities.

You will own production Databricks pipelines end-to-end, collaborate with architects and tax SMEs, and apply CI/CD and testing practices to improve data quality, reliability, and performance for US and LatAm clients.