Senior Data Engineer (Databricks on Azure)

Devsu; Colombia; A distancia; COP 90.000.000 - 150.000.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Private health insurance

Remote-friendly culture

Continuous training

AI training resources

Paid time off

Descripción de la vacante

Devsu is seeking a Data Engineer in Colombia to design, build, and operate scalable data pipelines for Tax tech platforms. You will work with Python, Spark, PySpark, and Databricks to deliver reliable, cloud‑native data solutions and improve processing workloads.

Join a remote‑friendly team that values collaboration, testing, CI/CD, and strong data quality practices while tackling production ETL/ELT challenges for US and LatAm clients.

Formación

• 3+ years of data engineering experience.
• Strong Python development experience.
• Strong Apache Spark and PySpark experience.
• Hands-on Databricks experience.
• Strong SQL and data modelling skills.
• Experience building, testing, deploying, and troubleshooting production ETL/ELT pipelines.
• Experience with Git-based source control and CI/CD pipelines.
• Experience consuming REST APIs for data ingestion, including auth, pagination, and error handling.
• Experience writing automated tests for data pipelines.
• Strong data quality, reliability, and best practices.

Responsabilidades

• Design, build, and support scalable data pipelines using Python, Spark, PySpark, and Databricks.
• Modernize and migrate legacy data workloads to cloud-native platforms.
• Build and maintain batch data ingestion pipelines from structured and unstructured sources.
• Integrate data from REST APIs, SharePoint, documents, and cloud platforms.
• Implement data quality, monitoring, observability, and operational controls.
• Optimize data workloads for performance, scalability, reliability, and cost.
• Develop document extraction, classification, metadata enrichment, and automation pipelines.
• Apply software engineering practices including Git, CI/CD, automated testing, and code reviews.
• Build, deploy, troubleshoot, and maintain production ETL/ELT pipelines.
• Collaborate with architects, developers, tax SMEs, and platform teams.
• Analyze existing code for maintainability, security, and reliability.

Conocimientos

Python

Apache Spark

PySpark

Databricks

SQL

Data modelling

Git CI/CD

REST APIs

Automated tests

Data quality

Collaboration

Descripción del empleo

We are looking for a Data Engineer to design, build, modernize, and operate scalable data pipelines supporting Tax technology platforms. This role focuses on developing and enhancing data processing workloads using Python, Apache Spark, PySpark, and Databricks, while ensuring reliable, secure, and maintainable data solutions. The ideal candidate has strong hands‑on engineering skills, experience with production ETL/ELT pipelines, data ingestion, APIs, data quality, and cloud‑based data platforms, along with strong problem‑solving and collaboration skills.

Responsibilities

• Design, build, and support scalable data pipelines using Python, Apache Spark, PySpark, and Databricks.
• Modernize and migrate legacy data processing workloads to secure, cloud‑native platforms.
• Build and maintain batch data ingestion pipelines from structured and unstructured sources.
• Integrate data from REST APIs, SharePoint, document repositories, enterprise applications, and cloud platforms.
• Implement data quality, monitoring, observability, and operational controls.
• Optimize data workloads for performance, scalability, reliability, and cost efficiency.
• Develop document extraction, classification, metadata enrichment, and automation pipelines.
• Apply software engineering practices including Git, CI/CD, automated testing, and code reviews.
• Build, deploy, troubleshoot, and maintain production ETL/ELT pipelines.
• Collaborate with architects, developers, tax subject matter experts, and platform teams.
• Analyze existing codebases and identify opportunities to improve maintainability, security, and reliability.
• Use AI‑assisted development tools to build, review, test, and maintain data pipeline code.

Requirements

Must Have

• 3+ years of experience in data engineering.
• Strong Python development experience.
• Strong Apache Spark and PySpark experience.
• Hands‑on experience with Databricks.
• Strong SQL and data modelling skills.
• Experience building, testing, deploying, and troubleshooting production ETL/ELT pipelines.
• Experience with Git‑based source control and CI/CD pipelines.
• Experience consuming REST APIs for data ingestion, including authentication, pagination, and failure handling.
• Experience writing automated tests for data pipelines.
• Strong understanding of data quality, reliability, and data engineering best practices.
• Strong problem‑solving, communication, and collaboration skills.

Nice to have

• Experience with Azure cloud services and modernising legacy data pipelines.
• Experience with document processing, intelligent document extraction, or unstructured data.
• Familiarity with AI‑assisted development tools such as Codex, GitHub Copilot, or similar.
• Experience with streaming data pipelines or large‑scale data repositories.
• Strong problem‑solving and collaboration skills, with the ability to analyse unfamiliar codebases, identify gaps and trade‑offs, and propose practical improvements.

Benefits

A stable, long‑term contract. Continuous Training. Private Health insurance. Flexible schedule. Work with some of the most talented software engineers in Latin America and the US, doing challenging work and world‑class software for clients in the US and around the world.

At Devsu, we believe in creating an environment where you can thrive both personally and professionally. By joining our team, you’ll enjoy:

• A stable, long‑term contract with opportunities for career growth
• Private health insurance
• A remote‑friendly culture that promotes work‑life balance
• Continuous training, mentorship, and learning programs to keep you at the forefront of the industry
• Free access to AI training resources and state‑of‑the‑art AI tools to elevate your daily work
• A flexible Paid Time Off (PTO) policy as well as paid holiday days
• Challenging, world‑class software projects for clients in the US and LatAm
• Collaboration with some of the most talented software engineers in Latin America and the US, in a diverse work environment

Join Devsu and discover a workplace that values your growth, supports your well‑being, and empowers you to make a global impact.