Big Data Engineer (Databricks)

Improving South America; United States; Remote; MXN 500,000 - 800,000; Full time

Job summary

Improving South America is seeking a Data Engineer with hands-on Databricks, Python, and SQL expertise to build and optimize data pipelines and governance initiatives. You will work closely with senior engineers on Delta Lake implementation, Unity Catalog, and CI/CD processes using Azure DevOps and DAB.

The ideal candidate has 5+ years of experience, cloud data platform knowledge, and healthcare or insurance industry exposure.

Qualifications

• 5+ years of hands-on Data Engineer experience.
• Experience with Databricks Asset Bundles (DAB) and Delta Lake.
• Experience with Azure DevOps and CI/CD.
• Healthcare or Insurance industry experience and cloud data platforms knowledge.

Responsibilities

• Develop and maintain data pipelines using Databricks.
• Support design and optimization of Delta Lake tables.
• Work with Unity Catalog to support data governance and access management.
• Participate in CI/CD processes using Azure DevOps and Databricks Asset Bundles (DAB).
• Implement incremental data processing and support schema evolution requirements.
• Collaborate with team members to troubleshoot data quality and performance issues.
• Assist with data modeling and platform optimization.
• Document solutions and follow engineering best practices.
• Ideal candidate has 5+ years of experience.

Skills

Databricks

Python

SQL

Delta Lake

Unity Catalog

CI/CD

Azure DevOps

Cloud-based data platforms

Data governance

Schema evolution

Tools

Databricks Asset Bundles (DAB)

Azure DevOps tooling

Delta Lake tooling

Job description

We are looking for a Data Engineer with hands‑on experience in Databricks, Python, and SQL to support the development and optimization of modern data solutions. The ideal candidate will contribute to data pipeline development, Delta Lake implementation, data governance initiatives, and CI/CD processes while working closely with senior engineers and cross‑functional teams.

Key Responsibilities

• Develop and maintain data pipelines using Databricks.
• Support the design and optimization of Delta Lake tables.
• Work with Unity Catalog to support data governance and access management.
• Participate in CI/CD processes using Azure DevOps and Databricks Asset Bundles (DAB).
• Implement incremental data processing and support schema evolution requirements.
• Collaborate with team members to troubleshoot data quality and performance issues.
• Assist with data modeling activities and platform optimization efforts.
• Document solutions and follow engineering best practices.
• Ideally 5 YOE

Required Qualifications

• Experience with Databricks Asset Bundles (DAB).
• Experience using Azure DevOps.
• Knowledge of incremental processing and schema evolution.
• Healthcare or Insurance industry experience.
• Experience with cloud‑based data platforms.

Nice to have

• Azure Databricks certifications.
• Exposure to Medallion Architecture.
• Data quality and monitoring framework

Conditions

• Contrato a largo plazo.
• 100% Remoto.
• Vacaciones y PTOs
• Posibilidad de recibir 2 bonos al año.
• 2 revisiones salariales al año.
• Clases de inglés.
• Equipamiento Apple.
• Plataforma de cursos en linea
• Budget para compra de libros.
• Budget para compra de materiales de trabajo
• mucho mas..