Remote Databricks Data Engineer | Delta Lake & CI/CD

Improving South America; United States; Remote; MXN 500,000 - 800,000; Full time

Job summary

Improving South America is seeking a Data Engineer with hands-on Databricks, Python, and SQL expertise to build and optimize data pipelines and governance initiatives. You will work closely with senior engineers on Delta Lake implementation, Unity Catalog, and CI/CD processes using Azure DevOps and DAB.

The ideal candidate has 5+ years of experience, cloud data platform knowledge, and healthcare or insurance industry exposure.

Qualifications

• 5+ years of hands-on Data Engineer experience.
• Experience with Databricks Asset Bundles (DAB) and Delta Lake.
• Experience with Azure DevOps and CI/CD.
• Healthcare or Insurance industry experience and cloud data platforms knowledge.

Responsibilities

• Develop and maintain data pipelines using Databricks.
• Support design and optimization of Delta Lake tables.
• Work with Unity Catalog to support data governance and access management.
• Participate in CI/CD processes using Azure DevOps and Databricks Asset Bundles (DAB).
• Implement incremental data processing and support schema evolution requirements.
• Collaborate with team members to troubleshoot data quality and performance issues.
• Assist with data modeling and platform optimization.
• Document solutions and follow engineering best practices.
• Ideal candidate has 5+ years of experience.

Skills

Databricks

Python

SQL

Delta Lake

Unity Catalog

CI/CD

Azure DevOps

Cloud-based data platforms

Data governance

Schema evolution

Tools

Databricks Asset Bundles (DAB)

Azure DevOps tooling

Delta Lake tooling

Job description

Improving South America is seeking a Data Engineer with hands-on Databricks, Python, and SQL expertise to build and optimize data pipelines and governance initiatives. You will work closely with senior engineers on Delta Lake implementation, Unity Catalog, and CI/CD processes using Azure DevOps and DAB.

The ideal candidate has 5+ years of experience, cloud data platform knowledge, and healthcare or insurance industry exposure.