Data Engineer: SAS-to-Python on Databricks Pipelines

Improving; Ciudad de México; Remote; MXN 600.000 - 1.000.000; Vollzeit

Zusammenfassung

Improving is a modern digital services company focused on positively changing the IT professional landscape. We seek a data or software engineer to migrate SAS code to Python/SQL on Databricks, validate outputs, and own deployment through DABs and GitLab CI/CD.

You will work with Unity Catalog, Workflows, and Asset Bundles, tuning Spark/Delta Lake performance and ensuring data quality while collaborating with client stakeholders. 4+ years of experience and strong English are required.

Qualifikationen

• 4+ years of professional data or software engineering experience.
• Strong Python and SQL skills.
• Spark and Delta Lake performance tuning experience.
• Data validation and reconciliation experience.
• Hands-on Databricks experience with Unity Catalog, Workflows, and Asset Bundles.
• Infrastructure as code or DAB-based deployment experience.
• GitLab CI/CD experience, including merge request workflows and automated testing
• Strong Git branching and code review discipline
• Clear written and spoken English for client-facing collaboration
• A strong sense of ownership: you scope, deliver, and flag risks without being prompted
• Nice to have: ability to read SAS code, exposure to healthcare data, and Azure experience

Aufgaben

• Migrate SAS code to Python and SQL on Databricks, focusing on pipeline reliability, validation, and delivery.
• Build and run pipelines that process SAS inventories and converted outputs, using existing accelerators.
• Validate converted code for parity against SAS outputs (row counts, checksums, schemas, data types).
• Own deployment through Databricks Asset Bundles (DABs) and GitLab CI/CD.
• Manage Unity Catalog objects, permissions, and environment promotion.
• Troubleshoot job failures and performance issues.
• Communicate directly with client and partner stakeholders.

Kenntnisse

Python

SQL

Spark

Delta Lake

Data validation

Databricks Asset Bundles

Unity Catalog

GitLab CI/CD

Azure

Tools

Databricks

Azure

GitLab CI/CD

Terraform

Jobbeschreibung

Improving is a modern digital services company focused on positively changing the IT professional landscape. We seek a data or software engineer to migrate SAS code to Python/SQL on Databricks, validate outputs, and own deployment through DABs and GitLab CI/CD.

You will work with Unity Catalog, Workflows, and Asset Bundles, tuning Spark/Delta Lake performance and ensuring data quality while collaborating with client stakeholders. 4+ years of experience and strong English are required.