Remote Data Engineer: Azure Databricks Python AI Pipelines

BairesDev; Mexico; On-site; PHP 5,552,127 - 9,253,546; Full time

Benefits offered by this job

Remote work

USD or local currency salary

Home office setup

Flexible hours

Paid parental leave and holidays

Global, multicultural team

Job summary

BairesDev is seeking a Data Engineer to design scalable data pipelines, leveraging Azure Databricks and Python. You will integrate AI capabilities into workflows and automate classification tasks, enabling the organization to process complex datasets across distributed cloud environments.

Collaborate with AI and product teams, monitor pipeline performance, ensure security and governance, and deliver high-quality code. This role offers 100% remote work and a competitive USD-based salary.

Qualifications

• 4+ years of experience in Data Engineering, Machine Learning Engineering, or Software Engineering.
• Proven expertise in developing and optimizing data pipelines using Azure Databricks and Python.
• Proficiency in integrating ML/AI solutions and LLM APIs into automated data workflows.
• Strong background in data manipulation, automation, and supporting classification tasks.

Responsibilities

• Design, build, and optimize end-to-end data pipelines and Lakehouse architectures using Azure Databricks.
• Integrate LLM APIs and machine learning models into data processing streams to support automated classification and insights.
• Develop and maintain high-quality Python code and SQL procedures to ensure efficient data transformation and ingestion.
• Collaborate with AI and product teams to implement data-driven solutions that drive significant business value.
• Monitor and troubleshoot pipeline performance, ensuring reliability and scalability across distributed cloud environments.
• Implement data governance and security best practices to maintain the integrity and compliance of the data platform.

Skills

Data engineering

Python

SQL

Cloud architectures

ML integration

Tools

Azure Databricks

LLM APIs

Python

SQL

Job description

BairesDev is seeking a Data Engineer to design scalable data pipelines, leveraging Azure Databricks and Python. You will integrate AI capabilities into workflows and automate classification tasks, enabling the organization to process complex datasets across distributed cloud environments.

Collaborate with AI and product teams, monitor pipeline performance, ensure security and governance, and deliver high-quality code. This role offers 100% remote work and a competitive USD-based salary.