Principal Orchestration Engineer

Aurum Data Solutions Inc.; São Paulo; On-site; BRL 180,000 - 320,000; Full time

Job summary

Aurum Data Solutions Inc. seeks a highly experienced Databricks Platform Engineer to lead the architecture, administration, and evolution of our enterprise Databricks platform in São Paulo, Brazil.

You will drive scalability, security, governance, performance, and cost optimization while supporting Data & AI initiatives. Responsibilities include designing and maintaining Databricks Jobs & Workflows, implementing CI/CD and IaC, and mentoring teams.

Qualifications

• Strong hands-on experience designing and operating Databricks in enterprise environments.
• Deep knowledge of Unity Catalog, governance, and access control.
• Experience with Databricks Jobs & Workflows.

Responsibilities

• Architect and manage the enterprise Databricks environment.
• Define and implement Unity Catalog, security, access control, and data governance standards.
• Design and maintain Databricks Jobs & Workflows for production environments.
• Implement CI/CD, Infrastructure as Code (IaC), and automation best practices.
• Integrate Databricks with Azure, APIs, and enterprise applications.
• Monitor and optimize platform performance, reliability, observability, and costs.
• Establish reusable frameworks, templates, and engineering standards.
• Support AI initiatives, including Databricks Genie, AI Agents, and Mosaic AI capabilities.
• Provide technical leadership, mentorship, and architectural guidance across teams.

Skills

Databricks platform

Python

SQL

Azure

CI/CD

DevOps practices

Data engineering

ETL/ELT

Tools

Terraform

Azure DevOps

Job description

We are seeking a highly experienced Databricks Platform Engineer to lead the architecture, administration, and evolution of our enterprise Databricks platform. This role focuses on platform scalability, security, governance, performance optimization, and supporting the organization's Data & AI initiatives.

Key Responsibilities

• Architect and manage the enterprise Databricks environment.
• Define and implement Unity Catalog, security, access control, and data governance standards.
• Design and maintain Databricks Jobs & Workflows for production environments.
• Implement CI/CD, Infrastructure as Code (IaC), and automation best practices.
• Integrate Databricks with Azure, APIs, and enterprise applications.
• Monitor and optimize platform performance, reliability, observability, and costs.
• Establish reusable frameworks, templates, and engineering standards.
• Support AI initiatives, including Databricks Genie, AI Agents, and Mosaic AI capabilities.
• Provide technical leadership, mentorship, and architectural guidance across teams.

Required Qualifications

• Strong hands-on experience designing and operating Databricks in enterprise environments.
• Deep knowledge of Unity Catalog, governance, and access control.
• Experience with Databricks Jobs & Workflows.
• Advanced Python and SQL skills.
• Strong experience with Microsoft Azure.
• Expertise in CI/CD, infrastructure automation, and DevOps practices.
• Solid background in data engineering, ETL/ELT, and production data pipelines.
• Experience troubleshooting platform, performance, and reliability issues.

Preferred Qualifications

• Experience with Databricks AI/BI Genie.
• Familiarity with Mosaic AI, LLMs, RAG, Semantic Search, and AI Agents.
• Understanding of AI governance, monitoring, evaluation, and security best practices.

Experience

• Large experience in Software Engineering, Data Engineering, Platform Engineering, Cloud Engineering, or related fields.
• Experience in a Senior, Lead, Staff, Principal, or equivalent technical leadership role.
• Proven experience building and operating enterprise-scale data platforms.
• Strong ability to define technical standards and drive cross-functional collaboration.