Cloud Data Platform Engineer

KOUD; United States; Hybrid; USD 36,000 - 56,000; Full time

Job summary

KOUD is seeking a Cloud Data Platform Engineer to design, implement, and operate scalable Databricks-based data platforms across Azure and/or AWS. You will govern platforms, implement Unity Catalog, Delta Lake, and secure IAM controls while enabling data engineering and ML initiatives.

The role emphasizes platform lifecycle, automation, and observability, with a focus on reliability, security, and collaboration across Data Engineering, Data Science, and AI teams.

Qualifications

• Bachelor's degree or equivalent practical experience in CS/IT/Engineering.
• Strong hands-on experience implementing and administering Databricks environments.
• Experience with Azure and/or AWS cloud platforms.
• Knowledge of Databricks workspace management, cluster configuration, and job scheduling.
• Experience with Unity Catalog and Delta Lake.
• Experience with Infrastructure as Code, particularly Terraform.
• Experience with CI/CD pipelines (Azure DevOps or GitHub Actions).
• Proficiency in Python, PowerShell, and/or Bash for automation.
• Knowledge of cloud networking (VNET/VPC) and security controls including RBAC.

Responsibilities

• Design, implement, and administer Databricks workspaces, clusters, and platform components.
• Build scalable, secure Databricks environments with high availability.
• Define platform standards, governance, and operational best practices.
• Configure Unity Catalog, Delta Lake, and workspace access controls.
• Manage infrastructure using IaC tools, especially Terraform.
• Automate deployments and configuration management across cloud environments.
• Develop observability for infrastructure and data workloads.
• Collaborate with Data Engineering, AI, and Cloud Infra teams.

Skills

Databricks

Terraform

Azure

AWS

CI/CD

Python

PowerShell

Bash

RBAC

Monitoring

Unity Catalog

Delta Lake

VNET/VPC design

Security

English

Education

Bachelor's degree in Computer Science, Information Technology, Engineering, or related fields

Tools

Databricks

Unity Catalog

Delta Lake

Terraform

Azure DevOps

GitHub Actions

Azure Monitor

CloudWatch

Log Analytics

Job description

Koud is looking for a Cloud Data Platform Engineer to work with one of our clients.The position focuses on Databricks platform engineering, cloud infrastructure, automation, security, governance, and operational reliability across Azure and/or AWS environments. The professional will be responsible for designing, implementing, administering, and optimizing scalable and secure data platforms, supporting data engineering, analytics, machine learning, and artificial intelligence initiatives.

Key Responsibilities:

• Design, implement, and administer Databricks workspaces, clusters, and platform components across Azure and/or AWS environments;
• Build and maintain scalable, secure, highly available, and reliable Databricks environments;
• Define platform standards, architectures, governance frameworks, and operational best practices;
• Configure and administer Unity Catalog, Delta Lake, workspace governance, and identity and access controls;
• Implement platform lifecycle management, upgrades, capacity planning, and optimization;
• Provision and manage cloud infrastructure using Infrastructure as Code tools, particularly Terraform;
• Automate platform deployments, configuration management, and operational processes;
• Design secure networks, private connectivity, and integration patterns across cloud environments;
• Implement backup, disaster recovery, high availability, and business continuity solutions;
• Monitor platform health, availability, performance, and capacity;
• Develop observability solutions for infrastructure and data workloads, including monitoring, logging, and auditing;
• Identify and troubleshoot issues related to the platform, networking, infrastructure, and workloads;
• Conduct root cause analyses and implement corrective actions;
• Implement enterprise security controls, RBAC, least-privilege access models, secrets management, encryption, and key management;
• Support compliance requirements and security audits;
• Develop operational runbooks, support processes, and reusable automation standards;
• Work closely with Data Engineering, Data Science, AI Engineering, and Cloud Infrastructure teams;
• Provide technical guidance on platform architecture, operational excellence, and cloud engineering best practices;
• Support the technical development of other professionals and contribute to the continuous evolution of the platform.

Requirements:

• Bachelor's degree in Computer Science, Information Technology, Engineering, or related fields, or equivalent practical experience;
• Strong experience in Cloud Engineering, Platform Engineering, Infrastructure Engineering, or related areas;
• Strong hands-on experience implementing and administering Databricks environments;
• Strong experience with enterprise cloud platforms, particularly Azure and/or AWS;
• Experience designing secure, scalable, and highly available cloud solutions;
• Solid knowledge of Databricks administration, including workspace management, cluster configuration and optimization, workflows, and job scheduling;
• Strong hands-on experience with Unity Catalog and Delta Lake;
• Strong experience with Infrastructure as Code, particularly Terraform;
• Experience with CI/CD pipelines using Azure DevOps and/or GitHub Actions;
• Knowledge of Python, PowerShell, and/or Bash for automation;
• Knowledge of cloud networking, including VNET/VPC design and Private Endpoints;
• Experience with RBAC, identity federation, secrets management, encryption, and key management;
• Experience with monitoring and observability tools such as Azure Monitor, CloudWatch, and Log Analytics;
• Strong problem-solving skills and the ability to work independently in complex enterprise environments;
• Good communication skills and the ability to collaborate with different technical teams and stakeholders;
• Fluent English.

Preferred:

• Databricks Certified Platform Administrator certification;
• Databricks Certified Data Engineer Professional certification;
• Azure Administrator Associate or Azure Solutions Architect certification;
• AWS Solutions Architect certification;
• Experience implementing enterprise AI and Machine Learning platforms;
• Experience in regulated environments, particularly in the Insurance or Financial Services sectors;
• Experience with AI and Machine Learning capabilities within the Databricks ecosystem.

Additional Information:

• Hybrid position, with 3 days per week working on-site in Pinheiros, São Paulo.