Senior Data Platform Engineer: Azure+Databricks Automation

Embedded Shishya; United States; On-site; USD 104,000 - 150,000; Full time

Job summary

Accesa, a mature platform initiative for a major financial services technology provider, is seeking a senior engineer to automate Databricks workspace provisioning, write Terraform for Azure, and develop orchestration in Airflow.

You will build data pipelines between on-prem data lakes and the cloud, maintain governance alignment, and collaborate with a German client team, including periodic on-site planning weeks; English is the working language, German is not mandatory.

Qualifications

• Strong Databricks experience including workspace administration and Unity Catalog.
• Solid Terraform / IaC experience in Azure.
• Apache Airflow for orchestration and automation.
• Data engineering fundamentals: data sync/integration pipelines.
• Kubernetes on OpenShift.
• Senior-level autonomy: own and challenge technical decisions.

Responsibilities

• Contribute to building and evolving the platform that automates provisioning of Databricks workspaces.
• Write Terraform code for deploying and managing Azure infrastructure.
• Develop orchestration and provisioning logic in Apache Airflow.
• Build data synchronization pipelines between on-premises data lakes and the cloud platform.
• Maintain Databricks governance in sync with on-premises data governance.
• Collaborate with the client's engineering team, including quarterly on-site PI Planning weeks in Germany.

Skills

Databricks

Terraform

Airflow

Kubernetes

OpenShift

Autonomy

Job description

Accesa, a mature platform initiative for a major financial services technology provider, is seeking a senior engineer to automate Databricks workspace provisioning, write Terraform for Azure, and develop orchestration in Airflow.

You will build data pipelines between on-prem data lakes and the cloud, maintain governance alignment, and collaborate with a German client team, including periodic on-site planning weeks; English is the working language, German is not mandatory.