Senior Databricks Engineer / Architect

Connect Tech+Talent; Deutschland; Remote; EUR 90.000 - 120.000; Vollzeit

Zusammenfassung

Connect Tech+Talent is seeking a Senior Databricks Engineer/Architect to join on a 100% remote, long-term contract. You will design, build, and optimize modern Data Lakehouse solutions on Databricks, automate infrastructure via Terraform, and ensure governance across multi-environments.

The role requires deep expertise in large-scale data pipelines, cloud architecture (Azure/AWS), and advanced data governance practices, with collaboration across data, software, and analytics teams.

Qualifikationen

• 10+ years building enterprise data solutions on Databricks.
• Mandatory experience writing, maintaining, and deploying Terraform modules for Databricks resources.
• Solid hands-on experience with Azure Databricks or AWS, PySpark, Python, and SQL.

Aufgaben

• Architect, build, and maintain production-grade data pipelines using Spark, Delta Lake, Databricks Workflows, and Unity Catalog.
• Automate Databricks workspace configurations, clusters, security, and cloud resources with Terraform.
• Design scalable ETL/ELT batch and streaming pipelines using Python, PySpark, and SQL.
• Troubleshoot and tune Spark jobs, memory allocation, and query execution for high-volume workloads.
• Implement Unity Catalog for access control, lineage, and data governance across environments.
• Establish and maintain CI/CD pipelines (GitHub Actions, Azure DevOps, AWS CodePipeline) for Databricks assets and Terraform configurations.
• Collaborate with data architects, software engineers, and analytics teams to translate business requirements into technical architecture.

Kenntnisse

Databricks

Terraform

Python

PySpark

SQL

Cloud platforms

CI/CD

Data modeling

Streaming

Git

Tools

Databricks

Unity Catalog

Delta Lake

Spark

GitHub Actions

Azure DevOps

AWS CodePipeline

Terraform modules

Jobbeschreibung

Job Description

Job Title: Senior Databricks Engineer / Architect

Location: 100% Remote

Job Type: Long-term Contract

Position Overview

We are seeking a Senior Databricks Engineer to join our enterprise data platform team supporting one of our clients. In this role, you will be responsible for designing, building, and optimizing modern Data Lakehouse solutions using the Databricks platform and automating infrastructure deployment via Terraform. The ideal candidate possesses deep expertise in high-throughput data pipelines, cloud architecture (Azure/AWS), infrastructure-as-code, and advanced data lake governance.

Key Responsibilities

• Databricks Architecture & Engineering: Architect, build, and maintain production-grade data pipelines using Spark, Delta Lake, Databricks Workflows, and Unity Catalog.
• Infrastructure as Code (IaC): Automate and manage Databricks workspace configurations, clusters, security, and cloud resource provisioning using Terraform.
• Data Pipeline Development: Design scalable ETL/ELT batch and streaming data processing pipelines using Python, PySpark, and SQL.
• Performance Optimization: Troubleshoot and tune Spark jobs, memory allocation, and query execution for high-volume enterprise workloads.
• Data Governance & Security: Implement Unity Catalog for access control, lineage, and data governance across multi-environment setups.
• CI/CD & MLOps Integration: Establish and maintain automated CI/CD deployment pipelines (GitHub Actions, Azure DevOps, or AWS CodePipeline) for Databricks assets and Terraform configurations.
• Collaboration: Work closely with data architects, software engineers, and analytics teams to translate business requirements into efficient technical architecture.

Required Qualifications & Skills

• Databricks Expertise: 10+ years of hands-on experience in building enterprise data solutions on the Databricks platform (Delta Lake, PySpark, Unity Catalog, Databricks SQL).
• Terraform Expertise: Mandatory experience writing, maintaining, and deploying Terraform modules specifically for Databricks resources and underlying cloud infrastructure.
• Cloud Platforms: Solid hands-on experience in either Azure (Azure Databricks, ADF, ADLS Gen2) or AWS (S3, IAM, EMR).
• Programming: Mastery of Python, PySpark, and advanced SQL.
• Data Engineering Fundamentals: Strong background in data modeling, lakehouse architecture, real-time streaming, and batch ingestion.
• DevOps & Automation: Experience with Git version control and CI/CD tools for automated code/infrastructure deployments.