Senior Databricks Engineer / Architect (Remote)

Connect Tech+Talent; Deutschland; Remote; EUR 90.000 - 130.000; Vollzeit

Zusammenfassung

Connect Tech+Talent is seeking a Senior Databricks Engineer to join our enterprise data platform team on a long-term contract. The role focuses on designing, building, and optimizing modern Data Lakehouse solutions using Databricks, Delta Lake, and Spark, with Terraform-driven infrastructure automation.

You will implement Unity Catalog for governance, develop scalable ETL/ELT pipelines in Python and PySpark, and collaborate with data architects and analytics teams to translate requirements into

Qualifikationen

• 10+ years of hands-on Databricks experience in enterprise data solutions.
• Experience with Terraform for Databricks resource provisioning and IaC.

Aufgaben

• Architect, build, and maintain production-grade data pipelines using Spark, Delta Lake, Databricks Workflows, and Unity Catalog.
• Automate Databricks workspace configurations, clusters, security, and cloud resource provisioning with Terraform.
• Design scalable ETL/ELT pipelines using Python, PySpark, and SQL.
• Troubleshoot and optimize Spark jobs for high-volume workloads.
• Implement Unity Catalog for governance across environments.
• Establish CI/CD pipelines for Databricks assets and Terraform configurations.
• Collaborate with data architects and analytics teams to translate business requirements into architecture.

Kenntnisse

Databricks

Terraform

Python

PySpark

SQL

Unity Catalog

Spark

CI/CD

Git

MLOps

Tools

Terraform

GitHub Actions

Azure DevOps

AWS CodePipeline

Jobbeschreibung

Job Title: Senior Databricks Engineer / Architect

Location: 100% Remote

Job Type: Long-term Contract

Job Description.

Position Overview

We are seeking a Senior Databricks Engineer to join our enterprise data platform team supporting one of our clients. In this role, you will be responsible for designing, building, and optimizing modern Data Lakehouse solutions using the Databricks platform and automating infrastructure deployment via Terraform. The ideal candidate possesses deep expertise in high-throughput data pipelines, cloud architecture (Azure/AWS), infrastructure-as-code, and advanced data lake governance.

Key Responsibilities

• Databricks Architecture & Engineering: Architect, build, and maintain production-grade data pipelines using Spark, Delta Lake, Databricks Workflows, and Unity Catalog.
• Infrastructure as Code (IaC): Automate and manage Databricks workspace configurations, clusters, security, and cloud resource provisioning using Terraform.
• Data Pipeline Development: Design scalable ETL/ELT batch and streaming data processing pipelines using Python, PySpark, and SQL.
• Performance Optimization: Troubleshoot and tune Spark jobs, memory allocation, and query execution for high-volume enterprise workloads.
• Data Governance & Security: Implement Unity Catalog for access control, lineage, and data governance across multi-environment setups.
• CI/CD & MLOps Integration: Establish and maintain automated CI/CD deployment pipelines (GitHub Actions, Azure DevOps, or AWS CodePipeline) for Databricks assets and Terraform configurations.
• Collaboration: Work closely with data architects, software engineers, and analytics teams to translate business requirements into efficient technical architecture.

Required Qualifications & Skills

• Databricks Expertise: 10+ years of hands‑on experience in building enterprise data solutions on the Databricks platform (Delta Lake, PySpark, Unity Catalog, Databricks SQL).
• Terraform Expertise: Mandatory experience writing, maintaining, and deploying Terraform modules specifically for Databricks resources and underlying cloud infrastructure.
• Cloud Platforms: Solid hands‑on experience in either Azure (Azure Databricks, ADF, ADLS Gen2) or AWS (S3, IAM, EMR).
• Programming: Mastery of Python, PySpark, and advanced SQL.
• Data Engineering Fundamentals: Strong background in data modeling, lakehouse architecture, real-time streaming, and batch ingestion.
• DevOps & Automation: Experience with Git version control and CI/CD tools for automated code/infrastructure deployments.