Databricks Administrator/Architect

EUROPEAN DYNAMICS Group; United States; Remote; USD 150,000 - 210,000; Full time

Job summary

EUROPEAN DYNAMICS seeks a Senior Databricks Administrator/Architect fluent in English for remote engagement. You will join the Development team and collaborate with a major client's IT team to design and implement scalable data platforms on Azure Databricks, Delta Lake, and Unity Catalog.

The role requires deep expertise in data engineering, security, governance, and cloud infrastructure, with strong Python/PySpark skills. Remote work with international collaboration is expected.

Qualifications

• Bachelor's degree in IT or related field and 9+ years of IT experience.
• 8+ years designing enterprise data platforms with Azure Databricks.
• 5+ years hands-on Azure Databricks in enterprise environments.
• Experience with Lakehouse, Delta Lake, Unity Catalog and governance.
• CI/CD with Azure DevOps, Terraform/Bicep, deployment automation.
• Strong security architecture knowledge: RBAC, PIM, encryption, Key Vault.
• Proficiency in SQL, Python and PySpark; familiarity with BI/analytics tools.
• English fluent; experience in international organizations is a plus.

Responsibilities

• Architect and implement scalable enterprise data platform solutions using Azure Databricks and related Azure services.
• Design data engineering frameworks for batch, streaming, ELT/ETL, and real‑time analytics.
• Administer Databricks workspaces, Unity Catalog, secrets, and governance policies.
• Define security with Entra ID, RBAC, PIM, encryption, private networking, and Key Vault.
• Establish data governance: lineage, quality, metadata, catalog, access policies, audits.
• Deploy CI/CD pipelines with Azure DevOps and DAB; use Terraform/Bicep for IaC.
• Tune Spark workloads, optimize clusters, caching, partitioning, and Photon usage.
• Create monitoring dashboards with Azure Monitor and Log Analytics; incident support.
• Develop architecture patterns, standards, naming conventions, and documentation.
• Assist migration from legacy warehouses to Databricks Lakehouse.
• Provide technical support for production incidents related to Databricks and data pipelines.

Skills

Azure Databricks

Delta Lake

Unity Catalog

Azure Data Factory

Python / PySpark

Azure AD / RBAC

Terraform / IaC

CI/CD pipelines

Azure Monitor / Log Analytics

Education

University degree in IT or related field

Tools

Terraform

Databricks

Azure Key Vault

Azure Monitor

Job description

Описание

We currently have a vacancy for a Senior Databricks Administrator/Architect fluent in English, to offer his/her services as an expert remotely. In the context of the first assignment, the successful candidate will be integrated in the Development team of the company that will closely cooperate with a major client’s IT team.

Задачи

Collaborate with Data Platforms and Integration team in architecting and implementing scalable enterprise data platform solutions utilizing Azure Databricks, Delta Lake, Unity Catalog, Azure Data Lake Storage Gen2, Azure Data Factory, Azure Event Hub, Azure SQL, and related Azure services; Design and develop enterprise data engineering frameworks supporting batch, streaming, ELT/ETL, and real-time analytics workloads; Administer and maintain Azure Databricks workspaces including cluster configuration, job scheduling, workspace administration, Unity Catalog implementation, secret management, compute policies, cluster policies, and workspace governance; Define and implement enterprise security architecture including identity integration with Microsoft Entra ID (Azure AD), Role-Based Access Control (RBAC), Privileged Identity Management (PIM), data encryption, private networking, managed identities, Key Vault integration, and secure access patterns; Establish enterprise data governance standards utilizing Unity Catalog including data lineage, data quality, metadata management, catalog organization, access policies, and audit capabilities; Deploy and Configure CI/CD pipelines utilizing Azure DevOps and Databricks Asset Bundles (DAB), Infrastructure as Code (Terraform/Bicep where applicable), deployment automation, release management, and environment promotion strategies; Optimize Databricks workloads through performance tuning, Spark optimization, cluster sizing, workload isolation, caching strategies, partitioning, Delta optimization, Photon, and cost optimization best practices; Design enterprise monitoring, logging, operational dashboards, alerting, and platform health monitoring utilizing Azure Monitor, Log Analytics, Application Insights, and Databricks system tables; Develop reusable architecture patterns, reference implementations, development standards, naming conventions, coding standards, operational procedures, and platform documentation; Support migration of legacy enterprise data warehouses, ETL solutions, and reporting platforms to Azure Databricks and modern Lakehouse architecture; Provide technical support and troubleshooting for production incidents related to Azure Databricks, Spark workloads, data pipelines, and platform infrastructure.

Требования

• University degree in combined with relevant IT professional experience of 9 years
• At least 8 years of experience in designing and implementing enterprise data platforms
• At least 5 years of hands-on experience hands-on Azure Databricks experience in enterprise environments
• Experience with architecting modern Lakehouse solutions using Azure Databricks and Delta Lake
• Experience in administering Azure Databricks workspaces, clusters, Unity Catalog, compute policies, job orchestration, and workspace security
• Experience in developing data engineering solutions using Apache Spark (PySpark and/or Spark SQL)
• Experience in implementing enterprise data governance utilizing Unity Catalog, metadata management, data lineage, and fine-grained access controls
• Experience implementing Infrastructure as Code using Terraform and/or Bicep
• Strong experience implementing CI/CD pipelines utilizing Azure DevOps, automated testing, release management, and deployment automation
• Experience designing and working with enterprise security architectures including RBAC, Managed Identities, Private Endpoints, Key Vault integration, encryption, and network security
• Experience implementing enterprise data quality frameworks, monitoring, observability, and operational dashboards
• Strong understanding of data modelling techniques including dimensional modelling, Data Vault, medallion architecture, and enterprise data warehouse concepts
• Strong knowledge of Azure Data Lake Storage Gen2, Azure Data Factory, Azure Event Hub, Azure SQL Database, Azure Key Vault, Azure Monitor, Microsoft Entra ID (Azure AD), and Azure networking
• Strong SQL programming skills and proficiency in Python and PySpark development
• Knowledge of AI/ML workloads, Databricks ML, Feature Store, Model Serving, and Generative AI architectures is considered an asset
• Experience with Power BI, Microsoft Fabric, Synapse Analytics, or other enterprise analytics platforms is highly desirable
• Experience working within large international organizations, NGOs, or United Nations agencies is desirable
• The following certifications are desirable: Databricks Certified Platform, Architecture/Microsoft Azure Solution Architecture Expert/Databricks, Certified Data Engineer Associate/Professional
• Excellent command of the English language.

Условия

Если вы ищете карьеру в увлекательной и динамичной компании, где вы будете предлагать свои услуги в составе команды крупного европейского института, работающего в международной, многоязычной и многокультурной среде, где вы можете ожидать реальных шансов сделать разницу, Мы предлагаем конкурентную оплату (на условиях контракта или с полной системой льгот), основанную на квалификации и опыте. Все заявки будут рассматриваться конфиденциально. EUROPEAN DYNAMICS (ED) (www.eurodyn.com) is a leading Software, Information and Communication Technologies company, operating internationally (Athens, Brussels, Luxembourg, Copenhagen, Stockholm, London, Nicosia, Hong-Kong, Valetta, etc). The company employs over 1000 engineers, IT experts and consultants (around 3% PhD, 36% MSc and 53% BSc or equivalent). We design and develop software applications using state-of-the-art technology. The group generates annual revenues in the range of EURO 40 million, with an EBITDA in the range of 20%. The value of our contract portfolio exceeds EURO 250 million. EUROPEAN DYNAMICS is a renowned supplier of IT services to government institutions, multinational corporations, public administrations and multinational companies, research and academic institutes. As part of our dedication to the diversity of our workforce, we are committed to Equal Employment Opportunity without regard for race, color, national origin, ethnicity, gender, disability, sexual orientation, gender identity, or religion. EUROPEAN DYNAMICS (ED) adheres to the General Data Protection Regulation principles by applying its Privacy Policy as published in www.eurodyn.com/privacy. By submitting an application to this position and by sharing your personal data with ED, you acknowledge and accept its Policy and authorise ED to process your personal data for the purposes of the company's recruitment opportunities, in line to the Policy.