Data Architect

Osmii; London; Hybrid; GBP 70,000 - 100,000; Full time

Job summary

Osmii is seeking a visionary Databricks Architect for a 12-month contract to lead the development of a greenfield data platform. This role involves designing a scalable architecture, implementing core data pipelines, and mentoring a team while using advanced technologies such as PySpark and Databricks SQL. The architect will play a crucial role in shaping the data strategy and ensuring robust governance and security within the Databricks environment.

Qualifications

• Hands-on experience designing and implementing solutions on the Databricks Lakehouse Platform.
• Expert-level skills in Python (PySpark) and SQL for data engineering.
• Strong understanding of data warehousing and dimensional modelling concepts.

Responsibilities

• Define and deliver a new data strategy and architecture for the Databricks Lakehouse.
• Lead architectural design and hands-on implementation of core data pipelines.
• Establish scalable data ingestion frameworks and optimize data processing performance.

Skills

Databricks Expertise

Cloud Data Architecture

Programming Proficiency

Data Modelling

ETL/ELT & Data Pipelines

Job description

Direct message the job poster from Osmii

Associate Director at Osmii, part of the HW Group

Databricks Architect (Contract) - Greenfield Data Platform

Location: Hybrid working (London)

Duration: 12-month initial contract

Are you a visionary Databricks Architect with a passion for building cutting-edge data platforms from the ground up? Do you thrive on shaping strategy and driving technical excellence in a greenfield environment?

Our client is embarking on a pivotal journey to establish a brand-new, enterprise-grade data platform using the full power of Databricks. This is a unique opportunity to lead the architectural design and implementation of a truly greenfield data ecosystem that will underpin all future data-driven initiatives, from advanced analytics to AI/ML.

We are looking for a hands-on architect who can translate business needs into robust, scalable, and secure Databricks solutions.

The Role:

As our Databricks Architect, you will be instrumental in defining and delivering our new data strategy and architecture. This is a greenfield project, meaning you'll have the exciting challenge of building the entire Databricks Lakehouse Platform from scratch. You will provide critical technical leadership, guidance, and hands-on expertise to ensure the successful establishment of a scalable, high-performance, and future-proof data environment.

Phase 1: Strategic Vision & Blueprint

• Data Strategy & Roadmap: Collaborate with business stakeholders and leadership to define the overarching data vision, strategy, and a phased roadmap for the Databricks Lakehouse Platform.
• Architectural Design: Lead the end-to-end design of the Databricks Lakehouse architecture (Medallion architecture), including data ingestion patterns, storage layers (Delta Lake), processing frameworks (Spark), and consumption mechanisms.
• Technology Selection: Evaluate and recommend optimal Databricks features and integrations (e.g., Unity Catalog, Photon, Delta Live Tables, MLflow) and complementary cloud services (e.g., Azure Data Factory, Azure Data Lake Storage, Power BI).
• Security & Governance Frameworks: Design robust data governance, security, and access control models within the Databricks ecosystem, ensuring compliance with industry standards and regulations.

Phase 2: Core Platform Build & Development

• Hands-on Implementation: Act as a lead engineer in the initial build-out of core data pipelines, ETL/ELT processes, and data models using PySpark, SQL, and Databricks notebooks.
• Data Ingestion & Integration: Establish scalable data ingestion frameworks from diverse sources (batch and streaming) into the Lakehouse.
• Performance Optimization: Design and implement solutions for optimal data processing performance, cost efficiency, and scalability within Databricks.
• CI/CD & Automation: Develop and implement Continuous Integration/Continuous Delivery (CI/CD) pipelines for automated deployment of Databricks assets and data solutions.
• Team Enablement: Provide mentorship and technical guidance to a growing team of Data Engineers and Analysts, fostering best practices and Databricks expertise.
• Data Quality & Monitoring: Implement comprehensive data quality checks, monitoring, and alerting mechanisms to ensure data integrity and reliability.
• MLOps Integration: Lay the groundwork for seamless integration with Machine Learning Operations (MLOps) capabilities for future AI initiatives.
• Documentation & Knowledge Transfer: Create comprehensive technical documentation and conduct knowledge transfer sessions to ensure long-term sustainability of the platform.

Required Skills & Experience

• Proven Databricks Expertise: Deep, hands-on experience designing and implementing solutions on the Databricks Lakehouse Platform (Delta Lake, Unity Catalog, Spark, Databricks SQL Analytics).
• Cloud Data Architecture: Extensive experience with Azure data services (e.g., Azure Data Factory, Azure Data Lake Storage, Azure Synapse) and architecting cloud-native data platforms.
• Programming Proficiency: Expert-level skills in Python (PySpark) and SQL for data engineering and transformation. Scala is a strong plus.
• Data Modelling: Strong understanding and practical experience with data warehousing, data lake, and dimensional modelling concepts.
• ETL/ELT & Data Pipelines: Proven track record of designing, building, and optimizing complex data pipelines for both batch and real-time processing.

Desirable Skills & Certifications

• Experience with other cloud providers (AWS, GCP).
• Knowledge of streaming technologies (Kafka, Event Hubs).

Seniority level

• Seniority level

  Mid-Senior level

Employment type

• Employment type

  Contract

Job function

• Job function

  Information Technology

• Industries

  Information Services

Referrals increase your chances of interviewing at Osmii by 2x

Get notified about new Data Architect jobs in London Area, United Kingdom.

London, England, United Kingdom 1 week ago

London, England, United Kingdom 1 week ago

Head of Architecture Standards, Patterns & Governance

London Area, United Kingdom £130,000.00-£150,000.00 1 month ago

London, England, United Kingdom 2 weeks ago

London, England, United Kingdom 1 month ago

London, England, United Kingdom 6 days ago

Central Operations Enterprise Data Architect - SVP

London, England, United Kingdom 2 weeks ago

Greater London, England, United Kingdom 1 week ago

London, England, United Kingdom 1 month ago

Enterprise Information Technology Architect

London, England, United Kingdom 5 hours ago

London, England, United Kingdom 3 weeks ago

Senior Cloud and Data Solution Architect

Business Data Architect – Keying and Linking

Data Architect - Data Modelling - Dimensional - Relational - Data Vault

London, England, United Kingdom 2 weeks ago

City Of London, England, United Kingdom £90,000.00-£90,000.00 2 weeks ago

Big Data Architect (Open source contributor)

We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.