SC Cleared Databricks Engineer

iO Associates; England; Remote; GBP 80,000 - 100,000; Full time

Job summary

A leading GovTech Consultancy is seeking SC Cleared Databricks Engineers for a fully remote, 6-month contract. You will design and optimize data pipelines using Databricks and work with complex datasets to provide strategic insights. Proven experience in Databricks and Apache Spark is required. This role offers competitive pay at £550 per day inside IR35. Apply soon, as CVs will be shortlisted quickly.

Qualifications

• Active SC Clearance (used within the last 12 months).
• Proven experience with Databricks (including notebooks, clusters, and job orchestration).
• Comfortable working independently in a fully remote environment.

Responsibilities

• Design and optimize data pipelines and analytics solutions using Databricks.
• Work with large and complex datasets to provide data-driven insights.

Skills

Databricks

Apache Spark

PySpark

ETL pipelines

SQL

Agile delivery (Scrum/Kanban)

Tools

Azure Databricks

Delta Lake

Job description

Databricks Engineer - SC Cleared
Fully Remote
6-month initial contract
Start Date: 17th November 2025
£550 per day Inside IR35

We are seeking two SC Cleared Databricks Analysts to support a high-profile data engineering and analytics programme with one of our GovTech Consultancy clients.

The successful candidates will play a key role in designing, developing, and optimising data pipelines and analytics solutions using Databricks within a secure environment. You will work with large and complex datasets from UK and potentially international sources, enabling advanced analytics and data-driven insights to inform strategic decisions.
This role is highly technical and data-focused, requiring strong experience in Databricks, Spark, and related data engineering tools. While stakeholder engagement is minimal, you will need to align technical solutions with business objectives and ensure data integrity and compliance throughout.

Key Requirements

• Active SC Clearance (used within the last 12 months)
• Proven experience with Databricks (including notebooks, clusters, and job orchestration)
• Strong knowledge of Apache Spark, PySpark, and distributed data processing
• Experience building and optimising ETL pipelines and data workflows
• Familiarity with Delta Lake, SQL, and data modelling best practices
• Ability to work with large, complex datasets from multiple sources
• Comfortable working independently in a fully remote environment
• Strong understanding of Agile delivery (Scrum/Kanban)
• Excellent attention to detail and structured approach to analysis
• Ability to interpret business needs and translate them into scalable data solutions

Desirable

• Experience with Azure Databricks and related cloud services (Azure Data Lake, Data Factory)
• Knowledge of data governance, security, and compliance in government environments
• Familiarity with CI/CD pipelines for data engineering

This is an excellent opportunity for experienced Databricks Engineers with SC Clearance to apply their expertise in a meaningful, data-driven role within a secure government programme.

Please apply promptly as CVs will be shortlisted and sent to the client later this week.