Data Engineer/Architect

E-Solutions; United Kingdom; Hybrid; GBP 80,000 - 100,000; Part time

Job summary

A leading technology company is looking for a Data Engineer in London. The role involves designing Databricks Lakehouse solutions, optimizing data processing with PySpark, and collaborating with teams for self-service analytics. The ideal candidate should have expertise in Databricks, SQL, and data modeling. This is a hybrid position with 3 days on-site per week.

Qualifications

• Experience with Databricks Lakehouse Architecture and Delta Lake.
• Strong understanding of PySpark for optimization.
• Advanced SQL querying skills.

Responsibilities

• Design and implement Databricks Lakehouse solutions.
• Establish data cataloging and lineage.
• Tune Spark jobs for efficiency.

Skills

Databricks

PySpark

SQL

Data Lake/Warehouse

Job description

Overview

Job Title - Data Engineer

Location - London, UK

Type - Contract ( inside IR 35)

Mode - Hybrid,3 days/week onsite

Responsibilities

• Databricks Lakehouse Architecture: Design and implement scalable Databricks Lakehouse solutions with Delta Lake for optimized storage and analytics.
• Data Governance & Cataloging: Establish data cataloging, lineage, and metadata management for improved discoverability.
• Performance Optimization: Tune Spark/PySpark jobs for efficiency in large-scale data processing.
• Data Modelling & Quality: Develop dimensional/data vault models and enforce data quality checks.
• Collaboration: Work with data scientists, analysts, and business teams to enable self-service analytics.
• CI/CD & Automation: Implement Databricks workflows and integrate with Azure/AWS/GCP data ecosystems.

Primary Skills (Must-Have)

• Databricks - Architecture, Delta Lake, Lakehouse, Unity Catalog/Data Catalog
• PySpark (optimization, UDFs, Delta operations)
• SQL (advanced querying, performance tuning)
• Data Lake/Warehouse best practices

Secondary Skills (Nice-to-Have)