Senior Data Engineer - Spark, PySpark & Cloud

Hackajob Ltd; Glasgow; On-site; GBP 61,000 - 101,000; Full time

Job summary

Hackajob Ltd. is seeking an experienced data engineering software engineer to design and deliver secure, scalable data platforms for enterprise-scale workloads. The role covers architecture, coding, and collaboration across teams to implement robust solutions.

You will work with Python/PySpark, Spark-based processing, and modern data platforms while mentoring juniors and driving best practices across the organization.

Qualifications

• Hands-on experience delivering system design, application development, testing, and operational stability at enterprise scale.
• Expertise in Python and/or PySpark.
• Knowledge of software application development with cloud, AI/ML, or data engineering depth.
• Experience in large-scale data processing, microservices, API design, Kafka, Redis, MemCached, observability tools, and orchestration frameworks.
• Advanced working knowledge of relational and NoSQL databases, vector stores, data lake architectures, and data governance.
• Practical cloud-native experience (AWS, Azure, or GCP).
• Ability to present and communicate effectively with senior leaders and executives.
• Commitment to inclusive, collaborative teamwork.
• Good problem-solving and analytical skills.
• Adaptability in a fast-paced environment.
• Focus on delivering secure and scalable solutions.
• Experience with Databricks or Snowflake.
• Deep hands-on experience with Spark/PySpark and other big data processing technologies.
• Expertise in open-source table formats and catalog services such as Apache Iceberg.
• Experience with LLM orchestration frameworks and model serving infrastructure or managed endpoints.
• Familiarity with emerging technologies in data engineering.
• Ability to drive innovation and continuous improvement.
• Passion for mentoring and developing others.

Responsibilities

• Develop secure, high-quality production code for data-intensive applications and platforms.
• Review code and mentor engineers to foster growth and excellence.
• Create durable, reusable software frameworks and patterns for use across teams.
• Drive adoption of advanced technical methods and industry-standard practices.
• Advise cross-functional teams on technological matters within your domain.
• Apply knowledge of tools within the Software Development Life Cycle, including AI-assisted development and automation.
• Enhance automation at scale to improve value and efficiency.
• Lead architectural decisions and engineering practices across multiple teams.
• Collaborate with stakeholders to deliver impactful solutions.
• Champion best practices in software development and data engineering.
• Support a culture of innovation, inclusion, and continuous improvement.

Skills

Python

PySpark

Cloud

AWS

GCP

Azure

Databricks

Snowflake

Spark

Kafka

Redis

NoSQL

API design

Airflow

LLM orchestration

Iceberg

observability

Microservices

Tools

Dynatrace

Splunk

Grafana

Job description

Hackajob Ltd. is seeking an experienced data engineering software engineer to design and deliver secure, scalable data platforms for enterprise-scale workloads. The role covers architecture, coding, and collaboration across teams to implement robust solutions.

You will work with Python/PySpark, Spark-based processing, and modern data platforms while mentoring juniors and driving best practices across the organization.