Data Analyst

Crossing Hurdles; Remote; GBP 50,000 - 70,000; Full time

Job summary

A leading data solutions provider in the UK is seeking a Data Engineer to design and optimize large-scale data pipelines utilizing Hadoop and Spark. Candidates should have hands-on experience with real-time data streaming using Kafka and be proficient in programming languages like Python and Java. This role requires collaboration with data scientists to support AI initiatives and offers a fully remote work environment. Ideal for those who excel in scalable data architecture and distributed systems.

Qualifications

• Strong hands-on experience with big data technologies including Hadoop and Spark.
• Proven expertise using Kafka for real-time data streaming and integration.
• Solid background in data engineering with experience building and scaling ETL pipelines.
• Practical experience working with major cloud platforms such as AWS, GCP, or Azure.

Responsibilities

• Design, develop, and optimize large scale data pipelines using Hadoop and Spark.
• Build and maintain scalable data architectures that support AI model training.
• Integrate and manage real-time data streams ensuring data reliability.
• Deploy and monitor distributed data processing systems on cloud platforms.

Skills

Hadoop

Spark

Kafka

Python

Java

Scala

Data Engineering

Cloud Platforms

Education

BSc in Computer Science

Data Engineering or related field

Job description

Responsibilities

• Design, develop, and optimize large scale data pipelines using Hadoop, Spark, and related big data technologies.
• Build and maintain scalable data architectures that support AI model training and analytics workloads.
• Integrate and manage real time data streams using Kafka, ensuring data reliability and quality.
• Deploy, orchestrate, and monitor distributed data processing systems on cloud platforms.
• Collaborate closely with data scientists and machine learning engineers to enable AI and LLM initiatives.
• Document complex data workflows and create clear training materials for technical teams.
• Enforce best practices across data engineering, including performance optimization, security, and scalability.
• Support AI and generative AI use cases through high quality data curation and pipeline design.

Requirements

• BSc in Computer Science, Data Engineering, or a closely related field.
• Strong hands on experience with big data technologies including Hadoop and Spark.
• Proven expertise using Kafka for real time data streaming and integration.
• Solid background in data engineering with experience building and scaling ETL pipelines.
• Practical experience working with major cloud platforms such as AWS, GCP, or Azure.
• Proficiency in programming or scripting languages such as Python, Scala, or Java.
• Excellent written and verbal communication skills with the ability to explain complex technical concepts.
• Strong problem solving and troubleshooting skills in distributed systems.
• Ability to work independently in a fully remote, collaborative environment.