Data Engineer: AI-Driven Big Data Pipelines (Remote)

Crossing Hurdles; Remote; GBP 50,000 - 70,000; Full time

Job summary

A leading data solutions provider in the UK is seeking a Data Engineer to design and optimize large-scale data pipelines utilizing Hadoop and Spark. Candidates should have hands-on experience with real-time data streaming using Kafka and be proficient in programming languages like Python and Java. This role requires collaboration with data scientists to support AI initiatives and offers a fully remote work environment. Ideal for those who excel in scalable data architecture and distributed systems.

Qualifications

• Strong hands-on experience with big data technologies including Hadoop and Spark.
• Proven expertise using Kafka for real-time data streaming and integration.
• Solid background in data engineering with experience building and scaling ETL pipelines.
• Practical experience working with major cloud platforms such as AWS, GCP, or Azure.

Responsibilities

• Design, develop, and optimize large scale data pipelines using Hadoop and Spark.
• Build and maintain scalable data architectures that support AI model training.
• Integrate and manage real-time data streams ensuring data reliability.
• Deploy and monitor distributed data processing systems on cloud platforms.

Skills

Hadoop

Spark

Kafka

Python

Java

Scala

Data Engineering

Cloud Platforms

Education

BSc in Computer Science

Data Engineering or related field

Job description

A leading data solutions provider in the UK is seeking a Data Engineer to design and optimize large-scale data pipelines utilizing Hadoop and Spark. Candidates should have hands-on experience with real-time data streaming using Kafka and be proficient in programming languages like Python and Java. This role requires collaboration with data scientists to support AI initiatives and offers a fully remote work environment. Ideal for those who excel in scalable data architecture and distributed systems.