Senior Data Engineer / Developer

Luxoft; Remote; GBP 60,000 - 80,000; Full time

Job summary

A leading technology firm in the United Kingdom is looking for an experienced Data Engineer / Developer to design and implement ETL/ELT pipelines for network telemetry data. The ideal candidate has over 5 years of experience with technologies like AWS Glue, S3, and Apache Iceberg. Responsibilities include optimizing data processing, managing data quality, and implementing modern data protection strategies. This role offers competitive compensation.

Qualifications

• 5+ years experience with AWS Glue, S3, Athena, IAM, Lakeformation, Lambda, S3 Tables.
• Experience with Apache Iceberg and lakehouse architectures.
• Proficiency in Java (Python, Scala) for data processing.
• Familiarity with data formats (JSON, Avro, Parquet).
• Experience in time-series database queries.
• Knowledge in data processing (Spark, DuckDB, Flink, AirFlow, Prefect).
• Understanding of Graph Theory.
• Excellent SQL knowledge.

Responsibilities

• Design ETL/ELT pipelines for network telemetry data.
• Interface with on-premises data collection systems.
• Implement Iceberg table structures, also for time-series data.
• Optimize data processing for analytics.
• Manage data quality and lineage.
• Implement pipeline observability.
• Implement modern data protection strategies.

Skills

AWS Glue

S3

Athena

IAM

Lakeformation

Lambda

Apache Iceberg

Java

Python

Scala

JSON

Avro

Parquet

Spark

DuckDB

Flink

AirFlow

Prefect

Graph Theory

SQL

Job description

Project description

• Client's Technology specializes in the development of industrial communication solutions for automation and control applications. Our focus is to provide connectivity solutions that link dissimilar automation devices as seamlessly as if they were all from the same supplier.
• We are looking for a Data Engineer / Developer to work on the next-gen Network Management System.

Responsibilities

• Design ETL/ELT pipelines for network telemetry data
• Interface with on-premises data collection systems
• Implement Iceberg table structures, also for time-series data
• Optimize data processing for analytics
• Manage data quality and lineage
• Implement pipeline observability
• Implement modern data protection strategies

SKILLS

Must have

• 5+ years experience with AWS Glue, S3, Athena, IAM, Lakeformation, Lambda, S3 Tables
• Apache Iceberg and lakehouse architectures
• Java (Python, Scala) for data processing
• Familiarity with data formats (JSON, Avro, Parquet)
• Time-series database queries
• Data processing (Spark, DuckDB, Flink, AirFlow, Prefect)
• Graph Theory
• Excellent SQL Knowledge

Nice to have

• AWS Certifications