Big Data Developer

Information Tech Consultants; Greater London; Sur place; GBP 65 000 - 95 000; Plein temps

Résumé du poste

Information Tech Consultants is seeking a Data Engineer to design, develop, and maintain scalable data pipelines and infrastructure. The role focuses on data integration, ETL/ELT, data warehousing, and cloud-based platforms to support analytics and BI.

The candidate will implement data quality checks, governance, and security while collaborating with analysts, scientists, and business stakeholders to optimize data processing across the organization.

Qualifications

• Bachelor's degree in CS/IT/Engineering or related field.
• 2–5 years of data engineering experience.
• Strong SQL and Python proficiency.
• Experience with ETL/ELT tools and data integration.
• Knowledge of relational and NoSQL databases.
• Data warehousing concepts and optimization.
• Familiarity with Spark, Airflow, and cloud platforms (AWS/Azure/GCP).
• Understanding of pipeline monitoring, testing, and performance.
• Strong analytical, problem-solving, and communication skills.

Responsabilités

• Design, develop, and maintain robust data pipelines for batch and real-time processing.
• Build and optimize ETL/ELT workflows from multiple sources.
• Develop complex SQL queries and optimize database performance.
• Work with data warehouses, data lakes, and distributed processing frameworks.
• Implement data quality checks, monitoring, and error handling.
• Collaborate with analysts, data scientists, BI developers, and stakeholders.
• Ensure data security, governance, integrity, and compliance.
• Troubleshoot pipeline failures and improve processing speed.
• Maintain documentation for data models, pipelines, and workflows.

Connaissances

SQL

Python

Data integration

Data warehousing concepts

Analytical skills

Communication

Problem solving

Formation

Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field

Outils

Apache Spark

Apache Airflow

Snowflake

Databricks

AWS

Description du poste

Education: Bachelors in Science (IT/Computer Science/Engineer)

Employment Type: Full-Time

UK based candidates only.

Job Summary

We are looking for a skilled Data Engineer to design, develop, and maintain scalable data pipelines and data infrastructure. The ideal candidate will have experience in data integration, ETL/ELT processes, data warehousing, and cloud-based data platforms to support analytics and business intelligence.

Key Responsibilities

• Design, develop, and maintain robust data pipelines for batch and real-time data processing.
• Build and optimize ETL/ELT workflows to extract, transform, and load data from multiple sources.
• Develop complex SQL queries and optimize database performance.
• Work with data warehouses, data lakes, and distributed data processing frameworks.
• Implement data quality checks, validation, monitoring, and error-handling mechanisms.
• Collaborate with data analysts, data scientists, BI developers, and business stakeholders.
• Ensure data security, governance, integrity, and compliance with organizational standards.
• Troubleshoot pipeline failures and improve data processing performance.
• Maintain technical documentation for data models, pipelines, and workflows.

Required Skills and Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
• 2-5 years of experience in data engineering or a related role.
• Strong proficiency in SQL and Python.
• Experience with ETL/ELT tools and data integration frameworks.
• Knowledge of relational and NoSQL databases.
• Experience with data warehousing concepts, dimensional modeling, and database optimization.
• Familiarity with Apache Spark, Apache Airflow, or similar data processing and orchestration tools.
• Experience with at least one cloud platform: AWS, Microsoft Azure, or Google Cloud Platform.
• Understanding of data pipeline monitoring, testing, and performance optimization.
• Strong analytical, problem-solving, and communication skills.

Preferred Qualifications

• Experience with Snowflake, Databricks, Amazon Redshift, Google BigQuery, or Azure Synapse Analytics.
• Knowledge of Kafka or other streaming technologies.
• Familiarity with Docker, Kubernetes, Git, and CI/CD pipelines.
• Experience with data lakehouse architecture and cloud-native data solutions.
• Knowledge of data governance, metadata management, and data cataloging.