data engineer in real-time big data

HireHi; Barcelona; Presencial; EUR 45.000 - 75.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

International health insurance

Health and Wellbeing Package (350 EUR)

Birthday day off

Me Time: 1 day off per year

Descripción de la vacante

GiG’s Data department разрабатывает и поддерживает проекты в реальном времени с большими данными, используя информацию для стратегических решений и роста организации.

Вакансия предполагает проектирование ETL, создание и поддержку data-пайплайнов, а также разработку высокопроизводительных сервисов для обработки и анализа больших массивов данных; требуются знания Spark, Kafka, NiFi, ClickHouse и SQL, опыт работы в облаке и на языках Python/Java.

Formación

• Глубокие знания технологий больших данных: Spark, Kafka, NiFi, ClickHouse.
• Умение проектировать схемы и модели данных.
• Опыт извлечения данных большими объёмами и работа с SQL-базами.

Responsabilidades

• Design, create, and maintain ETL processes, data integrations, and data pipelines.
• Ensure seamless handling of large volumes of data.
• Develop high-performance services to process, store, and analyse large datasets using optimised code.
• Design and implement data validation and quality checks tailored to business needs.
• Troubleshoot live production systems and resolve issues.
• Test and implement new technologies relevant to current and future projects.
• Review code, promote best practices, and implement robust solutions across big data projects.

Conocimientos

Big data technologies

Schema design

SQL & relational DBs

Python/Java

Cloud/distributed systems

Herramientas

Apache Spark

Apache Kafka

Apache NiFi

ClickHouse

SQL databases

Descripción del empleo

Описание:

GiG’s Data department develops and supports real-time big data projects, using information to support strategic decision-making and organisational growth.

Задачи:

• Design, create, and maintain ETL processes, data integrations, and data pipelines
• Ensure seamless handling of large volumes of data
• Develop high-performance services to process, store, and analyse large datasets using optimised code
• Design and implement data validation and quality checks tailored to business needs
• Troubleshoot live production systems and resolve issues
• Test and implement new technologies relevant to current and future projects
• Review code, promote best practices, and implement robust solutions across big data projects

Требования:

• Strong expertise in big data technologies such as Spark, Kafka, Nifi, ClickHouse, and big data datastores
• Strong understanding of schema design methodologies and data models
• Expertise in extracting large datasets using SQL and familiarity with SQL databases
• Object-oriented programming skills, ideally in Python and/or Java
• Strong experience with cloud computing and/or distributed systems

Условия:

• Great career development opportunities
• International health insurance
• Health and Wellbeing Package (350 EUR per year)
• Birthday day off
• Me Time: 1 day off per year