data engineer for AI and ML projects

HireHi; Polska; On-site; PLN 180,000 - 260,000; Full time

Benefits offered by this job

Funding training

Flexible hours

Cafeteria benefits

Referral bonuses PLN 6000

Revenue sharing opportunities

Team manager guidance

Technical mentoring

Team-building budget

Charitable initiatives

Inclusive culture

Job summary

Inetum Polska ищет опытного инженера по данным для разработки и поддержки ETL/ELT пайплайнов на Python в среде Databricks. Вакансия предполагает работу с PySpark, Pandas, Parquet и SQL Server, участие в проектах с AI/ML и обеспечении качества данных.

Кандидат будет сотрудничать с инженерами инфраструктуры и бизнес-стейкхолдерами, нацелен на эффективную интенсификацию миграций и обеспечения соблюдения политик управления данными.

Qualifications

• Опыт работы с Python для задач data engineering, включая PySpark и Pandas.
• Опыт работы с Databricks и Spark экосистемой.
• Знание ETL/ELT, моделирования данных и оркестрации пайплайнов.
• Опыт работы с Microsoft SQL Server, включая прямые подключения к БД.
• Практический опыт обработки Parquet и работы с большими историческими наборами данных.

Responsibilities

• Проектирование, разработка и развёртывание пайплайнов ETL/ELT на Python и Databricks.
• Обеспечение эффективной загрузки исторических наборов Parquet в Databricks.
• Валидация, согласование и QA для перенесённых данных.
• Работа над сопоставлением схем, трансформациями и обогащением метаданных.
• Интеграция управления данными, обеспечения качества и комплаенса в миграционные процессы.
• Оптимизация скорости пайплайнов с использованием Delta Lake и возможностей Databricks.
• Управление ресурсами и планирование передачи больших наборов данных.
• Сотрудничество с инженерами AI, учёными данными и бизнес-стейкхолдерами.
• Ведение документации по всем пайплайнам.

Skills

Python

PySpark

Pandas

Databricks

Spark

ETL/ELT

Data modeling

MS SQL Server

Parquet

Tools

Databricks

Delta Lake

Airflow

Databricks Workflows

Job description

Описание

Inetum Polska is part of the global Inetum Group and drives the digital transformation of businesses and public institutions through consulting, IT infrastructure and application management, software implementation, and custom software development.

Задачи

• Design, develop, and deploy Python-based ETL/ELT pipelines to migrate data from on-premises MS SQL Server into Databricks
• Ensure efficient ingestion of historical Parquet datasets into Databricks
• Implement validation, reconciliation, and quality assurance checks for migrated data
• Handle schema mapping, field transformations, and metadata enrichment to standardize datasets
• Integrate data governance, quality assurance, and compliance into migration activities
• Tune pipelines for speed and efficiency using Databricks capabilities such as Delta Lake when appropriate
• Manage resource usage and scheduling for large dataset transfers
• Work with AI engineers, data scientists, and business stakeholders to define data access patterns for upcoming AI POCs
• Partner with infrastructure teams to ensure secure connections between legacy systems and Databricks
• Maintain technical documentation for all data pipelines
• Follow data governance, compliance, and security best practices throughout the migration process

Требования

• Proven experience with Python for data engineering tasks, including PySpark and Pandas
• Hands-on experience with Databricks and the Spark ecosystem
• Solid understanding of ETL/ELT concepts, data modeling, and pipeline orchestration
• Experience with Microsoft SQL Server, including direct database connections
• Practical experience ingesting Parquet data and managing large historical datasets
• Familiarity with secure data transfer protocols between on-premises environments and cloud platforms
• Strong problem-solving skills and ability to work independently
• Nice to have: Knowledge of Delta Lake and structured streaming in Databricks, experience with AI/ML data preparation workflows, understanding of data governance and compliance requirements related to customer and contract data, familiarity with Databricks Workflows or Airflow, experience setting up Databricks environments from first use

Условия

• Training, certifications, and participation in technology conferences are fully funded
• Flexible working hours
• Cafeteria benefits system
• Referral bonuses of up to PLN6,000
• Additional revenue-sharing opportunities for initiating partnerships with new clients
• Dedicated Team Manager guidance
• Technical mentoring from an assigned technical leader
• Team-building budget for online and on-site team events
• Opportunities to participate in charitable initiatives and local sports programs
• Supportive and inclusive work culture