Data Engineer I: ETL Pipelines for AI & Databricks

Inetum; Warsaw (IN); Hybride; USD 49 000 - 81 000; Plein temps

Avantages offerts par ce poste

Cafeteria benefits

Referral bonuses up to PLN6,000

Training and certifications funded

Hybrid work model

Mentoring and career development

Résumé du poste

Inetum Polska in Poland is seeking a data engineer to design and implement Python-based ETL/ELT pipelines, migrating data from MS SQL Server to Databricks. You will optimize ingestion of large Parquet datasets and ensure data quality, governance, and compliance across migration processes.

The role collaborates with AI/ML teams and infrastructure to secure connections and define data access patterns for upcoming POCs, while leveraging Delta Lake capabilities where appropriate.

Qualifications

• Proven experience in Python for data engineering tasks (PySpark, Pandas, etc.).
• Hands-on experience with Databricks and the Spark ecosystem.
• Solid understanding of ETL/ELT concepts, data modeling, and pipeline orchestration.
• Experience working with Microsoft SQL Server, including direct database connections.
• Practical experience ingesting Parquet data and managing large historical datasets.
• Knowledge of Delta Lake and structured streaming in Databricks is a plus.
• Familiarity with secure data transfer protocols between on-premises environments and cloud platforms.

Responsabilités

• Design, develop, and deploy Python-based ETL/ELT pipelines to migrate data from the on-premises MS SQL Server into the Databricks instance.
• Ensure efficient ingestion of historical parquet datasets into Databricks.
• Implement validation, reconciliation, and quality assurance checks to ensure accuracy and completeness of migrated data.
• Handle schema mapping, field transformations, and metadata enrichment to standardize datasets.
• Ensure data governance, quality assurance, and compliance are integral to all migration activities.

Connaissances

Python

PySpark

Pandas

Databricks

Spark

ETL/ELT

SQL Server

Parquet

Delta Lake

Security protocols

Outils

Airflow

Databricks Workflows

Description du poste

Inetum Polska in Poland is seeking a data engineer to design and implement Python-based ETL/ELT pipelines, migrating data from MS SQL Server to Databricks. You will optimize ingestion of large Parquet datasets and ensure data quality, governance, and compliance across migration processes.

The role collaborates with AI/ML teams and infrastructure to secure connections and define data access patterns for upcoming POCs, while leveraging Delta Lake capabilities where appropriate.