Data engineer in insurance

Astek; Warszawa; On-site; PLN 166,000 - 221,000; Full time

Benefits offered by this job

Multisport

Private medical care

Life insurance

Job summary

Astek Group ищет Data Engineer с опытом работы на продакшн-проектах в области данных. Вы будете проектировать пайплайны в Azure Data Factory и PySpark в Microsoft Fabric, интегрировать источники, валидировать данные и обеспечивать качество и безопасность данных.

Требуется владение Python, PySpark, SQL и Git, опыт работы с облачными платформами и медальонной архитектурой. Предусмотрены обучение, сертификации и возможности карьерного роста.

Qualifications

• 3+ года опыта в роли Data Engineer на продакшн-проектах
• Python, PySpark и SQL на уровне свободной работы с большими данными
• Опыт работы в облачных платформах: Microsoft Fabric, Azure Synapse или Databricks
• Понимание архитектуры данных (медальонная архитектура) и форматов Delta/Parquet
• Опыт интеграции источников данных, включая Excel
• Опыт использования Git в команде
• Готовность подписать NDA и пройти требуемые проверки безопасности
• Будет плюсом сертификации DP-600/DP-700; опыт CQF; CI/CD для Data Platform; R; Agile/Scrum

Responsibilities

• Проектирование и реализация пайплайнов в Azure Data Factory и PySpark-ноутбуках в Microsoft Fabric
• Подключение источников данных и их валидация
• Трансформацию и подготовку пакетных данных
• Реализация правил качества данных и отчетности об ошибках
• Версионирование наборов данных и запись метаданных запуска
• Поддержка качества кода в Git-воркфлоу и участие в CI/CD
• Сотрудничество с Risk Team для определения правил трансформаций
• Соблюдение требований безопасности и регуляторики, интеграция стандартов безопасности в Azure
• Оптимизация процессов по производительности и затратам

Skills

Python

PySpark

SQL

Git

Tools

Microsoft Fabric

Azure Synapse

Databricks

Delta/Parquet

Job description

Описание: We are currently looking for a candidate for our client in the insurance industry for the position of Data Engineer with Microsoft Fabric (M/F).

Задачи

Designing and implementing pipelines in Azure Data Factory and PySpark notebooks in Microsoft Fabric. Onboarding file-based and database sources, including their technical validation and format standardization. Transforming and preparing batch data. Implementing data quality validation rules and error reporting mechanisms. Versioning data sets and recording run metadata, including parameters, sources, and versions. Maintaining code quality in a Git-based workflow: working with the repository, deployments through CI/CD, and participating in code reviews. Working closely with the Risk Team to define data transformation rules. Ensuring compliance with data privacy and security regulations. Integrating company security standards with the Azure environment. Optimizing processes for performance and cost.

Требования

You have at least 3 years of experience as a Data Engineer in production projects. You know Python (and PySpark) and SQL well enough to work freely and efficiently with large data sets. You have practical experience working with cloud data processing platforms: Microsoft Fabric, Azure Synapse, or Databricks. You understand and use a layered data architecture, for example, the medallion architecture. You have a good knowledge of Delta / Parquet formats. You have experience integrating diverse data sources, including working with Excel files from business users. You use the Git version control system effectively in a team environment. You are willing to sign an NDA and undergo a security check (mandatory requirement). Будет плюсом: Microsoft certifications: DP-600 or DP-700; Experience with Data Quality frameworks; Experience creating CI/CD processes dedicated to data platforms; Knowledge of R; Experience gained in the financial sector or another environment with strict audit requirements; Knowledge of agile project management methodologies (Agile, Scrum)

Условия

Salary: from PLN 900 to PLN 1,200 net per day, depending on your experience and competencies. Long-term cooperation. Technical training, certifications, and professional development. Mentoring Competence Center - becoming a member of the CC community from your first day at work is an opportunity to develop your skills, attend various conferences, and share knowledge and experience with people who face the same challenges every day. A clear career path.

• Employee benefits package (Multisport, private medical care, life insurance).
• A friendly work atmosphere, integration events, and team-building meetings.
• A referral program.