Senior Data Engineer

Develocraft sp. z o.o.; Województwo pomorskie; On-site; PLN 180,000 - 240,000; Full time

Job summary

Develocraft sp. z o.o. poszukuje inżyniera danych do projektowania, wdrażania i utrzymania platform danych i AI w środowisku Microsoft Fabric. Będziesz pracować nad architekturą lakehouse, pipeline'ami danych i modelowaniem danych.

Rola obejmuje integrację systemów, projektowanie API, utrzymanie CI/CD i praktyk MLOps, a także zapewnienie bezpieczeństwa i zgodności z regulacjami. Wymagana jest silna komunikacja z zespołami biznesowymi i technicznymi oraz samodzielność w realizacji zadań.

Qualifications

• Wymagane doświadczenie w projektowaniu i utrzymaniu platform danych i AI.
• Silne zrozumienie architektury Lakehouse i hurtowni danych.
• Znajomość Pythona, Spark, SQL i analityki danych.
• Doświadczenie w integracji systemów przez API i architektury zdarzeniowe.
• Praktyczna znajomość Azure DevOps, repozytoriów i CI/CD.
• Umiejętność dokumentowania technicznego i zapewnienia bezpieczeństwa.

Responsibilities

• Projektowanie, budowa i utrzymanie platformy danych Microsoft Fabric.
• Tworzenie i utrzymanie potoków danych, transformacji i dostaw.
• Projektowanie rozwiązań Lakehouse i Data Warehouse.
• Monitorowanie zdrowia platformy, rozwiązywanie incydentów.
• Modelowanie danych dla raportowania i AI obciążeń.
• Wdrażanie AI generatywnych i RAG, oraz integracja usług AI.
• Przygotowanie danych dla modeli AI i przebieg ich cyklu.
• Współpraca z interesariuszami w zakresie przypadków AI.
• Projektowanie API i usług dostępu do danych.
• Zapewnienie zgodności z zasadami bezpieczeństwa i regułami.

Skills

Data platform design

Data engineering

Lakehouse architecture

Python

Spark

SQL

APIs integration

Event-driven architecture

Data governance

Documentation

Stakeholder communication

Security best practices

Collaboration with teams

Regulatory awareness

Tools

Microsoft Fabric

Azure DevOps

CI/CD pipelines

MLOps

Azure AI

Job description

Nasze wymagania:

• Hands-on experience in designing, building, and maintaining data platforms, particularly Microsoft Fabric or similar solutions.
• Experience in Data Engineering, including data pipeline design, data transformation, and orchestration.
• Strong understanding of Lakehouse architecture, Data Warehousing, and data modeling.
• Proficiency in Python, Spark, SQL, and data analytics.
• Experience integrating systems through APIs, file-based interfaces, and event-driven architectures.
• Hands-on experience with Azure DevOps, including repositories, work item management, and deployment pipelines.
• Ability to manage services throughout their entire lifecycle, from design and implementation to maintenance and continuous improvement.
• Strong troubleshooting skills across data platforms, pipelines and integrations.
• Ability to produce clear and well-structured technical documentation.
• Understanding of security best practices, access control, and data governance.
• Self-driven with the ability to collaborate effectively with both business and technical teams.
• Strong communication skills, with the ability to explain technical concepts to non-technical stakeholders.
• Ability to translate business requirements into secure, scalable and maintainable technical solutions.
• Results-oriented mindset with a focus on delivering high-quality, secure, and maintainable solutions.
• Experience delivering enterprise data platforms and data engineering solutions.
• Experience supporting and maintaining data services in production.
• Experience supporting and maintaining production AI services.
• Experience working in regulated industries, particularly financial services, is advantageous.

O projekcie:

You will be responsible for designing, implementing, maintaining, and enhancing enterprise data and AI platforms, reporting directly to the Head of Enterprise Technology. In this role, you will work primarily with Microsoft Fabric, ensuring the platform is secure, scalable, reliable, and well-governed. You will play a key role in delivering data services that support reporting, analytics, business process automation, and AI-driven solutions across the organization.

Zakres obowiązków:

• Design, build, maintain, and enhance the Microsoft Fabric data platform, ensuring its availability, performance, scalability, and reliability.
• Develop and maintain data pipelines for data ingestion, transformation, and delivery.
• Design and manage Lakehouse and Data Warehouse solutions.
• Monitor platform health, troubleshoot incidents, and ensure operational continuity.
• Design, develop, and maintain data models for reporting, analytics, and AI workloads.
• Build optimized datasets and semantic models.
• Ensure data quality through validation, error handling, and adherence to data modeling best practices.
• Design, implement, and maintain AI solutions using Microsoft AI and Azure AI services.
• Develop Generative AI solutions, Retrieval-Augmented Generation (RAG) applications, intelligent search capabilities, and AI agents.
• Prepare and manage data for AI models, including prompt engineering, model lifecycle management, and deployment.
• Collaborate with business stakeholders to identify AI use cases and continuously monitor the quality and effectiveness of AI solutions.
• Integrate source systems with the Microsoft Fabric platform.
• Design APIs and data access services for reporting, business applications, and AI solutions.
• Integrate AI services with enterprise applications while ensuring compliance with security and architectural standards.
• Leverage Azure DevOps for source code management, work item tracking, and deployment automation.
• Implement and maintain CI/CD and MLOps practices.
• Create and maintain clear, comprehensive technical documentation.
• Manage access control, auditing, and data security across the platform.
• Ensure compliance with regulatory requirements and Responsible AI principles, including AI model governance, data lineage, monitoring, and risk management.
• Translate business requirements into scalable data and AI solutions using Microsoft Fabric, Azure, and AI technologies.
• Support the onboarding of new data sources, platform automation, and continuous platform enhancement.
• Manage and prioritize the engineering backlog using Azure DevOps.
• Act as a technical expert, providing guidance and resolving complex issues related to data platforms and AI solutions.