data engineer for cloud-native data platforms

Enfint; Warszawa; On-site; PLN 180,000 - 300,000; Full time

Job summary

Sigma Software ищет опытного Data Engineer для разработки масштабируемых облачных платформ данных. Ваша задача — создавать пайплайны, мигрировать данные в облако, настраивать Data Lake/Lakehouse и поддерживать качество данных.

Требуется 5+ лет опыта в Data Engineering, владение Python/SQL, Spark/PySpark, Databricks или Snowflake, опыт облачных провайдеров и инструментов DAG. Конкурентная ставка, гибкость и возможность работать в команде профессионалов.

Qualifications

• 5+ лет опыта в Data Engineering и разработке пайплайнов.
• Опыт Python и SQL для построения конвейеров обработки данных.
• Опыт Spark / PySpark, оптимизация запросов и производительности.
• Опыт Databricks или Snowflake.
• Опыт работы в облаке (Azure/AWS/GCP).
• Опыт потоковой обработки: Kafka, Spark Structured Streaming.
• Понимание ETL/ELT, Data Vault, Dimensional modelling.
• Опыт orchestration: Airflow, Azure Data Factory или аналог.
• Знание Terraform или IaC, обеспечение надежности систем.
• Уровень английского не ниже Upper-Intermediate.

Responsibilities

• Разрабатывать и разворачивать масштабируемые облачные платформы данных.
• Реализовывать конвейеры Ingestion в реальном времени.
• Определять и соблюдать стандарты платформы, модели Data Lake/Lakehouse.
• Рефакторинг Spark/PySpark скриптов для производительности.
• Вводить лучшие практики качества кода, тестирования и CI/CD.
• Продвигать инструменты AI и агентские рабочие процессы.
• Обеспечивать качество, наблюдаемость и устойчивость пайплайнов.
• Разрабатывать self-service инструменты и микросервисы.
• Сотрудничать с ML, Data Science и продуктовой командами.
• Участвовать в R&D по агентному AI и LLM-пайплайнам.

Skills

Python

SQL

Spark / PySpark

Databricks

Snowflake

Облачные платформы

Kafka

Airflow

Terraform

Modeling / Data Vault

LLM integration

Tools

dbt

Unity Catalog

Apache Atlas

MLflow

Feature Stores

Job description

Описание

Sigma Software provides IT services and consulting. The data engineering team builds cloud-native data platforms, migrates legacy systems to the cloud, and develops AI-ready data infrastructure.

Задачи

• Design and build scalable, cloud-native data platforms from greenfield to production
• Implement near-real-time ingestion pipelines using event-driven patterns
• Define and enforce platform standards, including Data Lake / Lakehouse principles, medallion architecture, and data contracts
• Refactor and optimise existing Spark and PySpark scripts for performance and maintainability
• Introduce best practices for code quality, testing, and CI/CD across data pipelines
• Drive adoption of AI tooling and agentic workflows within the data engineering team
• Ensure data quality, observability, and reliability across all pipelines and platforms
• Develop self-service tooling and microservices to simplify platform usage for other teams
• Collaborate with Machine Learning, Data Science, and Product teams as a key technical contributor and thought leader
• Drive R&D efforts around agentic AI architectures, event-driven systems, and LLM-ready data pipelines

Требования

• 5+ Years of professional experience in Data Engineering
• Strong Python and SQL development skills for pipeline development and optimisation
• Proficiency in Apache Spark / PySpark, including query optimisation and performance tuning
• Hands-on experience with Databricks or Snowflake
• Experience with at least one major cloud provider: Azure, AWS, or GCP
• Experience with stream processing technologies such as Kafka and Spark Structured Streaming
• Solid understanding of ETL/ELT patterns, data modelling, including dimensional and Data Vault models, and data warehousing
• Experience with orchestration tools such as Apache Airflow, Azure Data Factory, or equivalent
• Knowledge of Infrastructure as Code, such as Terraform or equivalent
• Understanding of production-grade system requirements: reliability, scalability, observability, and performance
• Upper-Intermediate English level
• Self-driven and proactive in identifying improvements
• Comfortable working in a fast-paced, innovative environment
• Strong problem-solving mindset with attention to detail
• Open to experimenting with emerging technologies and approaches
• Nice to have: familiarity with RAG pipeline design and LLM integration patterns, knowledge of data governance frameworks and tools such as Unity Catalog and Apache Atlas, experience with dbt for data transformation and modelling, familiarity with MLflow, Feature Stores, or ML platform integration

Условия

Office location: Warsaw, Mazowieckie, Poland.