Data Engineer

Datumo; Warszawa; Hybrid; PLN 180.000 - 240.000; Vollzeit

Benefits dieser Stelle

Remote work

20 days off

Mentor onboarding

Project switching

Training budget

Medicover private medical care

Medicover Sport card co-financing

English with native speaker

Company trips

Zusammenfassung

Datumo specializes in Big Data and Cloud consulting for clients worldwide, with focus in Western Europe, Poland and the USA. The team handles highly demanding projects and emphasizes best practices, testing, and CI/CD.

The work culture centers on growth, mentorship, and community. We seek senior data engineers with 3+ years of experience, strong Python and SQL, JVM languages, and hands-on exposure to Spark, BigQuery, Snowflake or Databricks.

Qualifikationen

• 3+ years of commercial experience in Big Data.
• Proven record with a chosen cloud provider (GCP preferred, Azure or AWS).
• Strong knowledge of Python and SQL.
• Good knowledge of JVM languages (Scala or Java or Kotlin).
• Understanding of Apache Spark or similar distributed data processing framework.
• Experience with data warehousing solutions (BigQuery, Snowflake, Databricks).
• Designing and implementing Big Data systems following best practices.
• Ensuring solution quality through automated tests, CI/CD and code review.
• Proven collaboration with businesses.
• English proficiency at min. B2 level, proficient in Polish.

Kenntnisse

Big Data

Python

SQL

JVM languages

Spark

CI/CD

Collaboration

English B2

Tools

GCP

Azure

AWS

BigQuery

Snowflake

Databricks

Airflow

Kafka

Docker

Kubernetes

Jobbeschreibung

Datumo specializes in providingBig Data and Cloudconsulting services to clients from all over the world, primarily in Western Europe, Poland and the USA. Core industries we support includee-commerce, telecommunications and life science.Our team consists ofexceptional peoplewhose commitment allows us to conducthighly demanding projects.

Our team members tend to stick around for more than 3 years, and when a project wraps up, we don't let them go - we embark on a journey to discover exciting new challenges for them. It's not just a workplace; it's a community that grows together!

What we expect:

Must-have:

• at least 3 years of commercial experience in Big Data,
• proven record with a selected cloud provider GCP preferred, Azure or AWS,
• strong knowledge of Python and SQL,
• good knowledge of JVM languages - Scala or Java or Kotlin,
• understanding of Apache Spark or similar distributed data processing framework,
• experience with data warehousing solution (BigQuery, Snowflake, Databricks),
• designing and implementing Big Data systems following best practices,
• ensuring solution quality through automatic tests, CI / CD and code review,
• proven collaboration with businesses,
• English proficiency at min. B2 level, proficient in Polish.

Nice to have:

• familiarity with Apache Airflow or similar pipeline orchestrator,
• knowledge of Apache Kafka, Docker and Kubernetes technologies,
• knowledge of real-time data processing,
• experience in Machine Learning projects,
• experience in Apache Flink,
• willingness to share knowledge (conferences, articles, open-source projects).

What's on offer:

• 100% remote work, with workation opportunity,
• 20 free days,
• onboarding with a dedicated mentor,
• project switching possible after a certain period,
• individual budget for training and conferences,
• benefits: Medicover private medical care, co-financing of the Medicover Sport card,
• opportunity to learn English with a native speaker,
• regular company trips and informal get-togethers.

Development opportunities in Datumo:

• participation in industry conferences,
• establishing Datumo's online brand presence,
• support in obtaining certifications (e.g. GCP, Azure, Snowflake),
• involvement in internal initiatives, like building technological roadmaps,
• access to internal technological training repositories.

Discover our exemplary projects:

The project integrates data from edge devices into the cloud using Azure services. The platform supports data streaming via either the IoT Edge environment with Java or Python modules, or direct connection using Kafka protocol to Event Hubs. It also facilitates batch data transmission to ADLS. Data transformation from raw telemetry to structured tables is done through Spark jobs in Databricks or data connections and update policies in Azure Data Explorer.

Petabyte-scale data platform migration to Google Cloud

The goal of the project is to improve scalability and performance of the data platform by transitioning over a thousand active pipelines to GCP. The main focus is on rearchitecting existing Spark applications to either Cloud Dataproc or Cloud BigQuery SQL, depending on the Client's requirements and automate it using Cloud Composer.

The project centers on developing and overseeing a data platform for an asset management company focused on ESG investing. Databricks is the central component. The platform, built on Azure cloud, integrates various Azure services for diverse functionalities. The primary task involves implementing and extending complex ETL processes that enrich investment data, using Spark jobs in Scala. Integrations with external data providers, as well as solutions for improving data quality and optimizing cloud resources, have been implemented.

Realtime Consumer Data Platform

The initiative involves constructing a consumer data platform (CDP) for a major Polish retail company. Datumo actively participates from the project’s start, contributing to planning the platform’s architecture. The CDP is built on Google Cloud Platform (GCP), utilizing services like Pub/Sub, Dataflow and BigQuery. Open-source tools, including a Kubernetes cluster with Apache Kafka, Apache Airflow and Apache Flink, are used to meet specific requirements. This combination offers significant possibilities for the platform.