Data engineer

Fireart; Warszawa; On-site; PLN 170,000 - 250,000; Full time

Benefits offered by this job

Competitive compensation

Long-term collaboration

Open environment

Flexible collaboration

Job summary

Fireart is seeking a Data Engineer to design, develop, and optimize our data infrastructure on Databricks and GCP. You will architect scalable ETL/ELT pipelines using BigQuery, Airflow, dbt, Dataflow, and Pub/Sub, ensuring high availability and data quality.

You will collaborate with Data Science, Analytics, and Product teams; implement CI/CD for data components; use Docker and Kubernetes for microservices; and uphold governance with GDPR/CCPA compliance.

Qualifications

• 3+ years of professional experience in data engineering.
• Strong experience with BigQuery environments and Google Cloud Storage data lakes.
• Proficiency in Apache Airflow for scheduling and ETL/ELT design.
• Experience with dbt transformations and data quality tooling.

Responsibilities

• Define and implement the data architecture on GCP, including data warehousing in BigQuery/Databricks.
• Build and optimize ETL/ELT pipelines using Airflow, dbt, Dataflow, and Pub/Sub.
• Implement event-driven or asynchronous data workflows between microservices.
• Establish CI/CD pipelines for data components and maintain infrastructure as code with Terraform.
• Collaborate with Data Science, Analytics, and Product teams to support analytics and ML initiatives.

Skills

BigQuery

Databricks

Airflow

dbt

Python/Java/Scala

GCP

Data quality

Data architecture

Tools

Terraform

RabbitMQ

Docker

Kubernetes

Jenkins

GitLab CI

Ansible

Job description

Описание:

We are looking for a Data Engineer to design, develop, and optimize our data infrastructure on Databricks. You will architect scalable pipelines using BigQuery, Google Cloud Storage, Apache Airflow, dbt, Dataflow, and Pub/Sub, ensuring high availability and performance across our ETL/ELT processes. You will leverage Great Expectations to enforce data quality standards.

Задачи:

Define and implement the overall data architecture on GCP, including data warehousing in BigQuery/Databricks, data lake patterns in Google Cloud Storage, and Data Mart (Data Mach) solutions. Integrate Terraform for Infrastructure as Code to provision and manage cloud resources efficiently. Establish both batch and real-time data processing frameworks to ensure reliability, scalability, and cost efficiency. Design, build, and optimize ETL/ELT pipelines using Apache Airflow for workflow orchestration. Implement dbt (Data Build Tool) transformations to maintain version-controlled data models in BigQuery, ensuring consistency and reliability across the data pipeline. Use Google Dataflow (based on Apache Beam) and Pub/Sub for large-scale streaming/batch data processing and ingestion. Automate job scheduling and data transformations to deliver timely insights for analytics, machine learning, and reporting. Implement event-driven or asynchronous data workflows between microservices. Employ Docker and Kubernetes (K8s) for containerization and orchestration, enabling flexible and efficient microservices-based data workflows. Implement CI/CD pipelines for streamlined development, testing, and deployment of data engineering components. Enforce data quality standards using Great Expectations or similar frameworks, defining and validating expectations for critical datasets. Define and uphold metadata management, data lineage, and auditing standards to ensure trustworthy datasets. Implement security best practices, including encryption at rest and in transit, Identity and Access Management (IAM), and compliance with GDPR or CCPA where applicable. Collaborate with Data Science, Analytics, and Product teams to ensure the data infrastructure supports advanced analytics, including machine learning initiatives. Maintain Data Mart (Data Mach) environments that cater to specific business domains, optimizing access and performance for key stakeholders.

Требования:

3+ years of professional experience in data engineering, with at least 1 year in mobile data Proven track record building and maintaining BigQuery environments and Google Cloud Storage based data lakes. Deep knowledge of Apache Airflow for scheduling/orchestration and ETL/ELT design. Experience implementing dbt for data transformations, RabbitMQ for event-driven workflows, and Pub/Sub + Dataflow for streaming/batch data pipelines. Familiarity with designing and implementing Data Mart (Data Mach) solutions, as well as using Terraform for IaC. Strong coding capabilities in Python, Java, or Scala, plus scripting for automation. Experience with Docker and Kubernetes (K8s) for containerizing data-related services. Hands-on with CI/CD pipelines and DevOps tools (e.g., Terraform, Ansible, Jenkins, GitLab CI) to manage infrastructure and deployments. Proficiency in Great Expectations (or similar) to define and enforce data quality standards. Expertise in designing systems for data lineage, metadata management, and compliance (GDPR, CCPA). Strong understanding of OLTP (Online Transaction Processing) and OLAP (Online Analytical Processing) systems. Excellent communication skills for both technical and non-technical audiences. High level of organization, self-motivation, and problem-solving aptitude.

Условия:

Competitive compensation based on your experience and expertise Long-term contract-based collaboration A collaborative and open environment Flexible collaboration arrangements