Remote Data Lakehouse Engineer — Real-Time Analytics

payabl.; United States; On-site; USD 120,000 - 180,000; Full time

Benefits offered by this job

Provident Fund

Annual learning budget

€150 Monthly Wolt allowance

SportsBenefits membership

Company car after one year

Complimentary parking space

25 Days vacation plus holidays and 10+

Job summary

payabl. provides global financial services, payments innovation, and banking services for businesses through its payabl.one platform. The role focuses on designing scalable data lakehouse solutions on AWS, enabling real-time data ingestion and trusted analytics layers across the bronze-to-gold medallion architecture.

Collaborate with analysts and engineers to model data, optimize PySpark jobs on EMR, and implement governance, quality checks, and cost-efficient processes.

Qualifications

• 3+ years of experience in data engineering or related roles.
• Strong SQL and data modeling skills for analytics and reporting use cases.
• Strong Python programming experience, ideally including PySpark.
• Experience designing and maintaining ETL/ELT pipelines in production.
• Experience with real-time or near-real-time data ingestion.
• Experience with Kafka or similar streaming technologies.
• Experience with CDC concepts and tools such as Debezium.
• Experience with cloud data lake or lakehouse architectures.
• Hands-on experience with AWS data services such as S3, Glue Catalog, Glue Jobs, EMR, and IAM.
• Experience with Apache Iceberg, Delta Lake, or similar open table formats.
• Experience designing curated analytics layers, including silver and gold layers in a medallion architecture.
• Experience with Apache Airflow, Dagster, or similar orchestration tools.
• Experience with relational databases such as MySQL, MariaDB, or PostgreSQL.
• Working knowledge of Unix/Linux environments and shell scripting.
• Understanding of data quality, governance, lineage, and production monitoring concepts.

Responsibilities

• Design, build, and maintain scalable data lakehouse solutions on AWS using S3, Iceberg, and Glue Catalog.
• Contribute to the evolution of the medallion architecture across bronze, silver, and gold layers.
• Build and support real-time and near-real-time ingestion pipelines using Debezium, Kafka, Kafka Connect, and Iceberg sinks.
• Monitor and troubleshoot streaming pipelines, connector issues, schema changes, data consistency, and ingestion reliability.
• Design and implement silver and gold datasets as trusted, business-ready data products.
• Work with analysts, analytics engineers, and business stakeholders to understand requirements and create reusable data models.
• Develop and maintain batch and distributed processing jobs using PySpark on AWS EMR and AWS Glue.
• Optimize data transformation jobs for performance, scalability, reliability, and cost efficiency.
• Build and maintain Apache Airflow workflows for API ingestion, batch processing, and orchestration.
• Support integrations from external and third-party systems using tools such as Airbyte.
• Implement data quality checks, validation processes, and reconciliation logic.
• Contribute to data governance practices covering documentation, ownership, lineage, access, and compliance.
• Collaborate on AWS infrastructure and deployment practices using S3, Glue, EMR, IAM, EKS, and related services.
• Support infrastructure-as-code workflows using Terraform and Terragrunt.

Skills

SQL

Data modeling

Python

PySpark

ETL/ELT

Kafka

Debezium

AWS

Airflow

Airbyte

dbt

Docker

Kubernetes

Tools

Airflow

Terraform

Terragrunt

Apache Iceberg

Delta Lake

Snowflake

Databricks

Airbyte

dbt

Docker

Kubernetes

Job description

payabl. provides global financial services, payments innovation, and banking services for businesses through its payabl.one platform. The role focuses on designing scalable data lakehouse solutions on AWS, enabling real-time data ingestion and trusted analytics layers across the bronze-to-gold medallion architecture.

Collaborate with analysts and engineers to model data, optimize PySpark jobs on EMR, and implement governance, quality checks, and cost-efficient processes.