Senior Data Engineer - Remote, Scale Big Data Pipelines

BHFT; United Arab Emirates; Remote; AED 335,000 - 670,000; Full time

Benefits offered by this job

Remote work worldwide

Health insurance

Professional training

Job summary

BHFT is seeking a senior data engineer to own ingestion pipelines and storage for large-scale analytics. You will design batchstream pipelines (Airflow, Kafka, dbt) and produce reusable SDKs in Python and Go for internal data producers.

You will optimize S3/GCS storage with partitioning and TTL, ensure cost efficiency, and collaborate with Data Science, Backend and DevOps to translate requirements into scalable platform capabilities, with strong testing, monitoring and runbooks.

Qualifications

• 7+ years building production-grade data systems.
• Familiarity with market data formats (MDP, ITCH, FIX) and market data providers.
• Expert-level Python; Go and C are a plus.
• Hands-on with Airflow and Kafka.
• Strong SQL proficiency: aggregations, joins, window functions, indexing.
• Designing high-throughput APIs and data access libraries.
• Strong Linux fundamentals; containers and cloud object storage.
• Mentoring code reviews and driving engineering excellence.

Responsibilities

• Architect batch ingestion pipelines for structured and unstructured data.
• Implement and tune scalable storage, partitioning, TTL, and cost controls.
• Develop internal libraries for schema management and data contracts.
• Embed monitoring, CI/CD, testing, and runbooks for reliability.
• Collaborate with Data Science, Backend and DevOps to translate requirements into platform capabilities.

Skills

Python

Go

Kafka

Airflow

dbt

SQL

Linux

Docker

AWS S3

GCS

REST

gRPC

Tools

Airflow

Kafka

dbt

Docker

AWS S3

GCS

REST

gRPC

Job description

BHFT is seeking a senior data engineer to own ingestion pipelines and storage for large-scale analytics. You will design batchstream pipelines (Airflow, Kafka, dbt) and produce reusable SDKs in Python and Go for internal data producers.

You will optimize S3/GCS storage with partitioning and TTL, ensure cost efficiency, and collaborate with Data Science, Backend and DevOps to translate requirements into scalable platform capabilities, with strong testing, monitoring and runbooks.