Senior Data Engineer

BHFT; United Arab Emirates; Remote; AED 335,000 - 670,000; Full time

Benefits offered by this job

Remote work worldwide

Health insurance

Professional training

Job summary

BHFT is seeking a senior data engineer to own ingestion pipelines and storage for large-scale analytics. You will design batchstream pipelines (Airflow, Kafka, dbt) and produce reusable SDKs in Python and Go for internal data producers.

You will optimize S3/GCS storage with partitioning and TTL, ensure cost efficiency, and collaborate with Data Science, Backend and DevOps to translate requirements into scalable platform capabilities, with strong testing, monitoring and runbooks.

Qualifications

• 7+ years building production-grade data systems.
• Familiarity with market data formats (MDP, ITCH, FIX) and market data providers.
• Expert-level Python; Go and C are a plus.
• Hands-on with Airflow and Kafka.
• Strong SQL proficiency: aggregations, joins, window functions, indexing.
• Designing high-throughput APIs and data access libraries.
• Strong Linux fundamentals; containers and cloud object storage.
• Mentoring code reviews and driving engineering excellence.

Responsibilities

• Architect batch ingestion pipelines for structured and unstructured data.
• Implement and tune scalable storage, partitioning, TTL, and cost controls.
• Develop internal libraries for schema management and data contracts.
• Embed monitoring, CI/CD, testing, and runbooks for reliability.
• Collaborate with Data Science, Backend and DevOps to translate requirements into platform capabilities.

Skills

Python

Go

Kafka

Airflow

dbt

SQL

Linux

Docker

AWS S3

GCS

REST

gRPC

Tools

Airflow

Kafka

dbt

Docker

AWS S3

GCS

REST

gRPC

Job description

Key Responsibilities

• Ingestion&Pipelines: Architect batchstream pipelines (Airflow Kafka dbt) for diverse structured and unstructured marked data. Provide reusable SDKs in Python and Go for internal data producers.

• Storage&Modeling: Implement and tune S3 columnoriented and timeseries data storage for petabytescale analytics; own partitioning compression TTL versioning and cost optimisation.

• Tooling & Libraries: Develop internal libraries for schema management data contracts validation and lineage; contribute to shared libraries and services for internal data consumers for research backtesting and real-time trading purposes.

• Reliability & Observability: Embed monitoring alerting SLAs SLOs and CI/CD; champion automated testing data quality dashboards and incident runbooks.

• Collaboration: Partner with Data Science QuantResearch Backend and DevOps to translate requirements into platform capabilities and evangelise best practices.

Qualifications

• 7years building productiongrade data systems.

• Familiarity with market data formats (e.g. MDP ITCH FIX proprietary exchange APIs) and market data providers.

• Expertlevel Python (Go and C nice to have).

• Handson with modern orchestration (Airflow) and event streams (Kafka).

• Strong SQL proficiency: aggregations joins subqueries window functions (first last candle histogram) indexes query planning and optimization.

• Designing highthroughput APIs (REST/gRPC) and data access libraries.

• Strong Linux fundamentals containers (Docker) and cloud object storage (AWSS3 / GCS).

• Proven track record of mentoring code reviews and driving engineering excellence.

Additional Information

What we offer:

• Working in a modern international technology company without bureaucracy legacy systems or technical debt.
• Excellent opportunities for professional growth and self-realization.
• We work remotely from anywhere in the world with a flexible schedule.
• We offer compensation for health insurance sports activities and professional training.

Remote Work

Yes

Employment Type

Full-time