Senior Data Engineer

BHFT; Dubai; Remote; USD 70,000 - 120,000; Full time

Job summary

A leading international tech company is seeking a Data Engineer to architect and optimize data pipelines and storage solutions for market data. The role involves collaboration with cross-functional teams to ensure high-quality data management and observability. Candidates should have expert-level Python skills, experience with orchestration tools like Airflow, and a strong understanding of SQL and market data formats.

Benefits

Flexible schedule

Compensation for health insurance

Support for sports activities

Professional training opportunities

Qualifications

• Expert-level Python; familiarity with market data formats.
• Hands-on experience with orchestration tools (Airflow) and event streams (Kafka).
• Strong SQL proficiency and designing high-throughput APIs.

Responsibilities

• Architect batch and stream pipelines for market data.
• Implement and tune data storage for analytics.
• Develop internal libraries for data management and validation.

Skills

Python

SQL

Linux

Kafka

Airflow

Docker

Job description

Ingestion & Pipelines: Architect batch and stream pipelines (Airflow, Kafka, dbt) for diverse structured and unstructured market data. Provide reusable SDKs in Python and Go for internal data producers.

Storage & Modeling: Implement and tune S3, column-oriented, and time-series data storage for petabyte-scale analytics; own partitioning, compression, TTL, versioning, and cost optimization.

Tooling & Libraries: Develop internal libraries for schema management, data contracts, validation, and lineage; contribute to shared libraries and services for internal data consumers for research, backtesting, and real-time trading purposes.

Reliability & Observability: Embed monitoring, alerting, SLAs, SLOs, and CI/CD; champion automated testing, data quality dashboards, and incident runbooks.

Collaboration: Partner with Data Science, QuantResearch, Backend, and DevOps teams to translate requirements into platform capabilities and promote best practices.

Qualifications

Required Skills & Experience

- Familiarity with market data formats (e.g., MDP, ITCH, FIX, proprietary exchange APIs) and market data providers.

- Expert-level Python (Go and C++ are a plus).

- Hands-on experience with modern orchestration tools (Airflow) and event streams (Kafka).

- Strong SQL proficiency: aggregations, joins, subqueries, window functions (first, last, candle, histogram), indexes, query planning, and optimization.

- Designing high-throughput APIs (REST / gRPC) and data access libraries.

- Strong Linux fundamentals, experience with containers (Docker), and cloud object storage (AWS S3 / GCS).

- Proven track record of mentoring, conducting code reviews, and driving engineering excellence.

Additional Information

What we offer:

• Work in a modern international tech company without bureaucracy, legacy systems, or technical debt.
• Excellent opportunities for professional growth and self-realization.
• Remote work from anywhere in the world with a flexible schedule.
• Compensation for health insurance, sports activities, and professional training.