Senior Market Data Engineer

BHFT

Dubai

Remote

AED 180,000 - 360,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Remote work from anywhere
Health insurance
Professional training
Competitive compensation

Job summary

BHFT is seeking a Data Engineer to design and maintain systems for acquiring, storing, and retrieving historical market data. You will ensure data quality through validation, cleansing, and normalization while building scalable time-series storage solutions.

The role involves integrating data from multiple vendors, collaborating with researchers, and optimizing data retrieval for backtesting and strategy development in a remote, international setting.

Qualifications

  • Commercial experience with financial instruments and markets, especially for historical data usage in algo trading.
  • Familiarity with market data formats (MDP, ITCH, FIX, SWIFT) and data providers.
  • Strong Python programming skills; Go/Rust is a plus.

Responsibilities

  • Design, develop, and maintain systems for acquiring, storing, and retrieving historical market data.
  • Ensure data integrity through validation, cleansing, and normalization processes.
  • Build scalable data storage solutions for large volumes of time-series data.
  • Create versioning and reconciliation systems to handle data changes and corrections.
  • Integrate data from multiple vendors, exchanges, and proprietary sources.
  • Develop internal tools for fast data access for research and backtesting.
  • Collaborate with quantitative researchers and traders to meet data needs.
  • Ensure performance and scalability of data retrieval for backtesting.
  • Optimize storage costs while maintaining data quality and accessibility.
  • Support compliance, auditing, and regulatory requirements around data storage.

Skills

Apache Hive
S3
Hadoop
Redshift
Spark
AWS
Apache Pig
NoSQL
Big Data
Data Warehouse
Kafka
Scala

Education

Bachelors degree in Computer Science Engineering or related field

Tools

Docker
Airflow
SLURM

Job description


  • Historical Data Capture and Storage: Design develop and maintain systems for the acquisition storage and retrieval of historical market data from multiple financial exchanges brokers and market data vendors


  • Data Integrity and Accuracy: Ensure the integrity and accuracy of historical market data including implementing data validation cleansing and normalization processes.


  • Data Architecture Development: Build and optimize data storage solutions ensuring they are scalable high-performance and capable of managing large volumes of time-series data.


  • Versioning and Reconciliation: Develop systems for data versioning and reconciliation to ensure that changes in exchange formats or corrections to past data are properly handled.


  • Data Source Integration: Implement robust integrations with various market data providers exchanges and proprietary data sources to continuously collect and store historical data.


  • Data Access Tools: Build internal tools to provide easy access to historical data for research and analysis ensuring performance ease of use and data integrity


  • Collaborate with Trading and Research Teams: Work closely with quantitative researchers and traders to understand their data requirements and optimize the systems for data retrieval and analysis for backtesting and strategy development.


  • Performance and Scalability: Develop scalable solutions to handle growing volumes of historical market data including ensuring efficient queries and data retrieval for research and backtesting needs.


  • Optimize Storage Costs: Work on optimizing data storage solutions balancing cost-efficiency with performance and ensuring that large datasets are managed effectively.


  • Compliance and Auditing: Ensure historical market data systems comply with regulatory requirements and assist in data retention integrity and reporting audits.



Qualifications :

Required Skills and Experience


  • Commercial experience of financial instruments and markets (equities futures options forex etc.) particularly understanding how historical data is used for algorithmic trading.


  • Familiarity with market data formats (e.g. MDP ITCH FIX SWIFT proprietary exchange APIs) and market data providers.


  • Strong programming skills in Python (Go/Rust is a nice to have)


  • Familiarity with ETL (Extract Transform Load) processes (or other data pipeline architecture) and tools to clean normalize and validate large datasets.


  • Commercial experience in building and maintaining large-scale time series or historical market data in the financial services industry.


  • Strong SQL proficiency: aggregations joins subqueries window functions (first last candle histogram) indexes query planning and optimization.


  • Strong problem-solving skills and attention to detail particularly in ensuring data quality and reliability.


  • Bachelors degree in Computer Science Engineering or related field.



Preferred Qualifications


  • Experience in a proprietary trading firm or buy-side environment working with historical market data and its vendors.


  • Experience with data governance and compliance related to financial data storage and retrieval.


  • Experience in working with distributed data systems and tools such as Hadoop Kafka Spark or similar technologies.


  • Proficiency in containerization orchestration - Docker Airflow SLURM tools.


  • Linux/Unix expertise particularly in managing and optimizing systems for data storage and processing.


  • Experience with cloud-based storage solutions such as AWS S3 Google Cloud Storage or Azure and the ability to optimize for performance and cost.


  • Familiarity with machine learning and data science workflows to support quantitative research teams.



Additional Information :

What we offer:


  • Working in a modern international technology company without bureaucracy legacy systems or technical debt.

  • Excellent opportunities for professional growth and self-realization.

  • We work remotely from anywhere in the world with a flexible schedule.

  • We offer compensation for health insurance sports activities and professional training.


Remote Work :

Yes


Employment Type :

Full-time


Key Skills

Apache Hive,S3,Hadoop,Redshift,Spark,AWS,Apache Pig,NoSQL,Big Data,Data Warehouse,Kafka,Scala


Department / Functional Area:

Data Engineering


Experience:

years


Vacancy:

1

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Sr. DevOps Data Hadoop Engineer
Sr. DevOps Data Hadoop Engineer

Synechron • Dubai

On-site
AED 360,000 - 600,000
Data Engineer
Data Engineer

CNTXT, Inc. • Abu Dhabi

On-site
AED 240,000 - 420,000
Senior Data Engineer (Investment Data)
Senior Data Engineer (Investment Data)

Luxoft • Abu Dhabi

On-site
AED 391,000 - 614,000
Data Engineer
Data Engineer

Acesoft Labs (India) Pvt Ltd • Dubai

Hybrid
AED 200,000 - 350,000
Quantitative Software Developer
Quantitative Software Developer

Decadatum • Dubai

On-site
AED 400,000 - 650,000
Senior Data Engineer: Lakehouse & PySpark
Senior Data Engineer: Lakehouse & PySpark

Luxoft • Abu Dhabi

On-site
AED 290,000 - 469,000
Data Engineer B2C (m/f/d)
Data Engineer B2C (m/f/d)

Halian • Abu Dhabi

Remote
AED 300,000 - 480,000
Remote work
6 month extendable contract
Senior Data Engineer (Investment Data)
Senior Data Engineer (Investment Data)

Luxoft Holding • Abu Dhabi

On-site
AED 290,000 - 469,000
Senior Data Engineer
Senior Data Engineer

Akkodis • Dubai

On-site
AED 350,000 - 600,000
Junior Data Analyst / Data Scientist Why Hiring Fast Track available
Junior Data Analyst / Data Scientist Why Hiring Fast Track available

HireHouse • United Arab Emirates

Remote
AED 100,000 - 145,000
Top locations