Senior Data Engineer - Real-Time Trading Data Pipelines

hermeneutic research; Singapore; Vor Ort; SGD 140.000 - 200.000; Vollzeit

Zusammenfassung

hermeneutic research seeks a Senior Data Engineer to architect, implement, and operate the full data infrastructure for Research and Trading. You will build scalable pipelines handling market data, real-time news, and external sources, ensuring quality and reliability.

The role requires hands-on expertise across SQL, Python/Java/Scala, ETL, and modern data platforms. You will collaborate with research and engineering to drive innovation while maintaining cost efficiency and data governance.

Qualifikationen

• Degree in Computer Science, Engineering, Mathematics, or related field.
• Experience with data quality frameworks (e.g., Great Expectations, Deequ).
• Proficiency with Git/GitHub for code versioning.
• Experience with infrastructure-as-code tools (e.g., Terraform, CloudFormation).
• Exposure to containerization/orchestration (Docker, Kubernetes).
• Familiarity with data governance, data lineage, and catalog tools.

Aufgaben

• Design, build, and optimize scalable pipelines for ingesting, transforming, and integrating large-volume datasets (market data, news feeds, unstructured data).
• Ensure data quality, consistency, and real-time monitoring using tools like DBT and other validation libraries.
• Develop processes to normalize and organize our data warehouse for cross-department use.
• Apply advanced data management to ensure scalability, availability, and storage efficiency.
• Collaborate with research and analytics teams to understand data needs and build frameworks for exploration and modeling.
• Create tools to overlay data from multiple sources.

Kenntnisse

SQL

Python

Java/Scala

ETL pipelines

Airflow/dbt

Data quality

Data modeling

Snowflake

Redshift

BigQuery

Synapse

Spark

Kafka

Flink

Security & privacy

Ausbildung

CS/Engineering/Math degree

Tools

GitHub

Terraform

Docker

Kubernetes

CloudFormation

Airflow

Jobbeschreibung

hermeneutic research seeks a Senior Data Engineer to architect, implement, and operate the full data infrastructure for Research and Trading. You will build scalable pipelines handling market data, real-time news, and external sources, ensuring quality and reliability.

The role requires hands-on expertise across SQL, Python/Java/Scala, ETL, and modern data platforms. You will collaborate with research and engineering to drive innovation while maintaining cost efficiency and data governance.