Senior Data Engineer - Real-Time Pipelines for Trading

hermeneutic research; United States; Remote; USD 140,000 - 210,000; Full time

Job summary

hermeneutic research seeks a Senior Data Engineer to architect, implement, and operate the full data infrastructure for its Research and Trading operations. You will build scalable pipelines, ingest market data, and integrate diverse sources while ensuring reliability and cost efficiency.

The role requires hands-on execution across the data lifecycle, collaborating with research and engineering teams to enable data exploration, model development, and real-time insights in a fast-paced trading

Qualifications

• Strong problem-solving and analytical thinking.

Responsibilities

• Design, build, and optimize scalable pipelines for ingesting, transforming, and integrating large-volume datasets (market data, news feeds and various unstructured data sources).
• Ensure data quality, consistency, and real-time monitoring using tools like DBT, 3rd party libraries that can facilitate data validation processes.
• Develop processes to normalize and organize our data warehouse for use across different departments.
• Apply advanced data management practices to ensure the scalability, availability, and efficiency of data storage.
• Ensure the infrastructure supports trading and research needs while maintaining data integrity, security, and performance at scale.
• Collaborate with research and analytics teams to understand their data needs and build frameworks that empower data exploration, analysis, and model development. Create tools for overlaying data from multiple sources.
• Ensure that data storage, processing, and management are done in a cost-effective manner, optimizing both hardware and software resources. Implement solutions that balance high performance with cost control.
• Stay ahead of the curve by continuously evaluating and adopting the most suitable technologies for the organization’s data engineering needs. Ensure that company’s systems align with the latest best practices in data management.

Skills

SQL

Python/Java/Scala

ETL pipelines

Airflow/dbt

Data modeling

Snowflake/Redshift/BigQuery

Spark/Kafka/Flink

Data security

Data quality checks

Anomaly detection

Education

Degree in Computer Science, Engineering, Mathematics, or related field

Tools

Apache Airflow

dbt

Git/GitHub

Terraform/CloudFormation

Docker/Kubernetes

Apache Atlas/Amundsen

Monte Carlo/Datadog

Job description

hermeneutic research seeks a Senior Data Engineer to architect, implement, and operate the full data infrastructure for its Research and Trading operations. You will build scalable pipelines, ingest market data, and integrate diverse sources while ensuring reliability and cost efficiency.

The role requires hands-on execution across the data lifecycle, collaborating with research and engineering teams to enable data exploration, model development, and real-time insights in a fast-paced trading