Senior Data Engineer: Real-Time Pipelines & OpenSearch

Metric5; St. Louis (MO); On-site; USD 140,000 - 170,000; Full time

Benefits offered by this job

Health & Dental Insurance

Vision Insurance

Life & Short Term Disability Insurance

401K with company match

Paid Vacation

9 Paid Holidays per year

Parental Leave

Employee Bonuses

Professional Development Reimbursement

Tuition Assistance Program

Job summary

Metric5 is seeking an experienced Data Engineer in St. Louis, MO, on-site ~2 days per week, to design and operate secure, scalable data infrastructure across client domains. You will build real-time streaming pipelines using Debezium, Kafka, and Flink, enabling AI-enhanced search capabilities in OpenSearch.

The role emphasizes reliability, performance, and strong collaboration with cross-functional teams. Responsibilities include CDC pipeline design, Flink optimization, OpenSearch ingest

Qualifications

• 7+ years of experience building scalable, real-time data streaming architectures and CDC patterns.
• Hands-on experience with Apache Kafka and deep proficiency in writing complex stream processing jobs using Apache Flink.
• Extensive experience with OpenSearch (or Elasticsearch), including cluster management, writing ingest pipelines, managing index templates, and writing complex, optimized search queries.
• Applied knowledge of integrating AI/ML models into data pipelines, working with vector databases (e.g., OpenSearch k-NN), or building AI-driven data products.

Responsibilities

• Design, build, and maintain CDC pipelines extracting data from MySQL using Debezium and routing to Apache Kafka.
• Develop and optimize Flink applications to consume Debezium topics, perform multi-table joins, and output denormalized records to Kafka.
• Configure Vector pipelines to consume Flink-processed topics, and reliably sink data into OpenSearch.
• Architect OpenSearch data management workflows, including ingest pipelines, index templates, and lifecycle policies.

Skills

Apache Kafka

Apache Flink

OpenSearch

AI/ML integration

Debezium

Kubernetes/OpenShift

Node.js/NestJS data contracts

High availability design

System observability

Education

Bachelors Degree

Tools

OpenShift

Kubernetes

Helm

ArgoCD

Job description

Metric5 is seeking an experienced Data Engineer in St. Louis, MO, on-site ~2 days per week, to design and operate secure, scalable data infrastructure across client domains. You will build real-time streaming pipelines using Debezium, Kafka, and Flink, enabling AI-enhanced search capabilities in OpenSearch.

The role emphasizes reliability, performance, and strong collaboration with cross-functional teams. Responsibilities include CDC pipeline design, Flink optimization, OpenSearch ingest