Senior Data Engineer: Real-Time Pipelines (Top Secret)

Metric5; Washington (District of Columbia); On-site; USD 140,000 - 170,000; Full time

Benefits offered by this job

Health & Dental Insurance

Vision Insurance

Life & Short Term Disability Insurance

401K with company match

Paid Vacation

9 Paid Holidays

Parental Leave

Employee Bonuses

Professional Development Reimbursement

Tuition Assistance Program

Job summary

Metric5 is seeking a Data Engineer to join our on-site team in Springfield, VA, with Top Secret clearance requirements.

You will build and sustain secure data infrastructure and real-time streaming pipelines using AWS, OpenShift, Kafka, Flink, and OpenSearch, collaborating with O&M, development, product, design, and client teams. The role emphasizes reliability, performance, and security across multiple domains.

Qualifications

• 5+ years of experience building scalable, real-time data streaming architectures and CDC patterns.
• Hands-on experience with Apache Kafka and deep proficiency in writing complex stream processing jobs using Apache Flink.
• Extensive experience with OpenSearch (or Elasticsearch), including cluster management, ingest pipelines, and optimized queries.
• Applied knowledge of integrating AI/ML models into data pipelines, working with vector databases (e.g., OpenSearch k‑NN).
• Experience with Debezium for CDC and Vector for observability and data routing.
• Proven experience deploying applications in Kubernetes/OpenShift with IaC and Helm/ArgoCD.

Responsibilities

• Design, build, and maintain CDC pipelines extracting data from MySQL using Debezium and routing to Kafka.
• Develop and optimize Flink applications to consume Debezium topics and output denormalized records to Kafka.
• Configure Vector pipelines to sink data into OpenSearch with reliable remapping.
• Architect OpenSearch data workflows including ingest pipelines and lifecycle policies.
• Leverage AI to enhance data enrichment and enable vector/semantic search features.
• Deploy and maintain data infrastructure on OpenShift using Helm and ArgoCD, following GitOps.

Skills

Real-time streaming

Apache Kafka

Apache Flink

OpenSearch

AI/ML in data pipelines

Debezium

Kubernetes

OpenShift

Node.js/NestJS

Education

Bachelor's Degree

Tools

OpenShift

Helm

ArgoCD

Vector

OpenSearch

Job description

Metric5 is seeking a Data Engineer to join our on-site team in Springfield, VA, with Top Secret clearance requirements.

You will build and sustain secure data infrastructure and real-time streaming pipelines using AWS, OpenShift, Kafka, Flink, and OpenSearch, collaborating with O&M, development, product, design, and client teams. The role emphasizes reliability, performance, and security across multiple domains.