Data Engineer - Top Secret Clearance

Metric5; St. Louis (MO); On-site; USD 140,000 - 170,000; Full time

Benefits offered by this job

Health & Dental Insurance

Vision Insurance

Life & Short Term Disability Insurance

401K with company match

Paid Vacation

9 Paid Holidays per year

Parental Leave

Employee Bonuses

Professional Development Reimbursement

Tuition Assistance Program

Job summary

Metric5 is seeking an experienced Data Engineer in St. Louis, MO, on-site ~2 days per week, to design and operate secure, scalable data infrastructure across client domains. You will build real-time streaming pipelines using Debezium, Kafka, and Flink, enabling AI-enhanced search capabilities in OpenSearch.

The role emphasizes reliability, performance, and strong collaboration with cross-functional teams. Responsibilities include CDC pipeline design, Flink optimization, OpenSearch ingest

Qualifications

• 7+ years of experience building scalable, real-time data streaming architectures and CDC patterns.
• Hands-on experience with Apache Kafka and deep proficiency in writing complex stream processing jobs using Apache Flink.
• Extensive experience with OpenSearch (or Elasticsearch), including cluster management, writing ingest pipelines, managing index templates, and writing complex, optimized search queries.
• Applied knowledge of integrating AI/ML models into data pipelines, working with vector databases (e.g., OpenSearch k-NN), or building AI-driven data products.

Responsibilities

• Design, build, and maintain CDC pipelines extracting data from MySQL using Debezium and routing to Apache Kafka.
• Develop and optimize Flink applications to consume Debezium topics, perform multi-table joins, and output denormalized records to Kafka.
• Configure Vector pipelines to consume Flink-processed topics, and reliably sink data into OpenSearch.
• Architect OpenSearch data management workflows, including ingest pipelines, index templates, and lifecycle policies.

Skills

Apache Kafka

Apache Flink

OpenSearch

AI/ML integration

Debezium

Kubernetes/OpenShift

Node.js/NestJS data contracts

High availability design

System observability

Education

Bachelors Degree

Tools

OpenShift

Kubernetes

Helm

ArgoCD

Job description

Metric5

Location: St. Louis, MO (On-site ~2 days a week)
Clearance: Top Secret Required

Position Overview

As a Data Engineer, you will work closely with O&M, development, product, design, and client teams to maintain, enhance, and deliver secure data infrastructure and search capabilities across client domains. The role requires experience building scalable, real-time data streaming architectures, designing CDC pipelines, and ensuring data is accessible, reliable, and mission-ready. You will help build, modernize, and sustain data workflows using AWS Cloud, OpenShift, and related technologies while supporting secure deployment, system performance, and continuous improvement.

Responsibilities

• Design, build, and maintain robust Change Data Capture (CDC) pipelines extracting data from MySQL databases using Debezium and routing to Apache Kafka.
• Develop and optimize Apache Flink applications to consume Debezium topics, perform complex multi-table joins, and output denormalized records back to Kafka.
• Configure and manage Vector pipelines to consume Flink-processed Kafka topics, perform object remapping, and reliably sink data into OpenSearch.
• Architect OpenSearch data management workflows, including the design and implementation of custom ingest pipelines, index templates, and lifecycle policies.
• Leverage AI skills to enhance data enrichment processes, implement vector/semantic search capabilities within OpenSearch, and support advanced analytics.
• Deploy, scale, and maintain data infrastructure on OpenShift using Helm charts and ArgoCD following GitOps best practices.
• Monitor system health, tune performance across the entire data streaming lifecycle (Kafka, Flink, OpenSearch), and ensure high availability and fault tolerance.
• Collaborate with backend engineers to ensure OpenSearch indexes are highly optimized for performant querying by downstream NestJS applications.
• Perform root cause analysis for system, application, data pipeline, and end-user issues, assisting Tier 2 support teams with complex problem resolutions.
• Maintain technical documentation related to data architecture, data flows, APIs, system configurations, and operational procedures while supporting system security coordination.
• Support occasional after-hours and weekend work for operational issue resolution, production deployments, data migrations, and maintenance windows.

Required Skills

• 7+ years of experience building scalable, real-time data streaming architectures and CDC patterns.
• Hands-on experience with Apache Kafka and deep proficiency in writing complex stream processing jobs using Apache Flink.
• Extensive experience with OpenSearch (or Elasticsearch), including cluster management, writing ingest pipelines, managing index templates, and writing complex, optimized search queries.
• Applied knowledge of integrating AI/ML models into data pipelines, working with vector databases (e.g., OpenSearch k-NN), or building AI-driven data products.
• Experience with Debezium for CDC and Vector (by Datadog) for observability and data routing.
• Proven experience deploying applications in Kubernetes/OpenShift environments with strong familiarity with infrastructure-as-code and deployment workflows using Helm and ArgoCD.
• Ability to work closely with software engineers (particularly those using Node.js/NestJS) to define data contracts and query patterns.
• Ability to design, develop, and operate highly available data services across availability zones and regions.
• Self-starter with strong problem-solving, analytical, decision-making, and verbal and written communication skills.

Preferred Skills

• Experience working with cloud platforms such as AWS.
• Support for data quality, data validation, metadata management, and data governance practices.
• Familiarity with secure software development practices, vulnerability remediation, access control, and compliance requirements in federal or classified environments.
• Familiarity with Agile, Scrum, SAFe, or other iterative development methodologies, with experience delivering solutions to government customers.
• Experience serving in an "on-call" role supporting emergency response to application or system issues on occasion.

Certifications

• Security+ certification is preferred.
• Other relevant certifications include CCNA, CCNP, CISA, CISSP, and CISM.

Years of Experience

5 years+

Education

Bachelors Degree

Salary

$140,000 - $170,000

About Metric5

Metric5 is a small business with big company benefits. We have a passionate team of smart, fun, caring professionals, and we are here for the long haul. Join our growing team in a business where your contributions make an enormous impact. Our organization offers a comprehensive employee benefits package, continuous professional development, with a best in class company culture that is enjoyable to work in and supports the growth of each of our professionals.

• Health & Dental Insurance with 100% of individual coverage paid for by the company
• Vision Insurance
• Life & Short Term Disability Insurance
• 401K with company match (employees are immediately vested)
• Paid Vacation
• 9 Paid Holidays per year (plus 2 paid floating holidays)
• Parental Leave
• Employee Bonuses
• Professional Development Reimbursement Program
• Tuition Assistance Program

Metric5 is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.

Metric5 is an Equal Opportunity/Affirmative Action employer. All qualified candidates will receive consideration for employment without regard to disability, protected veteran status, race, color, religious creed, national origin, citizenship, marital status, sex, sexual orientation/gender identity, age, or genetic information. Metric5’s commitment to diversity and inclusive selection practices includes ensuring qualified long-term unemployed job seekers receive equal consideration for employment.

Equal employment opportunity, including veterans and individuals with disabilities.

PI286031479