Platform Engineer with AWS AgentCore Evaluation

Aether Biomedical; United States; Hybrid; USD 43,000 - 58,000; Full time

Benefits offered by this job

Vacation days: up to 26 days/year

Health and life insurance

English language classes

Flexible workplace: hybrid/remote

Learning platforms access

Job summary

Aether Biomedical is seeking a Platform Engineer to advance our AI agent evaluation platform. You will build Python APIs, implement secure secrets management, and integrate semantic search and vector indexing within CI/CD pipelines.

This role emphasizes observability, security, and scalable deployment gates across production environments. Ideal candidates have 3+ years in platform/backend engineering, hands-on Vault/AWS Secrets Manager experience, and familiarity with OpenSearch/pgvector.

Qualifications

• 3+ years of experience in platform or backend engineering.
• Hands-on with Vault or AWS Secrets Manager.
• Experience with vector or semantic search implementations.
• Knowledge of credential management API integration.
• Familiar with MCP server capabilities and semantic taxonomy design.

Responsibilities

• Design and implement semantic discovery using vector embeddings or OpenSearch.
• Develop and maintain Python APIs for credentials retrieval.
• Integrate Vault and AWS Secrets Manager within platform services.
• Build semantic indexing pipelines and search experiences.
• Design semantic domain taxonomies to improve information retrieval.
• Collaborate with security and application teams on secure access patterns.
• Implement monitoring and logging for platform services.
• Support MCP server features across enterprise systems.
• Contribute to architecture reviews and platform roadmaps.
• Ensure compliance with security and credential management standards.

Skills

Platform/backend engineering

Secrets management (Vault / AWS SM)

Vector/semantic search

Python API development

MCP server capabilities

OpenSearch / pgvector

Tools

HashiCorp Vault

AWS Secrets Manager

OpenSearch

pgvector

Job description

Platform Engineer with AWS AgentCore Evaluation

Python

AWS AgentCore Evaluation API, AWS AgentCore Runtime, AWS CloudWatch, CloudWatch Embedded Metric Format (EMF), GitHub Actions, CI/CD Pipelines, Python, AWS Services, Monitoring and Observability Tools, Quality Evaluation Frameworks

Project Overview

The project focuses on establishing a scalable evaluation and monitoring platform for AI agents and intelligent applications. The solution provides automated deployment gates, online quality assessment, operational visibility, and regression detection capabilities to help maintain high standards of reliability and performance.

Team

Medium team (10-20 people)

You will work closely with platform engineers, DevOps specialists, AI engineers, and cloud architects. The team collaborates on automation, observability, deployment processes, and operational excellence, ensuring that AI systems meet quality requirements throughout the software delivery lifecycle.

Position Overview

We are looking for a Platform Engineer with experience in CI/CD automation, cloud observability, and AI platform operations to help build and maintain quality evaluation pipelines for agent based systems. In this role, you will integrate automated evaluation frameworks into deployment workflows, develop monitoring solutions, and support continuous quality validation across production environments.

Responsibilities

• Design and implement semantic discovery capabilities using vector embeddings, OpenSearch, or pgvector
• Develop and maintain Python based APIs for secure credential retrieval and secrets access
• Integrate and manage HashiCorp Vault and AWS Secrets Manager within platform services
• Build and optimize semantic indexing pipelines and search experiences
• Design and maintain semantic domain taxonomies to improve information discovery
• Collaborate with platform, security, and application teams to ensure secure access patterns
• Implement monitoring, logging, and operational best practices for platform services
• Support MCP server capabilities and integrations across enterprise systems
• Contribute to architecture discussions, technical design reviews, and platform roadmap initiatives
• Ensure compliance with security, governance, and credential management standards

Requirements

• 3+ years of experience in platform engineering or backend engineering
• Hands on experience with secrets management solutions such as HashiCorp Vault or AWS Secrets Manager
• Experience implementing vector search, semantic search, or embedding based search solutions
• Semantic discovery implementation using vector embeddings, OpenSearch, or pgvector
• HashiCorp Vault integration for credential and secrets management
• AWS Secrets Manager integration as an alternative secrets management solution
• Development of Python APIs for credential retrieval and access management
• MCP server capabilities and semantic domain taxonomy design

Nice to have

• Experience with AWS Bedrock embedding models for semantic indexing
• Experience with OpenSearch Service or pgvector on Amazon RDS
• Knowledge of OAuth token lifecycle management

What We Offer

• Vacation days: Up to 26 business days per year.
• 10 illness/special days off per year (fully paid, no medical papers needed) for all contract types
• Health and life insurance (Luxmed)
• MyBenefit platform with Multisport option
• Internal psychological support service
• English language classes from the first working day
• Access to external learning platforms: O’Reilly, LinkedIn Learning, Udemy, and a wide catalog of diverse internal training
• Flexible workplace: work from the office, from home, or choose a hybrid option
• Opportunities to develop as a public speaker, mentor, or technical interviewer
• Fully paid idle (bench) when not involved in a project

2892,45 - 6179,33 USD/month (depends on contract)