Observability engineer in AI agents

EPAM; United States; Hybrid; USD 140,000 - 210,000; Full time

Benefits offered by this job

Hybrid/remote work

Relocation opportunities

Global offices

International assignments

Certifications support

Training & learning database

Employee Stock Purchase Plan

Medical & family care

Job summary

EPAM в США ищет старшего инженера наблюдаемости для владения многоарендной консолидированной слоем телеметрии между агентскими средами. Вы будете разворачивать и поддерживать мониторинг на уровне нескольких аккаунтов и разрабатывать подходы к мониторингу AI-агентов.

Кандидат должен иметь 5+ лет опыта в наблюдаемости/SRE, глубокие знания New Relic, NRQL, Terraform и Crossplane, опыт работы с OpenTelemetry/ADOT и практиками безопасной высоконагруженной системы. Английский не ниже B2.

Qualifications

• 5+ лет опыта в области наблюдаемости или SRE.
• Опыт консолидации телеметрии для многоарендных окружений.
• Экспертные знания администрирования New Relic на масштабе.
• Навыки NRQL в кросс-аккаунтных запросах.
• Работа с инфраструктурой как код: Terraform, Crossplane.
• Знания OpenTelemetry/ADOT и методов интеграции.
• Уровень английского не ниже B2.

Responsibilities

• Владение и администрирование многоарендного консолидированного слоя New Relic.
• Сведение телеметрии из разных проектов с использованием NRQL.
• Создание и поддержка dashboards-как-код и policy-как-код с Terraform и Crossplane.
• Настройка alert-политик в Opsgenie, миграция к ServiceNow.
• Управление Ingestion OpenTelemetry/ADOT через OTLP и Firehose.
• Проектирование размерностей метрик и низко-кардинальных фасетов.
• Обеспечение согласованности между Технической, Бизнес и Аудит层.
• Сотрудничество с командами разработки для стандартизации практик наблюдаемости.

Skills

Observability

SRE

NRQL

New Relic

Terraform

Crossplane

OpenTelemetry

ADOT

Dashboarding

English Proficiency

Tools

New Relic providers

Terraform

Crossplane

Opsgenie

ServiceNow

Job description

Описание:

We are looking for a Senior Observability Engineer to take ownership of the multi-tenant New Relic consolidation layer aggregating telemetry across our agent estate, within an Enterprise Agent Development Platform empowering engineering teams to define, orchestrate, deploy, and observe AI agents at scale across the organization.

Задачи:

• Own and administer the multi-tenant New Relic consolidation layer across the agent estate
• Aggregate telemetry from multiple project tenants through cross-account NRQL queries
• Build and maintain dashboards-as-code and alert-policy-as-code with Terraform and Crossplane
• Set up New Relic alert policies routed to Opsgenie, with planned migration to ServiceNow
• Manage OpenTelemetry / ADOT ingestion into New Relic through OTLP and Firehose pipelines
• Design metric dimensions and low-cardinality facets for large-scale agent estates
• Establish cross-account and cross-tenant telemetry aggregation and reporting workflows
• Maintain separation across Technical, Business, and Audit observability models
• Partner with engineering teams to standardize agent observability practices across the platform

Требования:

• 5+ years of experience in observability or SRE engineering
• Hands-on experience consolidating observability/telemetry for AI agent estates within a production agentic AI project, such as multi-tenant New Relic aggregation, cost/token attribution for GenAI workloads, or AgentCore GenAI Observability integration
• Expertise in New Relic administration at scale, covering management account setup and cross-account configuration
• Proficiency in NRQL for cross-account queries aggregating telemetry from multiple project tenants
• Skills in dashboards-as-code and alert-policy-as-code with Terraform New Relic provider and Crossplane
• Knowledge of New Relic alert policies routed to Opsgenie, along with familiarity with ServiceNow
• Understanding of OpenTelemetry / ADOT consumption fundamentals, covering OTLP and Firehose ingestion into New Relic
• Competency in metric dimension design for large agent estates, ensuring stable, low-cardinality facets
• Background in running New Relic or equivalent SaaS observability platforms as a multi-tenant consolidation layer
• Experience operating within a Technical / Business / Audit observability segregation model
• English proficiency at an Upper-Intermediate level (B2) or higher

Будет плюсом:

• Familiarity with AWS Bedrock AgentCore observability, covering Transaction Search and GenAI Observability
• Understanding of Kinesis Firehose metric and log forwarding pipelines from CloudWatch into New Relic
• Expertise in cost/token attribution modeling for GenAI workloads
• Knowledge of Crossplane-based provisioning of observability resources

Условия:

• Delivering innovative solutions to industry leaders, making a global impact
• Enjoyable working environment, whether it is the vibrant office or the comfort of your own home
• Opportunity to work abroad for up to two months per year
• Relocation opportunities within our offices in 55+ countries
• Corporate and social events
• Leadership development, career advising, soft skills and well-being programs
• Certifications, including GCP, Azure and AWS
• Unlimited access to EPAM's internal learning database
• Free English classes with certified teachers
• Discounts in local language schools, including online courses for the Kazakh language
• Participation in the Employee Stock Purchase Plan
• Monetary bonuses for engaging in the referral program
• Medical & family care package
• Six trust days per year (sick leave without a medical certificate)
• Coverage of psychology sessions of your choice
• Benefits package (sports activities, a variety of stores and services)
• Housing support program: preferential mortgage access via Otbasy Bank partnership