Sr Specialist System Engineering (Kubernetes, CI/CD Pipeline, Python automation)

AT&T; Bengaluru; On-site; INR 4,200,000 - 7,000,000; Full time

Job summary

AT&T Bangalore-based team seeks an offshore DevOps/Infra lead to steward automation tools and 4G/5G core simulators across NPRD/Labs and production. You will design CI/CD pipelines, helm charts, and IaC across hybrid infra, while managing secrets, networking, storage, and observability at scale.

The role requires hands-on Kubernetes, Terraform/OpenStack/Bicep, and advanced security practices with Vault and cert-manager. Collaboration with US stakeholders is daily.

Qualifications

• Hands-on with Kubernetes in production and CI/CD (Azure DevOps, GitHub/GitOps).
• Strong automation in Python, Ansible, Bash.
• Deep experience with Helm 3 packaging and release management; artifact publishing (JFrog Artifactory OCI).
• Expertise in secrets management (HashiCorp Vault + Ansible Vault).
• IaC provisioning (OpenStack HEAT and Azure Bicep) across hybrid environments.
• Troubleshooting across K8s clusters, networking, and stateful services.
• Advanced K8s networking/storage (Multus, SR-IOV, MetalLB, Calico; Rook-Ceph/local storage).
• Understanding of 5G/LTE telecom networking and containerized NF needs (e.g., DPDK / SCTP / GTP).
• IP/Networking/routing/Ubuntu repo's packages knowledge and hands on experience.

Responsibilities

• Lead offshore DevOps/Infra support for automation tools and 4G/5G core simulators across NPRD/Labs and PROD environments.
• Design, build, and maintain Azure DevOps CI/CD pipelines using a 3-tier architecture to automate packaging, publishing, and deployments.
• Develop and maintain Helm v3 (OCI) charts; publish artifacts to JFrog Artifactory OCI.
• Automate Kubernetes deployments via Python orchestration, including config substitution and health checks.
• Own secrets management end-to-end: Vault and Ansible Vault with rotation.
• Provision and operate KVM/QEMU VM lifecycle with SR-IOV and host networking.
• Author IaC with OpenStack HEAT and Azure Bicep for hybrid environments.
• Configure advanced Kubernetes networking and security: Multus, MetalLB, Calico.
• Manage storage/stateful services: Rook-Ceph and local storage.
• Build/maintain large Ansible automation footprint for deployment, upgrades, and vault ops.
• Operate NF pipelines and promote releases across environments.
• Maintain per-instance configuration across 20+ K8s instances.
• Implement observability with kube-prometheus-stack and DevOps metrics.
• Manage certificates with cert-manager and Vault PKI integration.
• Maintain supply-chain security artifacts (bom.yaml, pinned versions).
• Administer multi-repo governance across 30+ repos.
• Build AI-assisted DevOps tooling for diagnostics and self-healing operations.
• Provide incident response and troubleshooting.

Skills

Kubernetes

CI/CD pipelines

Python

Ansible

Bash

Helm 3

Vault

IaC

OpenStack HEAT

Azure Bicep

Tools

JFrog Artifactory OCI

KVM/QEMU

Azure DevOps

GitHub/GitOps

Job description

Job Responsibilities

• Serve as the primary offshore DevOps/Infra support lead for Automation Tools and 4G/5G Core Network Simulators across NPRD/Labs and PROD environments.
• Design, build, and maintain Azure DevOps CI/CD pipelines using a 3-tier (hyper→macro→micro) architecture to automate packaging, publishing, and deployments.
• Develop and maintain Helm v3 (OCI) charts for application and infra components; package/publish artifacts to JFrog Artifactory OCI.
• Automate Kubernetes deployments via Python orchestration (e.g., deploy.py), including config substitution, sequencing, and post-deploy health checks.
• Own secrets management end-to-end: Ansible Vault encryption, HashiCorp Vault runtime retrieval (AppRole/KV), and token rotation with zero-plaintext controls.
• Provision and operate infrastructure: KVM/QEMU/libvirt VM lifecycle on Simzone hosts; implement SR-IOV passthrough and host networking configurations.
• Author/maintain Infrastructure-as-Code using OpenStack HEAT (on-prem VM provisioning) and Azure Bicep (NAKS/AON resources).
• Configure advanced K8s networking and security: Multus NADs (multi-NIC pods), MetalLB L2 VIPs, Calico policies, SR-IOV for performance I/O.
• Manage storage and stateful services: Rook-Ceph provisioning (RWX/RWO), local storage for single-node clusters, PVC sizing, and storage class governance.
• Build/maintain large Ansible automation footprint (deployment, upgrades, backup/restore, mechIDmgmt, vault operations, compliance tasks).
• Operate AOSM lifecycle pipelines (nfPL0→nfPL4) to publish NF definitions, build deployment configs, deploy to NAKS, and onboard to HCEV.
• Drive environment promotion (NPRD→PROD): release branching, versioning updates, quarterly release coordination, and consistency across lab/pre-prod/prod.
• Maintain per-instance configuration at scale (20+ K8s instances) using real-parameters-values.json / input-values.json and enforce correctness across environments.
• Implement observability and verification: deploy kube-prometheus-stack, integrate DevOps metrics (DORA Build Events API), and standardize service health checks.
• Own certificate lifecycle management via cert-manager and vault-issuer patterns, including rotation workflows and Vault PKI integration.
• Maintain supply-chain security artifacts (bom.yaml validation, images.json version tracking, reproducible builds with pinned versions).
• Administer multi-repo governance (30+ repos across GitHub + ADO): branch protections, PR governance, and cross-repo dependency coordination.
• Build AI-assisted DevOps tooling (agentic workflow with specialized agents) to accelerate analysis, pipeline engineering, diagnostics, and self-healing operations.
• Provide incident response and operational troubleshooting: kubectl diagnosis, log analysis, NF status checks, backup/restore execution, and chaos testing support.

Job Qualifications / Required Qualifications

• Hands-on experience operating Kubernetes platforms and production-grade CI/CD pipelines (Azure DevOps, GitHub/GitOps).
• Strong automation skills in Python, Ansible, and Bash for deployment orchestration and operational tooling.
• Deep experience with Helm 3 (OCI) packaging and release management; artifact publishing (JFrog Artifactory OCI).
• Expertise in secrets management (HashiCorp Vault + Ansible Vault), credential rotation, and secure repo practices.
• Experience with infrastructure provisioning and IaC (OpenStack HEAT and/or Azure Bicep) across hybrid environments.
• Strong troubleshooting and incident management skills across K8s clusters, networking, and stateful services.
• Working knowledge of advanced K8s networking/storage (Multus, SR-IOV, MetalLB, Calico; Rook-Ceph/local storage).
• Understanding of 5G/LTE telecom networking fundamentals and containerized NF operational needs (e.g., DPDK / SCTP / GTP considerations).
• IP/Networking/routing/firewalls/Ubuntu repo's packages knowledge and hands on experience.

Preferred Qualifications

• Experience operating AOSM pipelines (nfPL0→nfPL4) and NF onboarding workflows into NAKS/AON/HCEV environments.
• Experience managing KVM/QEMU/libvirt VM fleets on bare-metal, including SR-IOV passthrough and host networking.
• Experience with supply-chain security artifacts (BOM generation/validation, digest pinning, reproducible builds).
• Proven ability to manage large-scale repo/pipeline governance across many repos with strong PR/branch policy discipline.
• Experience building or integrating AI-assisted DevOps workflows for diagnostics, automation, and operational self-healing.

Additional Job Information

This is an offshore role that requires daily collaboration with U.S. stakeholders, including overlapping work hours to ensure effective partnership and meet business needs.

Weekly Hours:

40

Time Type:

Regular

Location:

IND:KA:Bangalore / Epip Area, Hoodi Village, Whitefield Rd - Eqp: Plot 111/112, Epip Area, Hoodi Village, Whitefield Road

AT&T and its subsidiaries are committed to equal employment opportunity. All hiring, promotion, and other employment decisions remain merit-based and free from discrimination on the basis of race, color, religion, religious creed, national origin, ancestry, age, sex, sexual orientation, gender, gender identity, gender expression, physical disability, mental disability, pregnancy, medical condition, genetic information, marital status, citizenship status, military status, veteran status, or any other characteristic protected by federal, state, or local laws. In addition, AT&T will provide reasonable accommodations to qualified individuals with disabilities. AT&T is a fair chance employer and does not initiate a background check until an offer is made.