Lead Engineer Devops

Tata Consultancy Services; Irving (TX); Hybrid; USD 125,000 - 140,000; Full time

Job summary

Tata Consultancy Services Limited in Irving, TX is seeking a Senior Data Engineer with data architecture, pipeline development, and Cloud DevOps responsibilities. The role focuses on building a distributed Lakehouse platform using OpenSource ecosystems and cloud services across AWS, GCP, and multi-cloud environments.

You will design, implement, and operate scalable ETL/ELT pipelines, ensure data quality, and enable secure cross-cloud data sharing while deploying containerized workloads on

Qualifications

• Bachelor's or master's degree in CS or related field.
• 7+ years in data engineering and distributed systems.
• Experience migrating legacy Hadoop to cloud-native lakehouse.
• Active certifications in Databricks, Snowflake, AWS, Kubernetes.
• Familiarity with Data Mesh and data modeling.

Responsibilities

• Design scalable lakehouse foundations using open formats.
• Build and maintain end-to-end data pipelines.
• Automate DAGs and workflows with Airflow/MWAA.
• Deploy data workloads on OpenShift and EKS.
• Implement secured cross-cloud data sharing patterns.

Skills

DevOps

DataOps

Distributed systems

Data Lakehouse

Education

Bachelor's or Master's in Computer Science / related field

Tools

Databricks

Snowflake

OpenShift

Kubernetes

MWAA

Airflow

Terraform

Job description

Roles & Responsibilities

Job Title: Senior / Lead Engineer Devops

Job Description:

We are looking for a versatile and experienced Senior Data Engineer with broad, end-to-end expertise spanning Data Architecture, Pipeline Development, and Cloud DevOps (DataOps). In this hybrid role, you will be instrumental in designing, building, and deploying our next-generation Distributed Big Data Lakehouse platform.

The ideal candidate will combine deep hands-on expertise in open-source Apache big data ecosystems with modern cloud-agnostic architectures across Databricks and Snowflake. You will lead the engineering of scalable ETL/ELT pipelines, manage orchestrations across AWS (including MWAA, SQS, EKS) and Google Cloud Platform (GCS, BigQuery), deploy containerized data workloads to Red Hat OpenShift and Amazon EKS, and implement enterprise-grade Secured Cross-Cloud Data Sharing solutions.

Key Responsibilities

Lakehouse & Data Architecture

• *Distributed Lakehouse Design: Architect and evolve open, reliable, and scalable Lakehouse foundations using open table formats (Apache Iceberg, Delta Lake, Apache Hudi) to eliminate data silos and enable both BI reporting and advanced analytics/AI.
• *Data Modeling & Governance: Design optimized dimensional, Data Vault, and relational data models; implement unified enterprise data catalogs, schema enforcement, data lineage, and metadata management across multi-cloud environments.
• *Secured Cross-Cloud Data Sharing: Architect secure, zero-trust cross-cloud data sharing patterns leveraging protocols like Delta Sharing, Snowflake Secure Data Sharing / Clean Rooms, and private cloud interconnects (AWS PrivateLink / GCP Cloud Interconnect) to share governed data without physical data duplication.

Data Engineering, ETL & Distributed Pipelines

• *Pipeline Development: Build, optimize, and maintain high-throughput batch and low-latency real-time/streaming data pipelines using Apache Spark (PySpark / Scala), Apache Flink, and Apache Kafka.
• *Orchestration & Workflow Scheduling: Design and automate complex DAGs and scheduling workflows utilizing Apache Airflow and AWS Managed Workflows for Apache Airflow (MWAA).
• *Cloud Data Integration: Integrate multi-source streaming and batch data from cloud storage (AWS S3, Google Cloud Storage / GCS) and message queues (AWS SQS, Kafka) into analytics warehouses (Snowflake) and lakehouse compute engines (Databricks Unity Catalog / Spark).
• *Data Quality & Observability: Implement automated data testing, anomaly detection, validation rules, and SLA tracking across all ETL stages.

DevOps, Containerization & Infrastructure as Code (DataOps)

• *Kubernetes & Container Orchestration: Package, deploy, scale, and monitor distributed data applications and microservices on Red Hat OpenShift and Amazon EKS / Kubernetes.
• *CI/CD Automation: Define and maintain robust CI/CD deployment pipelines (e.g., GitLab CI, GitHub Actions, Jenkins, Tekton, ArgoCD, Harness) enforcing automated unit/integration testing, container vulnerability scanning, and automated rollouts.
• *Infrastructure as Code (IaC): Provision, maintain, and version infrastructure using Terraform or CloudFormation, ensuring reproducible environments across Dev, UAT, and Production.

SDLC, Security & Enterprise Standards

Qualifications

Education & Experience

• *Bachelor's or Master's Degree in Computer Science, Data Engineering, Software Engineering, or related technical discipline.
• *7+ years of progressive experience in Data Engineering, Distributed Systems Architecture, and DataOps/DevOps.
• *Experience migrating legacy on-premises Hadoop/HDFS workloads to cloud-native Lakehouse platforms (OpenShift/EKS, Databricks, Snowflake).
• *Active certifications in relevant domains:
  
  • oDatabricks: Certified Data Engineer Professional
  • oSnowflake: SnowPro Core / Advanced Architect
  • oAWS: AWS Certified Data Engineer - Associate / Solutions Architect - Professional
  • oKubernetes: CKA (Certified Kubernetes Administrator) / Red Hat Certified Specialist in OpenShift
• *Familiarity with Data Mesh concepts, data contracts, and domain-driven data modeling.

Salary Range: $125,000 to $140,000 per year

Location

Irving, TX

Function

TECHNOLOGY

Role

Engineer

Desired Candidate Profile

Qualifications

BACHELOR OF COMPUTER SCIENCE

Skills

DevOps

Salary Range

$125,000 - $140,000