Senior Data Engineer: Azure Spark & Real-Time Pipelines

Tata Consultancy Services Limited; Eden Prairie (MN); On-site; USD 110,000 - 120,000; Full time

Benefits offered by this job

Medical Coverage

401K Plan

Paid Time Off

Job summary

Tata Consultancy Services in Eden Prairie, MN is seeking a senior data engineer to design and implement scalable batch and real-time data processing solutions using Spark, Scala, and cloud-native tools. You will build ETL pipelines, optimize performance, and ensure production readiness in Kubernetes/AKS environments.

You will lead client engagements, collaborate with stakeholders, and mentor junior engineers while delivering robust data platforms at scale.

Qualifications

• Bachelor's degree in Computer Science, Engineering.
• 10+ years of hands-on software engineering in distributed systems and cloud-native solutions.
• 9+ years in consulting or advisory capacity, structuring ambiguous problems and delivering solutions.
• Experience with APIs, data pipelines, distributed systems, and cloud platforms (AWS/Azure/GCP).
• Strong experience in Azure and cloud-native data engineering.
• Hands-on with Kubernetes, AKS, for deploying and operating containerized data-processing apps.
• Experience with Docker/Kubernetes concepts: Pods, Deployments, Services, ConfigMaps, Secrets, autoscaling.
• Experience deploying data pipelines, APIs, microservices, or Spark workloads in Kubernetes/AKS.
• Advanced capability using AI-assisted software development and productivity tools.
• Lead client engagements, including discovery, design, and delivery; strong stakeholder management.
• Experience structuring ambiguous problems and driving clear, actionable solutions.
• Experience with enterprise-scale data platforms; excellent communication and storytelling.

Responsibilities

• Develop scalable batch and real-time data processing using Spark and Scala.
• Design reusable ETL frameworks and ingestion pipelines.
• Process large-scale structured and unstructured datasets.
• Build optimized Spark jobs for high-performance distributed computing.
• Tune Spark applications for performance and efficiency.
• Improve data pipeline reliability and scalability; tackle data skew and memory issues.
• Design real-time data pipelines using Spark Structured Streaming.
• Integrate data from APIs, databases, files, and event streams; work with Kafka.
• Participate in Agile delivery; conduct code reviews and mentor juniors.
• Drive engineering best practices across teams.

Skills

Azure

Python

PySpark

Scala

Spark

Distributed systems

Data pipelines

Cloud platforms

Kubernetes

AKS

Docker

APIs

Spark Streaming

CI/CD

Client engagements

Stakeholder mgmt

Education

Bachelor's degree in Computer Science or Engineering

Tools

AKS

Docker

Kubernetes

Job description

Tata Consultancy Services in Eden Prairie, MN is seeking a senior data engineer to design and implement scalable batch and real-time data processing solutions using Spark, Scala, and cloud-native tools. You will build ETL pipelines, optimize performance, and ensure production readiness in Kubernetes/AKS environments.

You will lead client engagements, collaborate with stakeholders, and mentor junior engineers while delivering robust data platforms at scale.