Data Engineer

Tata Consultancy Services Limited; Eden Prairie (MN); Presencial; USD 110 000 - 120 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Medical Coverage

401K Plan

Paid Time Off

Resumo da oferta

Tata Consultancy Services in Eden Prairie, MN is seeking a senior data engineer to design and implement scalable batch and real-time data processing solutions using Spark, Scala, and cloud-native tools. You will build ETL pipelines, optimize performance, and ensure production readiness in Kubernetes/AKS environments.

You will lead client engagements, collaborate with stakeholders, and mentor junior engineers while delivering robust data platforms at scale.

Qualificações

• Bachelor's degree in Computer Science, Engineering.
• 10+ years of hands-on software engineering in distributed systems and cloud-native solutions.
• 9+ years in consulting or advisory capacity, structuring ambiguous problems and delivering solutions.
• Experience with APIs, data pipelines, distributed systems, and cloud platforms (AWS/Azure/GCP).
• Strong experience in Azure and cloud-native data engineering.
• Hands-on with Kubernetes, AKS, for deploying and operating containerized data-processing apps.
• Experience with Docker/Kubernetes concepts: Pods, Deployments, Services, ConfigMaps, Secrets, autoscaling.
• Experience deploying data pipelines, APIs, microservices, or Spark workloads in Kubernetes/AKS.
• Advanced capability using AI-assisted software development and productivity tools.
• Lead client engagements, including discovery, design, and delivery; strong stakeholder management.
• Experience structuring ambiguous problems and driving clear, actionable solutions.
• Experience with enterprise-scale data platforms; excellent communication and storytelling.

Responsabilidades

• Develop scalable batch and real-time data processing using Spark and Scala.
• Design reusable ETL frameworks and ingestion pipelines.
• Process large-scale structured and unstructured datasets.
• Build optimized Spark jobs for high-performance distributed computing.
• Tune Spark applications for performance and efficiency.
• Improve data pipeline reliability and scalability; tackle data skew and memory issues.
• Design real-time data pipelines using Spark Structured Streaming.
• Integrate data from APIs, databases, files, and event streams; work with Kafka.
• Participate in Agile delivery; conduct code reviews and mentor juniors.
• Drive engineering best practices across teams.

Conhecimentos

Azure

Python

PySpark

Scala

Spark

Distributed systems

Data pipelines

Cloud platforms

Kubernetes

AKS

Docker

APIs

Spark Streaming

CI/CD

Client engagements

Stakeholder mgmt

Formação académica

Bachelor's degree in Computer Science or Engineering

Ferramentas

AKS

Docker

Kubernetes

Descrição da oferta de emprego

Job Description

Must Have Technical/Functional Skills

• * Bachelor's degree in Computer Science, Engineering.
• * 10+ years of hands-on software engineering experience in deploying distributed systems and cloud-native
• solutions.
• * 9+ years of experience operating in a consulting or advisory capacity, including structuring ambiguous
• problems, developing solution recommendations, and delivering against client or business objectives.
• * Experience working with APIs, data pipelines, or distributed systems.
• * Experience working with at least one cloud platform (e.g., AWS, Azure, or GCP).
• * Demonstrated experience working in client-facing engagements, including partnering directly with
• stakeholders to understand needs, communicate solutions, and support delivery in real-world
• environments.
• * Strong hands-on experience with Microsoft Azure and cloud-native data engineering solutions.
• * Hands-on experience with Kubernetes, preferably Azure Kubernetes Service (AKS), for deploying, scaling,
• and operating containerized data-processing applications and services.
• * Experience with Docker/containerization and Kubernetes concepts, including Pods, Deployments, Services,
• ConfigMaps, Secrets, resource management, autoscaling, and workload troubleshooting.
• * Experience deploying and supporting data pipelines, APIs, microservices, or Spark-based workloads in
• Kubernetes/AKS environments, with a focus on reliability, scalability, and production readiness.
• * Demonstrated advanced capability using AI-assisted software development and productivity tools.
• * Proven ability to lead client engagements, including discovery, solution design, and delivery.
• * Strong stakeholder management skills, with the ability to influence both technical and executive audiences.
• * Experience structuring ambiguous problems and driving them to clear, actionable solutions.
• * Experience working with enterprise-scale data platforms.
• * Excellent communication and storytelling skills, with the ability to adapt messaging to different audiences.

Programming & Scripting

• * Languages: PySpark and Scala
• * Database technologies: SQL and NoSQL

Roles & Responsibilities

1. 1. Develop scalable batch and real-time data processing solutions using Spark and Scala.
2. 2. Design and implement reusable ETL frameworks and ingestion pipelines.
3. 3. Process large-scale structured and unstructured datasets.
4. 4. Build optimized Spark jobs for high-performance distributed computing.
5. 5. Tune Spark applications for performance and efficiency.
6. 6. Optimize partitioning strategies and query execution plans.
7. 7. Troubleshoot data skew, shuffle bottlenecks, and memory issues.
8. 8. Improve overall data pipeline reliability and scalability.
9. 9. Design and develop real-time data pipelines using Spark Structured Streaming.
10. 10. Integrate data from APIs, databases, flat files, and event streams.
11. 11. Work with Kafka-based event-driven architectures.
12. 12. Participate in Agile delivery activities, including sprint planning.
13. 13. Conduct code reviews and mentor junior engineers.
14. 14. Drive engineering best practices across teams

Salary Range: $110,000- $120,000 a year

TCS Employee Benefits Summary:

• Discretionary Annual Incentive.
• Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
• Family Support: Maternal & Parental Leaves.
• Insurance Options: Auto & Home Insurance, Identity Theft Protection.
• Convenience & Professional Growth: Commuter Benefits & Certification & amp; Training Reimbursement.
• Time Off: Vacation, Time Off, Sick Leave & Holidays.
• Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.

Location

Eden Prairie, MN

Function

TECHNOLOGY

Role

Engineer

ID

438447

Skills

Azure | Python

Salary Range

$110,000 - $120,000

Desired Candidate Profile

Qualifications

BACHELOR OF ENGINEERING