Hadoop, Spark, Scala Data Engineer

Tata Consultancy Services; Hyderabad; On-site; INR 1,200,000 - 2,000,000; Full time

Job summary

Tata Consultancy Services in India is seeking a data engineer to design and implement scalable data pipelines. You will work with large-scale data processing using Spark, Hadoop, Kafka, and SQL/NoSQL databases, building robust ETL processes and data warehouses.

Collaborate with cross-functional teams, optimize performance and cost, ensure data quality and reliability across AWS/Azure/GCP environments.

Qualifications

• Experience with big data technologies (Spark, Hadoop) and real-time data processing.
• Proficiency in Python or Java/Scala for data manipulation.
• Strong SQL and NoSQL database knowledge.
• Experience building data warehouses on cloud platforms (Redshift, BigQuery, Snowflake).
• Hands-on ETL development using NiFi, Talend, or Informatica.
• Experience with AWS, Azure, or Google Cloud for data infrastructure.
• Ability to design scalable data pipelines and ensure data quality and reliability.

Responsibilities

• Design, develop, and maintain scalable data pipelines for extracting, transforming, and loading data from multiple sources.
• Collaborate with cross-functional teams to integrate data and ensure data accuracy and accessibility.
• Monitor, optimize, and troubleshoot data processing workflows for performance and cost efficiency.
• Implement data quality checks and governance across the data lifecycle.
• Ensure robust data pipelines and storage solutions across cloud environments.

Skills

Big Data

Spark

HDFS

Kafka

Scala

Python

Java

SQL

NoSQL

Cloud Platforms

Tools

Apache NiFi

Talend

Informatica

Job description

Location: Bangalore, Chennai and Hyderabad

• Proficiency with Big Data, Spark, HDFS, Kafka, Scala, AWS, Azure, GCP, ETL, Hadoop, Hive etc.
• Strong experience in programming languages such as Python, Java, or Scala for data manipulation and engineering tasks.
• Expertise in SQL and NoSQL databases
• Hands-on experience with big data technologies like Hadoop, Spark, Kafka, and Hive to handle large-scale data processing and real-time data streams.
• In-depth knowledge of data warehousing solutions such as Amazon Redshift, Google BigQuery, and Snowflake for building and managing data warehouses.
• Proficiency in designing, developing, and maintaining ETL (Extract, Transform, Load) processes using tools like Apache NiFi, Talend, or Informatica.
• Familiarity with cloud platforms like AWS, Azure, or Google Cloud for deploying, managing, and scaling data infrastructure and services.
• Design, develop, and maintain scalable data pipelines for extracting, transforming, and loading data from various sources to ensure seamless data flow and accessibility.
• Collaborate with cross-functional teams to integrate data from multiple disparate sources, ensuring consistency, accuracy, and reliability of data.
• Optimize data processing workflows and storage solutions for performance, scalability, and cost-efficiency
• Implement data quality checks and validation processes to ensure the accuracy, completeness, and consistency of data throughout the data lifecycle.
• Monitor the performance of data pipelines and infrastructure, identifying and resolving issues to maintain system stability and reliability.