Walk-in | Data Engineer (Hadoop)

Tata Consultancy Services; Chennai District; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Tata Consultancy Services is seeking a data engineer to design, develop, and maintain scalable data pipelines that extract, transform, and load data from diverse sources. You will collaborate with cross-functional teams to ensure data consistency, accuracy, and availability across environments.

You will optimize data workflows for performance and cost, implement data quality checks, and monitor pipeline health to ensure reliable operations at scale.

Qualifications

• Proficiency in Python/Java/Scala for data tasks.
• Strong SQL and NoSQL database expertise.
• Hands-on with big data tools: Hadoop, Spark, Kafka, Hive.
• Experience building data warehouses (Redshift, BigQuery, Snowflake).
• ETL design/development with NiFi, Talend or Informatica.

Responsibilities

• Design, develop, and maintain scalable data pipelines for ETL from multiple sources.
• Collaborate with cross-functional teams to ensure data consistency and reliability.
• Optimize data processing workflows for performance and cost-efficiency.
• Implement data quality checks and validation across the data lifecycle.
• Monitor pipelines and infrastructure to maintain stability and uptime.

Skills

Python

Java

Scala

SQL

NoSQL

ETL development

Tools

Hadoop

Spark

Kafka

Hive

Redshift

BigQuery

Snowflake

NiFi

Talend

Informatica

Job description

Role & responsibilities

• Design, develop, and maintain scalable data pipelines for extracting, transforming, and loading data from various sources to ensure seamless data flow and accessibility.
• Collaborate with cross-functional teams to integrate data from multiple disparate sources, ensuring consistency, accuracy, and reliability of data.
• Optimize data processing workflows and storage solutions for performance, scalability, and cost-efficiency
• Implement data quality checks and validation processes to ensure the accuracy, completeness, and consistency of data throughout the data lifecycle.
• Monitor the performance of data pipelines and infrastructure, identifying and resolving issues to maintain system stability and reliability.

Preferred candidate profile

Must-Have

• Strong experience in programming languages such as Python, Java, or Scala for data manipulation and engineering tasks.
• Expertise in SQL and NoSQL databases
• Hands-on experience with big data technologies like Hadoop, Spark, Kafka, and Hive to handle large-scale data processing and real-time data streams.
• In-depth knowledge of data warehousing solutions such as Amazon Redshift, Google BigQuery, and Snowflake for building and managing data warehouses.
• Proficiency in designing, developing, and maintaining ETL (Extract, Transform, Load) processes using tools like Apache NiFi, Talend, or Informatica.