Hadoop Data Engineer

Tata Consultancy Services; Chennai District; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

Tata Consultancy Services in Chennai is seeking a skilled Data Engineer to design, develop, and maintain scalable data pipelines for extracting, transforming, and loading data from diverse sources. You will ensure seamless data flow and accessibility across teams and systems.

The role involves collaborating with data science, analytics, and IT partners to optimize processing workflows, validate data quality, and improve storage efficiency, reliability, and cost control.

Qualifications

• Strong experience in programming languages such as Python, Java, or Scala for data manipulation and engineering tasks.
• Expertise in SQL and NoSQL databases
• Hands-on experience with big data technologies like Hadoop, Spark, Kafka, and Hive to handle large-scale data processing and real-time data streams.
• In-depth knowledge of data warehousing solutions such as Amazon Redshift, Google BigQuery, and Snowflake for building and managing data warehouses.
• Proficiency in designing, developing, and maintaining ETL (Extract, Transform, Load) processes using tools like Apache NiFi, Talend, or Informatica.

Responsibilities

• Design, develop, and maintain scalable data pipelines for extracting, transforming, and loading data from various sources to ensure seamless data flow and accessibility.
• Collaborate with cross-functional teams to integrate data from multiple disparate sources, ensuring consistency, accuracy, and reliability of data.
• Optimize data processing workflows and storage solutions for performance, scalability, and cost-efficiency
• Implement data quality checks and validation processes to ensure the accuracy, completeness, and consistency of data throughout the data lifecycle.
• Monitor the performance of data pipelines and infrastructure, identifying and resolving issues to maintain system stability and reliability.

Skills

Python/Java/Scala

SQL/NoSQL

Big data stack

Data warehousing

ETL design/development

Tools

Apache NiFi

Talend

Informatica

Job description

• Strong experience in programming languages such as Python, Java, or Scala for data manipulation and engineering tasks.
• Expertise in SQL and NoSQL databases
• Hands-on experience with big data technologies like Hadoop, Spark, Kafka, and Hive to handle large-scale data processing and real-time data streams.
• In-depth knowledge of data warehousing solutions such as Amazon Redshift, Google BigQuery, and Snowflake for building and managing data warehouses.
• Proficiency in designing, developing, and maintaining ETL (Extract, Transform, Load) processes using tools like Apache NiFi, Talend, or Informatica.

Responsibility of / Expectations from the Role

Design, develop, and maintain scalable data pipelines for extracting, transforming, and loading data from various sources to ensure seamless data flow and accessibility.

Collaborate with cross-functional teams to integrate data from multiple disparate sources, ensuring consistency, accuracy, and reliability of data.

Optimize data processing workflows and storage solutions for performance, scalability, and cost-efficiency

Implement data quality checks and validation processes to ensure the accuracy, completeness, and consistency of data throughout the data lifecycle.

Monitor the performance of data pipelines and infrastructure, identifying and resolving issues to maintain system stability and reliability.