Walk-in | Big Data Engineer

Tata Consultancy Services Limited; Chennai; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tata Consultancy Services Limited in Chennai seeks a big data engineer to design, develop, and maintain scalable data pipelines for extracting, transforming, and loading data from diverse sources.

Key tech includes Hadoop, Spark, Kafka, Hive, SQL/NoSQL, and ETL tools (NiFi, Talend, Informatica); you’ll optimize processing, storage, and data quality.

Cloud familiarity (AWS/Azure/GCP) and data modeling are a plus; collaborate with cross-functional teams in a fast-paced environment.

Qualifications

• Strong experience in programming languages for data manipulation and engineering tasks (Python, Java, Scala).
• Expertise in SQL and NoSQL databases.
• Hands-on with big data tech (Hadoop, Spark, Kafka, Hive) for large-scale processing and real-time streams.
• In-depth knowledge of data warehousing (Redshift, BigQuery, Snowflake) for building data warehouses.
• Proficiency in designing, developing, and maintaining ETL processes (NiFi, Talend, Informatica).

Responsibilities

• Design, develop, and maintain scalable data pipelines from diverse sources.
• Collaborate with cross-functional teams to ensure data consistency and reliability.
• Optimize data processing workflows and storage for performance and cost-efficiency.
• Implement data quality checks to ensure accuracy and completeness of data.
• Monitor data pipelines and infrastructure to maintain system stability and reliability.

Skills

Python

Java

Scala

SQL

NoSQL

ETL

Data pipelines

Data modeling

Tools

Hadoop

Spark

Kafka

Hive

NiFi

Talend

Informatica

AWS

Azure

GCP

Job description

Job Description

Job Title: bigdata engineer:

Must-Have:

• Strong experience in programming languages such as Python, Java, or Scala for data manipulation and engineering tasks.
• Expertise in SQL and NoSQL databases
• Hands-on experience with big data technologies like Hadoop, Spark, Kafka, and Hive to handle large-scale data processing and real-time data streams.
• In-depth knowledge of data warehousing solutions such as Amazon Redshift, Google BigQuery, and Snowflake for building and managing data warehouses.
• Proficiency in designing, developing, and maintaining ETL (Extract, Transform, Load) processes using tools like Apache NiFi, Talend, or Informatica.

Good-to-Have:

• Familiarity with cloud platforms like AWS, Azure, or Google Cloud for deploying, managing, and scaling data infrastructure and services.
• Strong understanding of data modeling concepts and techniques to create efficient and scalable data models.
• Experience with version control systems such as Git for code management and collaboration.
• Knowledge of data governance, data quality standards, and data security practices to ensure compliance and protection of sensitive information

Responsibility of / Expectations from the Role

1 Design, develop, and maintain scalable data pipelines for extracting, transforming, and loading data from various sources to ensure seamless data flow and accessibility.

2 Collaborate with cross-functional teams to integrate data from multiple disparate sources, ensuring consistency, accuracy, and reliability of data.

3 Optimize data processing workflows and storage solutions for performance, scalability, and cost-efficiency

4 Implement data quality checks and validation processes to ensure the accuracy, completeness, and consistency of data throughout the data lifecycle.

5 Monitor the performance of data pipelines and infrastructure, identifying and resolving issues to maintain system stability and reliability.