Data Engineer

Tata Consultancy Services Limited; Chennai; On-site; INR 900,000 - 1,500,000; Full time

Job summary

Tata Consultancy Services Limited is seeking a Data Engineer to design and maintain scalable data pipelines, ensuring reliable extraction, transformation, and loading of data from multiple sources. You will collaborate with cross-functional teams to integrate diverse data sources and maintain data quality across the lifecycle.

The role focuses on optimizing ETL workflows, data modeling, and governance, with hands-on work using tools like Airflow, BigQuery, Dataproc, Talend, and Informatica.

Qualifications

• : Experience with Google Data Engineering services and big data tools.
• : Strong programming in Python and PySpark for data manipulation and engineering tasks.
• : Expertise in SQL and NoSQL databases and data warehousing concepts.
• : ETL design and development with Airflow, Talend or Informatica.

Responsibilities

• Design, develop, and maintain scalable data pipelines to extract, transform, and load data from multiple sources.
• Collaborate with cross-functional teams to unify data from disparate sources ensuring data quality and reliability.
• Optimize data processing workflows and storage for performance and cost-efficiency.
• Implement data quality checks and validation across the data lifecycle.
• Monitor and troubleshoot data pipelines and infrastructure for stability.

Skills

Python

PySpark

SQL

NoSQL

BigQuery

Dataproc

ETL

Airflow

Talend

Informatica

Data warehousing

Tools

Git

Job description

• Expertise in Google Data Engineering services – Big Query, Dataproc etc.
• Strong experience in programming languages such as Python, PySpark data manipulation and engineering tasks.
• Expertise in SQL and NoSQL databases
• In-depth knowledge of data warehousing solutions
• Deep understanding on data warehousing concepts
• Proficiency in designing, developing, and maintaining ETL (Extract, Transform, Load) processes using tools like Apache Airflow, Talend, or Informatica.

Good to have skills:

• Understanding of DevOps Concepts in GCP.
• Strong understanding of data modeling concepts and techniques to create efficient and scalable data models.
• Experience with version control systems such as Git for code management and collaboration.
• Knowledge of data governance, data quality standards, and data security practices to ensure compliance and protection of sensitive information.

Responsibility of / Expectations from the Role

1 Design, develop, and maintain scalable data pipelines for extracting, transforming, and loading data from various sources to ensure seamless data flow and accessibility.

2 Collaborate with cross-functional teams to integrate data from multiple disparate sources, ensuring consistency, accuracy, and reliability of data.

3 Optimize data processing workflows and storage solutions for performance, scalability, and cost-efficiency

4 Implement data quality checks and validation processes to ensure the accuracy, completeness, and consistency of data throughout the data lifecycle.

5 Monitor the performance of data pipelines and infrastructure, identifying and resolving issues to maintain system stability and reliability.