Data Analyst/Engineer (Collibra) - AWS

Tata Consultancy Services; Ernakulam, Chennai District, Bengaluru; Hybrid; INR 1,200,000 - 1,800,000; Full time

Job summary

Tata Consultancy Services is seeking a Data Analyst/Engineer (collibra)-AWS to build and maintain data pipelines and transformation logic using PySpark/Databricks, enabling analytics and reporting use cases.

Responsibilities include developing ETL/ELT pipelines, data cleansing, optimization of Spark jobs, and collaboration with cross-functional teams to deliver datasets for analytics and BI needs.

Qualifications

• Experience building data pipelines using PySpark/Databricks.
• Strong SQL and Python scripting for data transformation.
• Familiarity with Collibra or data governance tools is a plus.

Responsibilities

• Develop ETL/ELT data pipelines using PySpark/Spark leveraging lakeflow declarative pipelines.
• Perform data transformation, cleansing, and enrichment.
• Optimize Spark jobs for performance and cost efficiency.
• Build datasets for analytics, reporting, and BI consumption.
• Leverage AWS query federation SDK/Databricks Datasource API for query federation.
• Work with multiple data sources (S3, APIs, etc.).
• Support data validation, reconciliation, and testing.
• Collaborate with architects, analysts, and downstream teams.
• Experience in Data transformation & pipeline development.
• Good to have experience in AWS Cloud, AWS Databricks, Python, SQL, PySpark, Spark SQL, Python, SQL, Collibra.

Skills

PySpark

Databricks

Spark

Python

SQL

AWS

Collibra

Tools

Databricks

Collibra

AWS

Job description

Role - Data Analyst/Engineer (collibra)-AWS

Desired Experience Range- 6-12 Years

Location- Chennai, Kochi, Trivandum, Bengaluru.

Job Description-

Responsible for building and maintaining data pipelines and transformation logic using PySpark/Databricks, enabling analytics and reporting use cases.

Roles & Responsibilities

• Develop ETL/ELT data pipelines using PySpark/Spark leveraging lakeflow declarative pipelines
• Perform data transformation, cleansing, and enrichment
• Optimize Spark jobs for performance and cost efficiency
• Build datasets for analytics, reporting, and BI consumption
• Leverage AWS query federation SDK/Databricks Datasource API for query federation
• Work with multiple data sources (S3, APIs, etc.) 1
• Support data validation, reconciliation, and testing
• Collaborate with architects, analysts, and downstream teams
• Experience in Data transformation & pipeline development
• Good to have experience in AWS Cloud ,AWS Databricks , Python , SQL, PySpark, Spark SQL ,Python , SQL,Collibra.