Sr.Data Engineer

Excelencia iTech Consulting; Tiruchirappalli; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Chennai City Corporation, India seeks a data engineering specialist to develop, maintain, and enhance data pipelines using ETL/ELT, with expertise in star and snowflake schemas.

You will design complex SQL queries, optimize performance, and work with cloud data warehouses and big data tools such as Kafka, Spark, and Hadoop to deliver scalable data solutions.

Qualifications

• Experience in data warehousing development and maintenance.
• End-to-end lifecycle experience: requirements, sourcing, testing, deployment.
• Advanced SQL knowledge and query optimization.
• Hands-on with big data tools and cloud platforms.

Responsibilities

• Develop, maintain, and enhance data pipelines (ETL/ELT) and processes.
• Optimize data storage and data processing for performance and scalability.
• Collaborate with cross-functional teams to meet business requirements.

Skills

SQL

ETL/ELT pipelines

PowerCenter

IDMC

IICS

Scala

C++

Java

Python

Snowflake

Databricks

Redshift

BigQuery

MS Synapse

MS ADLS

Kafka

Spark

Hadoop

NoSQL

Tools

PowerCenter

IDMC

IICS

Job description

Chennai City Corporation, India

• Development, maintenance, and enhancement of Data Pipelines(ETL/ELT) and processes with thorough knowledge of star/snowflake schemas
• Developing complex SQL queries and SQL optimization
• Hands on experience inPowerCenter/IDMC/IICS.
• Developmentexperience must be full Life Cycle experience including business requirementsgathering, data sourcing, testing/data reconciliation, and deployment withinBusiness Intelligence/Data Warehousing Architecture.
• Monitoring andoptimizing data storage and data processing
• Use of Objectfunction/object-oriented scripting languages including Scala, C++, Java andPython
• Cloud datawarehouse and data lake platforms (e.g., Snowflake, Databricks, Redshift,BigQuery, MS Synapse, MS ADLS, Apache Haddop, etc.)
• Developingdata services that are fit for purpose, resilient, scalable, and future proof,to meet user needs
• Advanced SQLknowledge and experience working with relational databases and workingfamiliarity with various databases.
• Demonstratedunderstanding and experience using software and tools including ETL, relationalSQL and NoSQL databases, big data tools like Kafka, Spark, and Hadoop
• Experiencebuilding and optimizing 'big data' data pipelines, architecture, and data sets.
• Experienceperforming root cause analysis on internal and external data and processes toanswer specific business questions and identify opportunities for improvement.
• Strong analyticskills related to working with unstructured datasets.
• Experience andknowledge of project management best practices and agile software developmentmethodologies
• A successfulhistory of manipulating, processing, and extracting value from large,disconnected datasets.
• Workingknowledge of message queuing, stream processing, and highly scalable 'big data'data stores.
• Strong projectmanagement and organizational skills.
• Experiencesupporting and working with cross-functional teams in a dynamic environment.
• Understandingof data science concepts
• Familiaritywith AI/ML frameworks and libraries

Requirements

• Inventory and warehouse management data
• Demand planning and forecasting
• Production orders, BOMs, routings
• Quality systems (defects, CAPA,nonconformance)