Manager

Ex; Chennai; On-site; INR 1,400,000 - 2,400,000; Full time

Job summary

Ex in Chennai, India, is seeking an experienced Data Engineer to lead data engineering strategy across multiple projects. You will design and optimize PySpark-based applications, work with SQL, NoSQL, and MDM-like platforms, and build real-time data solutions and REST APIs.

You will collaborate with offshore teams, implement AWS Glue/Lambda workflows, and drive agile delivery with a strong focus on data quality and performance.

Qualifications

• BE/B.Tech/ B.Sc. in Computer Science/ Statistics, Econometrics from an accredited college or university.
• Minimum 4 years of extensive experience in design, build and deployment of PySpark-based applications.
• Hands-on experience in Redis, Valkey, OpenSearch database.
• In-depth knowledge of Python core programming language – lists, dictionaries, tuples.
• Good understanding on Informatica PowerCenter for building data pipelines.
• Understanding of Master Data Management processes & domain is preferred.
• Hands-on experience writing complex SQL queries, exporting, and importing large amounts of data using utilities.
• Ability to build abstracted, modularized reusable code components.
• Hands-on experience in generating/parsing XML, JSON documents, and REST API request/responses.
• Hands-on experience in Redis, Valkey, OpenSearch database.
• Able to quickly adapt and learn.
• Able to jump into an ambiguous situation and take the lead on resolution.
• Able to communicate and coordinate across various teams.
• Are comfortable tackling new challenges and new ways of working
• Are ready to move from traditional methods and adapt into agile ones
• Comfortable challenging your peers and leadership team.
• Can prove yourself quickly and decisively.
• Excellent communication skills and Good Customer Centricity.
• Strong Target & High Solution Orientation.

Responsibilities

• Ability to design, build and unit test applications on Spark framework on Python.
• Build Python and PySpark based applications based on data in both Relational (Oracle, SQL Server) and NoSQL database e.g. Redis, Valkey
• Build ingestion pipelines to load data into in-memory caching database.
• Build data pipelines to load and export data to/from the Master Data Management or Reference Data Management Platform using PySpark.
• Hands on experience doing at least one MDM implementation using TIBCO, Informatica, GoldenSource or custom-built MDM/RDM platforms
• Experience with designing/building Data APIs and its interaction with data consumers
• Optimize performance of the built Spark applications using configurations around Spark Context, Spark-SQL and Data Frame
• Build Python programs using module libs e.g. pandas, requests, json, flask, pickle
• Experience in processing large amounts of structured data, including integrating data from multiple sources.
• Ability to design & build real-time applications using REST API, JSON, XML
• Ability to build solutions on AWS services using Glue ETL, Lambda functions on Python
• Create and maintain integration and regression testing framework on Jenkins integrated with BitBucket and/or GIT repositories
• Participate in the agile development process, and document and communicate issues and bugs relative to data standards in scrum meetings
• Work collaboratively with onsite and offshore team.
• Ability to solve complex data-driven scenarios and triage towards defects and production issues
• Participate in code release and production deployment.
• Challenge and inspire team members to achieve business results in a fast paced and quickly changing environment

Skills

PySpark

Python

SQL

REST API

AWS

Jenkins

Education

BE/B.Tech/ B.Sc in Computer Science/ Statistics/ Econometrics

Tools

Redis

OpenSearch

Informatica PowerCenter

Git

Job description

• Job Role Application Development-Applications Development Engineering
• Experience (In Years) 9-12

Job Description

Lead data engineering strategy across multiple projects and teams. Drive innovation in data engineering practices and model optimization. Represent the data engineering team in executive meetings. Develop long-term plans for data engineering and data infrastructure.

Responsibilities

Key Responsibilities

• Ability to design, build and unit test applications on Spark framework on Python.
• Build Python and PySpark based applications based on data in both Relational (Oracle, SQL Server) and NoSQL database e.g. Redis, Valkey
• Build ingestion pipelines to load data into in-memory caching database.
• Build data pipelines to load and export data to/from the Master Data Management or Reference Data Management Platform using PySpark
• Hands on experience doing at least one MDM implementation using TIBCO, Informatica, GoldenSource or custom-built MDM/RDM platforms
• Experience with designing/building Data APIs and its interaction with data consumers
• Optimize performance of the built Spark applications using configurations around Spark Context, Spark-SQL and Data Frame.
• Build Python programs using module libs e.g. pandas, requests, json, flask, pickle
• Experience in processing large amounts of structureddata, including integrating data from multiple sources.
• Ability to design & build real-time applications using REST API, JSON, XML
• Ability to build solutions on AWS services using Glue ETL, Lambda functions on Python
• Create and maintain integration and regression testing framework on Jenkins integrated with BitBucket and/or GIT repositories
• Participate in the agile development process, and document and communicate issues and bugs relative to data standards in scrum meetings
• Work collaboratively with onsite and offshore team.
• Ability to solve complex data-driven scenarios and triage towards defects and production issues
• Participate in code release and production deployment.
• Challenge and inspire team members to achieve business results in a fast paced and quickly changing environment

Qualifications

Requirements

• BE/B.Tech/ B.Sc. in Computer Science/ Statistics, Econometrics from an accredited college or university.
• Minimum 4 years of extensive experience in design, build and deployment of PySpark-basedapplications.
• Hands-on experience in Redis, Valkey, OpenSearchdatabase
• In-depth knowledge of Python core programming language – lists, dictionaries, tuples
• Good understanding on Informatica PowerCenter for building data pipelines
• Understanding of Master Data Management processes & domain is preferred.
• Hands-on experience writing complex SQL queries, exporting, and importing large amounts of data using utilities.
• Ability to build abstracted, modularized reusable code components.
• Hands-on experience in generating/parsing XML, JSON documents, and REST API request/responses
• Hands-on experience in Redis, Valkey, OpenSearchdatabase
• Able to quickly adapt and learn.
• Able to jump into an ambiguous situation and take the lead on resolution.
• Able to communicate and coordinate across various teams.
• Are comfortable tackling new challenges and new ways of working
• Are ready to move from traditional methods and adapt into agile ones
• Comfortable challenging your peers and leadership team.
• Can prove yourself quickly and decisively.
• Excellent communication skills and Good Customer Centricity.
• Strong Target & High Solution Orientation.