Senior PySpark Data Engineer

Valuelabs; Chennai District; On-site; INR 2,500,000 - 5,500,000; Full time

Job summary

Valuelabs is seeking a senior data architect/engineer with 8+ years of IT experience to lead data management and analytics initiatives in banking services. The role emphasizes architecture, strategy, governance, and end-to-end data solutions across large, complex systems.

Hands-on work with DW/Big Data platforms (Hadoop, Spark, Hive) and cloud-native designs, plus CI/CD, data security and mentoring of teams. Strong communication and planning skills are required.

Qualifications

• 8+ years in information technology with deep data engineering and architecture involvement.
• Expertise in data architecture, data strategy and roadmap for large organizations and end-to-end data management.
• Experience transforming traditional DW to big data approaches with risk and data security awareness.
• Strong knowledge of DW dimensional modeling, star/snowflake schemas, SCDs, hierarchies, and data classification.
• Hands-on in cloud-native principles, designs and deployments.
• Extensive CI/CD experience and data governance, profiling, security, metadata management, and archival.
• Ability to drive delivery in a matrixed IT environment and handle multiple tasks with tight deadlines.
• Proven problem solving, communication, and presentation skills; self-starter with planning and estimation abilities.

Responsibilities

• Lead delivery in a matrixed IT environment with internal partners.
• Define data architecture, strategy and roadmaps for large-scale data solutions.
• Oversee data quality, governance, security and metadata management.
• Hands-on work with big data tools (Hadoop, Spark, Hive) and Python/PySpark.
• Drive CI/CD improvements and cloud-native deployments.
• Manage risk, data security and compliance in banking domain projects.
• Communicate effectively with stakeholders and mentor team members.

Skills

Informatica BDM Development

Data Architecture

Data Strategy

Data Governance

Data Quality

Data Profiling

Data Security

Metadata Management

Data Archival

Dimensional Modeling

Star/Snowflake Schemas

Slowly Changing Dimensions

Dimensional Hierarchies

Cloud Native

CI/CD

Data Processing Frameworks

Hadoop

Presto

Tez

Hive

Spark

Linux

Python

PySpark

GIT

Jenkins

Communication

Planning & Estimation

Education

Master or Bachelor's degree in CS/ISM or related

Tools

Hadoop

Presto

Tez

Hive

Spark

Linux

Python

PySpark

GIT

Jenkins

Job description

Secondary Skill: Informatica BDM Development

Education: Master or Bachelors degree in computer science, information systems management or related field.

Experience - 8+ yrs

• Experience: More than 8+ years of experience in information technology, with 3+ years spent in data engineering, architecture and technology solutions definitions and implementations.
• Extensive experience in banking services or related domain Knowledge & Skill: Expertise in Data Architecture, Data Strategy and Roadmap for large and complex organization and systems and implemented large scale end-to-end Data Management & Analytics solutions.
• Experience in transforming traditional Data Warehousing approaches to Big Data based approaches and proven track record of managing risks and data security.
• Expertise with DW Dimensional modeling techniques, Star & Snowflake schemas, modeling slowly changing dimensions and role playing dimensions, dimensional hierarchies, and data classification.
• Experiences in cloud native principals, designs and deployments.
• Extensive experience working with and enhancing Continuous Integration (CI) and Continuous Development (CD) environments.
• Expertise in Data Quality, Data Profiling, Data Governance, Data Security, Metadata Management, and Data Archival Define workload migration strategies using appropriate tools.
• Drive delivery in a matrixed environment working with various internal IT partners.
• Demonstrated ability to work in a fast paced and changing environment with short deadlines, interruptions, and multiple tasks/projects occurring simultaneously.
• Must be able to work independently and have skills in planning, strategy, estimation, scheduling. Strong problem solving, influencing, communication, and presentation skills, self-starter. Experience with data processing frameworks and platforms (Hadoop, Presto, Tez, Hive, Spark etc.).
• Hands-on experience in work with bigdata platforms and related programming languages (e.g. Linux, Python, PySpark, GIT, Jenkins). Behavioral Competencies: Business Acumens Organization leadership skills Coaching and mentoring Excellent analytical skills
• Demonstrate critical and systems of thinking ability.
• Ability to negotiate and influence Disciplined, organized Flexibility / Adaptability Visionary Performance driven

Preferred candidate profile