PySpark Senior Data Engineer

Valuelabs; Chennai District; Hybrid; INR 900,000 - 1,400,000; Full time

Job summary

Valuelabs in Chennai, India is seeking a senior Data Architect with 8+ years in IT and 3+ years in data engineering and architecture. You will shape data strategy, design scalable data platforms, and oversee end-to-end analytics initiatives in banking.

Strong CI/CD, security, and governance experience are required. Candidate should lead cross-functional teams, mentor juniors, and drive modernization of legacy DW into big data ecosystems using Hadoop, Spark, Hive, and Presto.

Qualifications

• 8+ years in IT with 3+ years in data engineering, architecture, and strategy.
•  
• Experience in banking services or related domain knowledge with data solutions.

Responsibilities

• Lead delivery in a matrixed IT environment with internal partners.
• 1.5) Define data architecture roadmaps and oversee large-scale analytics implementations.

Skills

Data Architecture

Big Data Technologies

Python / PySpark

Cloud (AWS / GCP)

CI/CD (Jenkins, Git)

Data Governance & Security

Stakeholder Management

Informatica

Banking domain knowledge

Education

Master's degree in CS or related field

Tools

Hadoop

Hive

Spark

Presto

Tez

Job description

Role & responsibilities :

Delivery Domain: Digital Products |

Delivery Area: Data and Analytics |

Secondary Skill: Informatica BDM Development

Education: Master or Bachelors degree in computer science, information systems management or related field.

• Experience: More than 8+ years of experience in information technology, with 3+ years spent in data engineering, architecture and technology solutions definitions and implementations.
• Extensive experience in banking services or related domain Knowledge & Skill: Expertise in Data Architecture, Data Strategy and Roadmap for large and complex organization and systems and implemented large scale end-to-end Data Management & Analytics solutions.
• Experience in transforming traditional Data Warehousing approaches to Big Data based approaches and proven track record of managing risks and data security.
• Expertise with DW Dimensional modeling techniques, Star & Snowflake schemas, modeling slowly changing dimensions and role playing dimensions, dimensional hierarchies, and data classification.
• Experiences in cloud native principals, designs and deployments.
• Extensive experience working with and enhancing Continuous Integration (CI) and Continuous Development (CD) environments.
• Expertise in Data Quality, Data Profiling, Data Governance, Data Security, Metadata Management, and Data Archival Define workload migration strategies using appropriate tools.
• Drive delivery in a matrixed environment working with various internal IT partners.
• Demonstrated ability to work in a fast paced and changing environment with short deadlines, interruptions, and multiple tasks/projects occurring simultaneously.
• Must be able to work independently and have skills in planning, strategy, estimation, scheduling. Strong problem solving, influencing, communication, and presentation skills, self-starter. Experience with data processing frameworks and platforms (Hadoop, Presto, Tez, Hive, Spark etc.).
• Hands-on experience in work with bigdata platforms and related programming languages (e.g. Linux, Python, PySpark, GIT, Jenkins). Behavioral Competencies: Business Acumens Organization leadership skills Coaching and mentoring Excellent analytical skills
• Demonstrate critical and systems of thinking ability.
• Ability to negotiate and influence Disciplined, organized Flexibility / Adaptability Visionary Performance driven