Senior Data Engineer

Principal Financial Group; Hyderabad; On-site; INR 2,500,000 - 4,500,000; Full time

Job summary

Principal Financial Group in Hyderabad, India is seeking a data engineer with 3+ years of experience to design, build, and optimize enterprise data pipelines using AWS Glue, S3, Lambda, Athena, and Snowflake. You will collaborate with ML Ops, data scientists, and other engineers to ensure scalable, secure data delivery.

The role emphasizes feature pipelines for an enterprise feature store, CI/CD practices, and automated testing.

Qualifications

• Graduate degree in a computer science or engineering field.
• Significant experience with AWS Glue, Snowflake, and PySpark.
• Proficiency in Python, SQL, and data pipeline development.

Responsibilities

• Build and maintain enterprise data pipelines using AWS tools and Snowflake.
• Deliver analytics-ready datasets and ensure data quality and lineage.
• Collaborate with Data Engineering, MLOps, and Data Scientists on training/inference pipelines.

Skills

Python

Snowflake

AWS Glue

PySpark

Education

Bachelor's degree in CS/Engineering

Tools

GitHub

Snowflake Snowpipe

Snowpark

Job description

Additional Information

Must-Have Skills

• Significant experience working with AWS services such as Glue, S3, Lambda, EventBridge, IAM, and Athena.
• Skilled in developing and maintaining automated build and deployment processes on GitHub, along with Snowflake functionalities such as streams, tasks, stored procedures, snowpipe, and Schemachange.
• Must have - Python, Snowflake, AWS Glue, PySpark.

Good-to-Have Skills

• Development and performance optimization of AWS Glue PySpark jobs, managing distributed compute resources and worker nodes to process high-volume datasets efficiently.
• Familiarity with data science development practices, including AWS SageMaker and model deployment and monitoring approaches, without direct ownership of deployment.
• Exposure to enterprise-grade feature stores, or to third-party feature store solutions on cloud platforms.
• Familiarity with Model Lifecycle Management practices and positioning data for training / inference pipelines.
• Infrastructure as code (AWS CDK) and deployment automation (GitHub Actions).
• Domain background in retirement, annuities, wealth, or insurance.
• Nice to have AWS Cloud or SnowPro certification.

Responsibilities

• Bring a background spanning more than three years in data engineering, focusing on AWS tools like Glue, S3, Lambda, EventBridge, IAM, and Athena, combined with Snowflake expertise involving snowpipe, streams, tasks, stored procedures, and Snowpark.
• Build and maintain feature pipelines for the enterprise feature store, applying proven feature engineering practices such as encoding, scaling, and imputation, aligned with evolving industry and consulting standards.
• Deliver analytics-ready datasets, following established Exploratory Data Analysis techniques along with technical feasibility, scalability, and integration with enterprise systems.
• Build reusable Enterprise data pipelines using software engineering practices including CI/CD concepts, Test Driven Development practices, and Secure Coding practices.
• Take ownership of moderately complex tasks end to end - design, build, test, deploy, and support - writing readable, maintainable code while adhering to coding practices and data protection standards.
• Engage in requirement analysis, testing, debugging, documentation, and quality review across data management process builds, models, and architectures.
• Work with multi-functional teams across Data Engineering, Machine Learning Operations, and Data Scientists to position data for training and inference pipelines, without directly owning model development or deployment activities.
• Own data quality for what you deliver, including automated tests, reconciliation, and monitoring, so that feature assets stay trustworthy in production.
• Document feature lineage, definitions, and usage so that other teams reuse what you build instead of rebuilding it.
• Leverage AI-powered engineering tools, including GitHub Copilot and other approved GenAI solutions, to improve developer efficiency, code quality, documentation, test case generation, and resolving challenges.
• Demonstrate an outlook of AI adoption and continuous learning. Proactively find opportunities to use AI‑assisted development techniques. These techniques help speed delivery, automate repetitive tasks, improve data engineering workflows, and boost operational efficiency.
• Communicate assumptions, seek clarification on tasks to minimize the need for rework, and actively participate in team meetings and discussions.
• Follow established project execution processes, coding standards, and industry guidelines.
• Get actively involved in training, self-development, and exchanging expertise, contributing actively to organization-level cloud initiatives.
• Solve problems with analytical thinking and apply root cause analysis.
• Easily switch between working autonomously or in cooperation with others.

Abilities

• proven experience as a data engineer with a demonstrated history of success.
• Proficiency in Python, SQL, and working with relational databases, along with hands‑on PySpark.
• Experience building and optimizing data pipelines.
• Experience working with CI/CD Pipelines in GitHub.
• Working understanding of Feature Engineering, with the ability to contribute features that meet the team’s standards for reuse, documentation, and lineage.
• Excellent communication and collaboration skills.

Qualifications

• Graduate - Bachelor’s degree or Master’s degree (B.E. / B.Tech / M.S. / M.Tech / MCA) in Computer Science, Technology, Engineering, Mathematics, or a related subject area.