Senior Data Engineer

Principal Global Services; Hyderabad; On-site; INR 1,800,000 - 3,200,000; Full time

Job summary

Principal Global Services is seeking a data engineer with 3+ years of hands-on experience in AWS and Snowflake to build and maintain enterprise data pipelines and an analytics-ready feature store. You will work with Python, PySpark, and SQL to prepare datasets, apply feature engineering, and ensure data quality.

The role emphasizes collaboration across Data Eng, ML Ops, and Data Scientists within a large enterprise.

Qualifications

• Significant hands-on experience as a data engineer with AWS and Snowflake.
• Proficiency in Python, SQL, and PySpark; strong data handling skills.
• Experience building and optimizing data pipelines and automated build/deploy processes in GitHub.
• Knowledge of feature engineering, data lineage, reuse, and documentation.
• Excellent communication and collaboration skills.

Responsibilities

• Develop and maintain enterprise feature pipelines and feature store.
• Deliver analytics-ready datasets using EDA concepts and scalable architecture.
• Build reusable data pipelines with CI/CD, testing, and secure coding practices.
• Own end-to-end tasks: design, build, test, deploy, support.
• Ensure data quality with automated tests and monitoring; document lineage and usage.

Skills

Python

SQL

PySpark

AWS Glue

S3

Lambda

EventBridge

IAM

Athena

GitHub CI/CD

Snowflake

Snowpipe

Snowflake Streams

Snowflake Tasks

Feature Engineering

CI/CD

Education

Bachelor’s degree or Master’s degree in Computer Science/Engineering

Tools

GitHub

Snowflake

AWS Glue

Snowpipe

Schemachange

GitHub Actions

Job description

Responsibilities

• Bring a background spanning more than three years in data engineering, focusing on AWS tools like Glue, S3, Lambda, EventBridge, IAM, and Athena, combined with Snowflake expertise involving snowpipe, streams, tasks, stored procedures, and Snowpark.
• Build and maintain feature pipelines for the enterprise feature store, applying proven feature engineering practices such as encoding, scaling, and imputation, aligned with evolving industry and consulting standards.
• Deliver analytics-ready datasets, following established Exploratory Data Analysis techniques along with technical feasibility, scalability, and integration with enterprise systems.
• Build reusable Enterprise data pipelines using software engineering practices including CI/CD concepts, Test Driven Development practices, and Secure Coding practices.
• Take ownership of moderately complex tasks end to end - design, build, test, deploy, and support - writing readable, maintainable code while adhering to coding practices and data protection standards.
• Engage in requirement analysis, testing, debugging, documentation, and quality review across data management process builds, models, and architectures.
• Work with multi-functional teams across Data Engineering, Machine Learning Operations, and Data Scientists to position data for training and inference pipelines, without directly owning model development or deployment activities.
• Own data quality for what you deliver, including automated tests, reconciliation, and monitoring, so that feature assets stay trustworthy in production.
• Document feature lineage, definitions, and usage so that other teams reuse what you build instead of rebuilding it.
• Leverage AI-powered engineering tools, including GitHub Copilot and other approved GenAI solutions, to improve developer efficiency, code quality, documentation, test case generation, and resolving challenges.
• Demonstrate an outlook of AI adoption and continuous learning. Proactively find opportunities to use AI-assisted development techniques. These techniques help speed delivery, automate repetitive tasks, improve data engineering workflows, and boost operational efficiency.
• Communicate assumptions, seek clarification on tasks to minimize the need for rework, and actively participate in team meetings and discussions.
• Follow established project execution processes, coding standards, and industry guidelines.
• Get actively involved in training, self-development, and exchanging expertise, contributing actively to organization-level cloud initiatives.
• Solve problems with analytical thinking and apply root cause analysis.
• Easily switch between working autonomously or in cooperation with others.

Abilities

• proven experience as a data engineer with a demonstrated history of success.
• Proficiency in Python, SQL, and working with relational databases, along with hands-on PySpark.
• Experience building and optimizing data pipelines.
• Experience working with CI/CD Pipelines in GitHub.
• Working understanding of Feature Engineering, with the ability to contribute features that meet the team’s standards for reuse, documentation, and lineage.
• Excellent communication and collaboration skills.

Qualifications

• Graduate - Bachelor’s degree or Master’s degree (B.E. / B.Tech / M.S. / M.Tech / MCA) in Computer Science, Technology, Engineering, Mathematics, or a related subject area.

Additional Information

Must-Have Skills

• Significant experience working with AWS services such as Glue, S3, Lambda, EventBridge, IAM, and Athena.
• Skilled in developing and maintaining automated build and deployment processes on GitHub, along with Snowflake functionalities such as streams, tasks, stored procedures, snowpipe, and Schemachange.
• Must have - Python, Snowflake, AWS Glue, PySpark.

Good-to-Have Skills

• Development and performance optimization of AWS Glue PySpark jobs, managing distributed compute resources and worker nodes to process high-volume datasets efficiently.
• Familiarity with data science development practices, including AWS SageMaker and model deployment and monitoring approaches, without direct ownership of deployment.
• Exposure to enterprise-grade feature stores, or to third-party feature store solutions on cloud platforms.
• Familiarity with Model Lifecycle Management practices and positioning data for training / inference pipelines.
• Infrastructure as code (AWS CDK) and deployment automation (GitHub Actions).
• Domain background in retirement, annuities, wealth, or insurance.
• Nice to have AWS Cloud or SnowPro certification.