Senior Data Engineer

Principal-Global-Services; Hyderabad; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

Principal-Global-Services is seeking a data engineer with 3+ years of experience to design and build enterprise data pipelines in AWS and Snowflake. You will work on Glue, S3, Lambda, EventBridge, IAM, Athena, Snowpipe, streams, tasks, and Snowpark, delivering analytics-ready datasets and feature pipelines.

Collaborate with data science and ML teams, implement CI/CD, unit tests, and quality controls, document lineage, and promote AI-powered tooling to raise developer efficiency while protecting

Qualifications

• 3+ years as a data engineer with AWS and Snowflake.
• Proficient in Python, SQL and PySpark.
• Experience building and maintaining data pipelines and feature stores.

Responsibilities

• Develop and maintain enterprise data pipelines using AWS and Snowflake.
• Deliver analytics-ready datasets and feature engineering outputs.
• Collaborate with Data Engineering, MLOps and Data Scientists to enable training and inference.
• Implement CI/CD, tests, and secure coding practices.
• Document lineage, data quality tests, and monitoring.

Skills

Data engineering

Python

SQL

PySpark

Communication

Education

Bachelor’s or Master’s degree in CS/Engineering/Math

Tools

GitHub CI/CD

Snowflake

AWS Glue

S3

Lambda

EventBridge

IAM

Athena

Snowpipe

Snowflake Streams

Snowflake Tasks

Job description

Responsibilities

• Bring a background spanning more than three years in data engineering, focusing on AWS tools like Glue, S3, Lambda, EventBridge, IAM, and Athena, combined with Snowflake expertise involving snowpipe, streams, tasks, stored procedures, and Snowpark.
• Build and maintain feature pipelines for the enterprise feature store, applying proven feature engineering practices such as encoding, scaling, and imputation, aligned with evolving industry and consulting standards.
• Deliver analytics-ready datasets, following established Exploratory Data Analysis techniques along with technical feasibility, scalability, and integration with enterprise systems.
• Build reusable Enterprise data pipelines using software engineering practices including CI/CD concepts, Test Driven Development practices, and Secure Coding practices.
• Take ownership of moderately complex tasks end to end - design, build, test, deploy, and support - writing readable, maintainable code while adhering to coding practices and data protection standards.
• Engage in requirement analysis, testing, debugging, documentation, and quality review across data management process builds, models, and architectures.
• Work with multi-functional teams across Data Engineering, Machine Learning Operations, and Data Scientists to position data for training and inference pipelines, without directly owning model development or deployment activities.
• Own data quality for what you deliver, including automated tests, reconciliation, and monitoring, so that feature assets stay trustworthy in production.
• Document feature lineage, definitions, and usage so that other teams reuse what you build instead of rebuilding it.
• Leverage AI-powered engineering tools, including GitHub Copilot and other approved GenAI solutions, to improve developer efficiency, code quality, documentation, test case generation, and resolving challenges.
• Demonstrate an outlook of AI adoption and continuous learning. Proactively find opportunities to use AI-assisted development techniques. These techniques help speed delivery, automate repetitive tasks, improve data engineering workflows, and boost operational efficiency.
• Communicate assumptions, seek clarification on tasks to minimize the need for rework, and actively participate in team meetings and discussions.
• Follow established project execution processes, coding standards, and industry guidelines.
• Get actively involved in training, self-development, and exchanging expertise, contributing actively to organization-level cloud initiatives.
• Solve problems with analytical thinking and apply root cause analysis.
• Easily switch between working autonomously or in cooperation with others.

Abilities

• proven experience as a data engineer with a demonstrated history of success.
• Proficiency in Python, SQL, and working with relational databases, along with hands-on PySpark.
• Experience building and optimizing data pipelines.
• Experience working with CI/CD Pipelines in GitHub.
• Working understanding of Feature Engineering, with the ability to contribute features that meet the team’s standards for reuse, documentation, and lineage.
• Excellent communication and collaboration skills.

Qualifications

• Graduate - Bachelor’s degree or Master’s degree (B.E. / B.Tech / M.S. / M.Tech / MCA) in Computer Science, Technology, Engineering, Mathematics, or a related subject area.

Additional Information

Must-Have Skills

• Significant experience working with AWS services such as Glue, S3, Lambda, EventBridge, IAM, and Athena.
• Skilled in developing and maintaining automated build and deployment processes on GitHub, along with Snowflake functionalities such as streams, tasks, stored procedures, snowpipe, and Schemachange.
• Must have - Python, Snowflake, AWS Glue, PySpark.

Good-to-Have Skills

• Development and performance optimization of AWS Glue PySpark jobs, managing distributed compute resources and worker nodes to process high-volume datasets efficiently.
• Familiarity with data science development practices, including AWS SageMaker and model deployment and monitoring approaches, without direct ownership of deployment.
• Exposure to enterprise-grade feature stores, or to third-party feature store solutions on cloud platforms.
• Familiarity with Model Lifecycle Management practices and positioning data for training / inference pipelines.
• Infrastructure as code (AWS CDK) and deployment automation (GitHub Actions).
• Domain background in retirement, annuities, wealth, or insurance.
• Nice to have AWS Cloud or SnowPro certification.