Senior Data Engineer

IntraEdge, Inc.; Pune District; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

IntraEdge, Inc. is seeking an experienced Data Engineer in Pune, India, to design, develop, and maintain scalable data pipelines using Python, PySpark, AWS Glue, and Snowflake. The role involves data ingestion, transformation, and data modeling with Snowflake, ensuring quality and security across platforms.

You will automate workflows with AWS Lambda and S3, monitor pipelines, and collaborate with analytics teams to meet reporting needs, while upholding best practices and documentation.

Qualifications

• 9 years of experience in Data Engineering or a related role.
• Strong hands-on experience with Python and PySpark.
• Expertise in AWS services, particularly Glue, Lambda, S3, CloudWatch, and IAM.
• Strong experience with Snowflake, including data modeling, warehouse design, and query optimization.
• Excellent knowledge of SQL, ETL/ELT processes, data warehousing, and large-scale data processing.
• Experience with Git, CI/CD pipelines, and deployment practices.
• Understanding of data governance, security, and compliance standards.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Python, PySpark, and AWS Glue.
• Build data ingestion and transformation solutions for structured and unstructured data.
• Work extensively with Snowflake for data modeling, warehouse management, performance tuning, and query optimization.
• Automate data workflows using AWS Lambda, S3, and other AWS services.
• Ensure data quality, consistency, security, and reliability across data platforms.
• Monitor, troubleshoot, and optimize data pipelines for performance and scalability.
• Collaborate with cross-functional teams to support analytics and reporting requirements.
• Maintain technical documentation and follow data engineering best practices.

Skills

Python

PySpark

AWS Glue

AWS Lambda

S3

Snowflake

SQL

ETL/ELT

Git

CI/CD

data governance

security/compliance

Tools

DBT

Job description

We are looking for an experienced Data Engineer with strong expertise in Python, PySpark, AWS services, and Snowflake to design, develop, and maintain scalable data pipelines and robust data integration solutions.

Key Responsibilities

• Design, develop, and optimize ETL/ELT pipelines using Python, PySpark, and AWS Glue.
• Build data ingestion and transformation solutions for structured and unstructured data.
• Work extensively with Snowflake for data modeling, warehouse management, performance tuning, and query optimization.
• Automate data workflows using AWS Lambda, S3, and other AWS services.
• Ensure data quality, consistency, security, and reliability across data platforms.
• Monitor, troubleshoot, and optimize data pipelines for performance and scalability.
• Collaborate with cross-functional teams to support analytics and reporting requirements.
• Maintain technical documentation and follow data engineering best practices.

Required Skills & Qualifications

• 9 years of experience in Data Engineering or a related role.
• Strong hands-on experience with Python and PySpark.
• Expertise in AWS services, particularly Glue, Lambda, S3, CloudWatch, and IAM.
• Strong experience with Snowflake, including data modeling, warehouse design, and query optimization.
• Excellent knowledge of SQL, ETL/ELT processes, data warehousing, and large-scale data processing.
• Experience with Git, CI/CD pipelines, and deployment practices.
• Understanding of data governance, security, and compliance standards.

Must-Have Skills

Python, PySpark, AWS Glue, AWS Lambda, S3, Snowflake, SQL, ETL/ELT.

Good-to-Have Skills

• Hands-on experience with DBT (Data Build Tool) for data transformation, data modeling, testing, and documentation.
• Familiarity with DBT integration with Snowflake and modern data warehouse solutions.