Data Engineer

Straive; Bengaluru Urban; On-site; INR 1,500,000 - 2,000,000; Full time

Job summary

Straive is seeking an experienced Data Engineer for a critical role in bridging high-performance big data processing on AWS with elite data warehousing in Snowflake. The ideal candidate should possess 5-6 years of experience in data engineering, proficient in AWS EMR and Snowflake, and able to develop robust data pipelines. Key responsibilities include leveraging AWS for data processing, implementing data validation frameworks, and managing Snowflake environments. This is an excellent opportunity to work at a data-driven company in a dynamic environment.

Qualifications

• 5-6 Years of experience in data engineering roles.
• Expert in Snowflake-specific SQL and performance tuning.
• Advanced Python scripting for automation.

Responsibilities

• Leverage AWS EMR for data processing and Athena for querying.
• Implement Python-based data validation libraries.
• Architect and maintain end-to-end Snowflake environments.
• Build and maintain Snowflake integrations with API services.
• Develop robust ETL pipelines for data optimization.

Skills

Snowflake-specific SQL extensions

Advanced Python scripting

Data quality testing

AWS EMR

Snowflake Data Engineering features

Hands-on with SnowPipe

Tools

AWS EMR

Snowflake

Boto3

Job description

Experience Level: 5–6 Years

Role Overview

This role is for a data powerhouse who can bridge the gap between high-performance big data processing on AWS and elite-level data warehousing in Snowflake. You will be responsible for the entire lifecycle of data, ensuring that every pipeline includes robust validation and observability.

Key Responsibilities

• Big Data Processing: Leverage AWS EMR (Spark/Hive) for heavy-lift data processing and Athena for serverless ad-hoc querying of S3 data lakes.
• Data Validation: Implement Great Expectations (GX) or equivalent Python-based validation libraries to prevent "data silent failure" within EMR and Snowflake ingestion layers.
• Snowflake Engineering: Architect end-to-end Snowflake environments, utilizing SnowSQL for scripting and SnowPipe for continuous, automated data ingestion.
• System Integration: Build and maintain Snowflake Integrations (Storage Integrations with S3, API Integrations, and Notification Services).
• Advanced ETL: Develop robust pipelines that transition data from raw AWS storage into optimized Snowflake schemas.

Technical Requirements

• SQL: Expert mastery of Snowflake-specific SQL extensions, CTEs, and performance tuning (Micro-partitioning).
• Python: Advanced scripting for automation, Boto3 for AWS resource management, and data manipulation.
• Data Quality: Proven experience implementing automated data testing and validation frameworks within production pipelines.
• AWS Stack: Extensive experience with EMR, Redshift, Athena, S3, and IAM security policies.
• Snowflake Mastery: Proven experience with Snowflake Data Engineering features (Streams, Tasks, and External Tables).
• Hands-on usage of SnowPipe for real-time or near-real-time data loading.