Senior AWS Data Engineer

Bounteous; Gurugram District; Hybrid; INR 1,800,000 - 2,400,000; Full time

Job summary

Bounteous is seeking a data platform engineer to build scalable data pipelines, ingest data, and analyze large datasets on a bespoke data platform. You will leverage Python, PySpark, and SQL to implement efficient processing and modeling.

This role involves working with AWS (EMR, S3), Delta/Parquet storage, and Spark jobs, with a focus on robust CI/CD and data quality practices in a collaborative setting.

Qualifications

• Experience building end-to-end data pipelines.
• Strong Python and SQL skills.
• Hands-on PySpark experience.
• AWS (EMR, S3) exposure.
• Airflow for orchestration and scheduling.
• Knowledge of data platform engineering principles.

Responsibilities

• Build and maintain scalable data pipelines across stages such as data download, ingestion, and analysis within a bespoke data platform.
• Develop high-performance data processing logic using Python and PySpark.
• Perform large-scale transaction data analysis using custom algorithms and in-house logic to detect financial crime patterns.
• Work with high-volume datasets and optimize pipeline performance.
• Design efficient data models and transformations for large-scale processing.
• Write optimized SQL queries on PostgreSQL (RDS), leveraging: Window functions / Partitioning /Query performance optimization
• Languages: Python, SQL
• AWS exposure (especially EMR and S3)
• Work with Delta tables and Parquet-based storage formats for efficient data processing.
• Build and maintain Spark batch and streaming jobs.
• Implement engineering best practices including unit testing, static code analysis, and CI/CD practices.
• Contribute to data platform architecture and system design decisions.

Skills

Python

PySpark

SQL

Data pipelines

Airflow

AWS

PostgreSQL

Delta/Parquet

Spark/Distributed processing

Code quality/CI-CD

Tools

Apache Airflow

Delta Tables

Parquet

PostgreSQL

Job description

Key Responsibilities

• Build and maintain scalable data pipelines across stages such as data download, ingestion, and analysis within a bespoke data platform.
• Develop high-performance data processing logic using Python and PySpark.
• Perform large-scale transaction data analysis using custom algorithms and in-house logic to detect financial crime patterns.
• Work with high-volume datasets and optimize pipeline performance.
• Design efficient data models and transformations for large-scale processing.
• Write optimized SQL queries on PostgreSQL (RDS), leveraging: Window functions / Partitioning /Query performance optimization
• Languages: Python, SQL
• AWS exposure (especially EMR and S3)
• Work with Delta tables and Parquet-based storage formats for efficient data processing.
• Build and maintain Spark batch and streaming jobs.
• Implement engineering best practices including unit testing, static code analysis, and CI/CD practices.
• Contribute to data platform architecture and system design decisions.

Required Skills

• Strong Python programming expertise
• Deep understanding of: Functional programming/ Object-Oriented Programming (OOP)/ Design patterns/ Python execution and invocation mechanisms
• Hands-on experience with Apache Spark / PySpark
• Strong SQL expertise including: Window functions/ Partitioning/ Query optimization
• Experience building end-to-end data pipelines
• Data Platform Engineering
• Experience designing scalable data processing architectures
• Strong understanding of distributed data processing
• Familiarity with standard data pipeline engineering practices
• AWS exposure (especially EMR and S3)
• Experience with Apache Airflow for job orchestration and scheduling