Data Engineer (Python, SQL, PySpark & AWS) For Bangalore

Apptad; Bengaluru; Hybrid; INR 1,200,000 - 1,800,000; Full time

Job summary

Apptad is seeking a skilled Data Engineer to design, build, and maintain scalable data pipelines powering analytics, reporting, and ML initiatives. You will work extensively with PySpark for large-scale processing and leverage AWS to create cloud-native data infrastructure.

The role involves building data lakes, data warehouses on AWS, and orchestrating workflows with Airflow or Glue. You will optimize Spark jobs and collaborate with analysts, data scientists, and stakeholders to meet data

Qualifications

• Experience designing ETL/ELT pipelines using PySpark for batch and streaming data.
• Experience building data lake and data warehouse solutions on AWS (S3, Redshift, Glue, EMR, Athena).
• Knowledge of orchestrating workflows with Airflow or AWS Step Functions and Glue Workflows.
• Ability to optimize Spark jobs for performance, cost, and scalability.
• Experience ingesting data from APIs, databases, flat files, and streaming platforms like Kafka/Kinesis.

Responsibilities

• Design, develop, and maintain ETL/ELT pipelines using PySpark for batch and streaming data processing.
• Build and manage data lake and data warehouse solutions on AWS (S3, Redshift, Glue, EMR, Athena, Lake Formation).
• Develop and orchestrate workflows using AWS Step Functions, Apache Airflow, or AWS Glue Workflows.
• Optimize Spark jobs for performance, cost, and scalability (partitioning, caching, cluster tuning).
• Ingest data from multiple sources (APIs, databases, flat files, streaming platforms like Kafka/Kinesis).
• Implement data quality checks, validation frameworks, and monitoring/alerting for pipeline health.
• Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements.
• Design and maintain data models (star/snowflake schemas) for analytics use cases.
• Write clean, well-documented, testable code following engineering best practices (CI/CD, version control).
• Ensure data security, governance, and compliance (IAM policies, encryption, access controls).
• Troubleshoot and resolve production data pipeline issues

Skills

Python

PySpark

Spark

SQL

AWS

ETL

Data Pipelines

CI/CD

Data Quality

Data Security

Tools

Apache Airflow

Glue

Redshift

S3

EMR

Kafka

Kinesis

Job description

Skills- Data Engineer (Python, Spark, SQL,AWS)

We're looking for a skilled Data Engineer to design, build, and maintain scalable data pipelines that power analytics, reporting, and machine learning initiatives. You'll work extensively with PySparkfor large-scale data processing and the AWSecosystem to build reliable, cloud-native data infrastructure.

Key Responsibilities

• Design, develop, and maintain ETL/ELT pipelines using PySparkfor batch and streaming data processing
• Build and manage data lake and data warehouse solutions on AWS(S3, Redshift, Glue, EMR, Athena, Lake Formation)
• Develop and orchestrate workflows using AWS Step Functions, Apache Airflow, or AWS Glue Workflows
• Optimize Spark jobs for performance, cost, and scalability (partitioning, caching, cluster tuning)
• Ingest data from multiple sources (APIs, databases, flat files, streaming platforms like Kafka/Kinesis)
• Implement data quality checks, validation frameworks, and monitoring/alerting for pipeline health
• Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements
• Design and maintain data models (star/snowflake schemas) for analytics use cases
• Write clean, well-documented, testable code following engineering best practices (CI/CD, version control)
• Ensure data security, governance, and compliance (IAM policies, encryption, access controls)
• Troubleshoot and resolve production data pipeline issues