Data Engineer / Data Analyst / Python Developer

Aetherrise Solutions; Bengaluru; Hybrid; INR 2,800,000 - 4,800,000; Full time

Benefits offered by this job

Hybrid work opportunities

Health insurance

Learning & certification support

Enterprise-scale AWS/Azure projects

Flexible work environment

Leaves & holidays

Recognition programs

Career growth

Work-life balance

Job summary

Aetherrise Solutions in Bengaluru, India, seeks a Senior Data Engineer to design, develop, and maintain scalable ETL/ELT data pipelines using Python, PySpark, SQL, AWS Glue, Azure Data Factory, and Apache Airflow. You will build cloud data platforms on AWS and Azure, develop ingestion pipelines from diverse sources, and optimize Spark jobs for performance and cost efficiency.

Collaborate with analysts and stakeholders to deliver trusted datasets, implement data quality and governance, and

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, Data Science, Engineering, or a related field.
• 38 years of experience in Data Engineering, Cloud Data Engineering, Data Analytics, or related domains.
• Strong programming experience in Python, SQL, and PySpark.
• Hands-on experience with AWS cloud data services, particularly AWS Glue, S3, Redshift, Lambda, EMR, Step Functions, and CloudWatch.
• Hands-on experience with Microsoft Azure data services, particularly Azure Data Factory, Azure Databricks, ADLS Gen2, Azure Synapse Analytics, and Azure Monitor.
• Strong understanding of ETL/ELT, data warehousing, data lakes, data lakehouse architecture, and big-data processing.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Python, PySpark, SQL, AWS Glue, Azure Data Factory, and Apache Airflow.
• Build and manage cloud-based data platforms across AWS and Microsoft Azure.
• Develop data ingestion pipelines from databases, APIs, files, streaming platforms, and other sources.
• Optimize Spark jobs, SQL queries, ETL workflows, and cloud resources for performance and cost efficiency.
• Collaborate with analysts and stakeholders to deliver scalable data solutions.

Skills

Python

SQL

PySpark

Apache Airflow

AWS

Azure

Spark

Databricks

Snowflake

ETL/ELT

Data modeling

Education

Bachelor's or Master's in Computer Science / IT / Data Science

Tools

AWS

Azure

Snowflake

Databricks

Apache Airflow

S3

ADLS

Job description

Role & Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Python, PySpark, SQL, AWS Glue, Azure Data Factory, and Apache Airflow.
• Build and manage cloud-based data platforms across AWS and Microsoft Azure, supporting enterprise data engineering and analytics requirements.
• Develop data ingestion pipelines to collect data from databases, APIs, files, streaming platforms, and other structured and semi-structured sources.
• Implement data processing and transformation solutions using PySpark, Apache Spark, Python, and SQL.
• Develop and optimize data lakes and data warehouses using Amazon S3, Amazon Redshift, Snowflake, Azure Data Lake Storage (ADLS), and Azure Synapse Analytics.
• Work with AWS services including AWS Glue, Lambda, EMR, Athena, S3, Redshift, Step Functions, SQS, SNS, and CloudWatch.
• Work with Azure services including Azure Data Factory, Azure Databricks, ADLS Gen2, Azure Synapse Analytics, Azure Functions, and Azure Monitor.
• Design and implement scalable data models, including fact tables, dimension tables, star schemas, and analytical data models.
• Develop complex SQL queries, stored procedures, transformations, and data integration processes for reporting and analytics.
• Implement data quality, validation, reconciliation, and monitoring frameworks to ensure data accuracy, completeness, and reliability.
• Optimize Spark jobs, SQL queries, ETL workflows, and cloud resources for improved performance and cost efficiency.
• Develop Python-based automation scripts, data processing frameworks, utilities, and integrations.
• Configure and maintain workflow orchestration using Apache Airflow for reliable scheduling, dependency management, and pipeline monitoring.
• Collaborate with business analysts, data architects, developers, and stakeholders to understand requirements and deliver scalable data solutions.
• Support analytics, reporting, and BI initiatives by providing trusted, curated, and high-quality datasets.
• Implement CI/CD and version-control practices using Git, GitHub/Azure DevOps, and cloud-native deployment processes.
• Troubleshoot pipeline failures, data issues, performance bottlenecks, and production incidents.
• Follow best practices for coding, testing, documentation, security, governance, and Agile/Scrum development.

Preferred Candidate Profile

• Bachelor's or Master's degree in Computer Science, Information Technology, Data Science, Engineering, or a related field.
• 38 years of experience in Data Engineering, Cloud Data Engineering, Data Analytics, or related domains.
• Strong programming experience in Python, SQL, and PySpark.
• Hands-on experience with AWS cloud data services, particularly AWS Glue, S3, Athena, Redshift, Lambda, EMR, Step Functions, and CloudWatch.
• Hands-on experience with Microsoft Azure data services, particularly Azure Data Factory, Azure Databricks, ADLS Gen2, Azure Synapse Analytics, and Azure Monitor.
• Strong understanding of ETL/ELT, data warehousing, data lakes, data lakehouse architecture, and big-data processing.
• Experience with Apache Spark and PySpark for large-scale distributed data processing.
• Experience with Apache Airflow for data pipeline orchestration and workflow automation.
• Knowledge of cloud data warehouses such as Snowflake, Amazon Redshift, and Azure Synapse Analytics.
• Experience with Databricks, Delta Lake, and Medallion Architecture is preferred.
• Strong understanding of data modeling, dimensional modeling, partitioning, file formats, and performance optimization.
• Familiarity with Git, CI/CD, DevOps practices, Agile/Scrum, and cloud security principles.
• Experience working with Parquet, JSON, CSV, Avro, and other structured/semi-structured data formats.
• Strong analytical, troubleshooting, problem-solving, and communication skills.
• Ability to work independently and collaboratively in a fast-paced, enterprise environment.

Perks & Benefits

• Hybrid and remote work opportunities.
• Health insurance and employee wellness benefits.
• Learning and certification support for AWS, Azure, Snowflake, Databricks, and other cloud technologies.
• Opportunities to work on enterprise-scale AWS and Azure data platforms.
• Exposure to modern technologies in cloud computing, big data, analytics, and data engineering.
• Career growth and continuous learning opportunities.
• Flexible work environment and employee-friendly policies.
• Employee recognition and rewards programs.
• Paid leaves, holidays, and work-life balance initiatives.
• Opportunity to work on challenging Data Engineering, Cloud Migration, Data Lake, Lakehouse, and Analytics projects.