Data Engineer

Authorasist Hyderabad; Hyderabad, Bengaluru; Hybrid; INR 1,500,000 - 2,500,000; Full time

Job summary

Authorasist Hyderabad is seeking a Data Engineer to design, build, and maintain scalable ETL/ELT data pipelines and data-processing workflows. You will craft optimized SQL queries, automate data ingestion, and ensure data quality across diverse sources and cloud platforms.

You will work with Apache Airflow, Spark/PySpark, and cloud services (AWS/Azure/GCP), collaborating with analysts, scientists, and engineers to deliver robust data solutions and reliable analytics support.

Qualifications

• 4+ years of professional experience in Data Engineering.
• Strong proficiency in Python and SQL.
• Hands-on experience with ETL/ELT pipelines.
• Experience with Apache Spark / PySpark.
• Strong understanding of data warehousing and dimensional modeling.
• Experience with Apache Airflow or similar workflow orchestration tools.
• Hands-on experience with at least one major cloud platform: AWS, Azure, or GCP.
• Experience with cloud data services such as S3, Glue, Redshift, Databricks, BigQuery, Synapse, or equivalent.
• Good understanding of relational and non-relational databases.
• Experience with Git and CI/CD practices.
• Strong understanding of data quality, performance optimization, and pipeline monitoring.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines.
• Build reliable data ingestion and transformation workflows from multiple sources.
• Develop optimized and complex SQL queries for data processing and analytics.
• Work with large datasets and ensure data quality, accuracy, and consistency.
• Develop and maintain data workflows using tools such as Apache Airflow.
• Design and implement data solutions on cloud platforms such as AWS/Azure/GCP.
• Work with data warehouses such as Snowflake, BigQuery, Redshift, or Azure Synapse.
• Write clean, reusable, and efficient Python code for data processing and automation.
• Implement data validation, monitoring, logging, and error-handling mechanisms.
• Optimize data pipelines and queries for performance and cost.
• Collaborate with Data Analysts, Data Scientists, Software Engineers, and business stakeholders.
• Participate in code reviews, testing, deployment, and production support.
• Troubleshoot data pipeline failures and resolve data-related issues.

Skills

Data Engineering

Python

SQL

ETL/ELT pipelines

Apache Spark / PySpark

Data warehousing

Dimensional modeling

Apache Airflow

AWS

Azure

GCP

S3

Glue

Redshift

Databricks

BigQuery

Synapse

Git

CI/CD

Data quality

Performance optimization

Pipeline monitoring

Databricks and Delta Lake

Kafka

Docker/Kubernetes

dbt

Terraform / IaC

Agile/Scrum

Tools

Docker

Kubernetes

Job description

Hi

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines.
• Build reliable data ingestion and transformation workflows from multiple sources.
• Develop optimized and complex SQL queries for data processing and analytics.
• Work with large datasets and ensure data quality, accuracy, and consistency.
• Develop and maintain data workflows using tools such as Apache Airflow.
• Design and implement data solutions on cloud platforms such as AWS/Azure/GCP.
• Work with data warehouses such as Snowflake, BigQuery, Redshift, or Azure Synapse.
• Write clean, reusable, and efficient Python code for data processing and automation.
• Implement data validation, monitoring, logging, and error-handling mechanisms.
• Optimize data pipelines and queries for performance and cost.
• Collaborate with Data Analysts, Data Scientists, Software Engineers, and business stakeholders.
• Participate in code reviews, testing, deployment, and production support.
• Troubleshoot data pipeline failures and resolve data-related issues.

Required Skills

• 4+ years of professional experience in Data Engineering.
• Strong proficiency in Python and SQL.
• Hands-on experience with ETL/ELT pipelines.
• Experience with Apache Spark / PySpark.
• Strong understanding of data warehousing and dimensional modeling.
• Experience with Apache Airflow or similar workflow orchestration tools.
• Hands-on experience with at least one major cloud platform: AWS, Azure, or GCP.
• Experience with cloud data services such as S3, Glue, Redshift, Databricks, BigQuery, Synapse, or equivalent.
• Good understanding of relational and non-relational databases.
• Experience with Git and CI/CD practices.
• Strong understanding of data quality, performance optimization, and pipeline monitoring.

Good to Have

• Experience with Databricks and Delta Lake.
• Knowledge of Kafka or other streaming technologies.
• Experience with Docker/Kubernetes.
• Knowledge of dbt.
• Exposure to Terraform or Infrastructure as Code.
• Experience working in Agile/Scrum environments.