Data Engineer

Raad7 Informatics; Hyderabad; Hybrid; INR 1,500,000 - 2,800,000; Full time

Job summary

Raad7 Informatics is seeking an experienced Data Engineer to design, implement, and maintain end-to-end data pipelines. The role requires hands-on expertise with Databricks on Azure/AWS, PySpark, Python, SQL, and Apache Airflow to ensure reliable data flows and robust data quality.

The candidate will analyze existing pipelines, troubleshoot issues, and collaborate with cross-functional teams to deliver scalable data engineering solutions in production environments.

Qualifications

• Hands-on experience with Databricks on Azure or AWS.
• Proficient in PySpark and Python.
• Strong SQL skills with query writing and optimization.
• Experience with Apache Airflow for orchestration.
• Experience maintaining and enhancing existing data pipelines.
• Ability to analyze existing code and implement changes safely.
• Strong troubleshooting and problem-solving skills.

Responsibilities

• Understand and analyze existing data engineering pipelines, codebases, and business logic.
• Develop, maintain, and enhance data pipelines using Databricks, PySpark, Python, and SQL.
• Work with Apache Airflow for workflow orchestration, scheduling, and monitoring.
• Perform impact analysis before implementing code changes or enhancements.
• Troubleshoot pipeline failures, resolve data processing issues, and perform root cause analysis.
• Understand upstream and downstream dependencies to ensure smooth data flow.
• Optimize data processing jobs and SQL queries for performance and scalability.
• Perform data validation and ensure data quality and consistency.
• Support end-to-end pipeline development, maintenance, and production issues.
• Collaborate with cross-functional teams to deliver reliable data engineering solutions.

Skills

Databricks

PySpark

Python

SQL

Apache Airflow

Tools

Databricks on Azure/AWS

Airflow

Job description

Job Summary

We are looking for an experienced Data Engineer with strong hands-on expertise in Azure/AWS Databricks, PySpark, Python, SQL, and Apache Airflow. The candidate will be responsible for understanding existing data pipelines, analyzing business and technical logic, implementing enhancements, troubleshooting issues, and supporting end-to-end data engineering activities.

Key Responsibilities

• Understand and analyze existing data engineering pipelines, codebases, and business logic.
• Develop, maintain, and enhance data pipelines using Databricks, PySpark, Python, and SQL.
• Work with Apache Airflow for workflow orchestration, scheduling, and monitoring.
• Perform impact analysis before implementing code changes or enhancements.
• Troubleshoot pipeline failures, resolve data processing issues, and perform root cause analysis.
• Understand upstream and downstream dependencies to ensure smooth data flow.
• Optimize data processing jobs and SQL queries for performance and scalability.
• Perform data validation and ensure data quality and consistency.
• Support end-to-end pipeline development, maintenance, and production issues.
• Collaborate with cross-functional teams to deliver reliable data engineering solutions.

Mandatory Skills

• Strong hands-on experience with Databricks on Azure or AWS.
• Proficiency in PySpark and Python.
• Strong SQL skills, including query writing and optimization.
• Experience with Apache Airflow for pipeline orchestration.
• Experience maintaining and enhancing existing data pipelines.
• Ability to analyze existing code, understand dependencies, and implement changes safely.
• Strong troubleshooting and problem-solving skills.

Preferred Qualifications

• Experience with cloud-based data engineering solutions on Azure or AWS.
• Knowledge of ETL/ELT processes and data pipeline architecture.
• Experience in production support, performance tuning, and data quality validation.