Senior Data Engineer

Jobtailor; Hyderabad; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

Jobtailor is seeking a data engineering professional in Hyderabad to design, develop, and maintain robust ETL/ELT pipelines using SQL, Python, and Spark.

You will build scalable data workflows with Apache Airflow, leverage AWS services like S3 and Redshift, and work with Databricks Delta Lake to deliver reliable data solutions. Strong collaboration with analytics and product teams is essential.

Qualifications

• Strong proficiency in SQL and Python.
• Hands-on experience with Spark (PySpark).
• Experience with Apache Airflow for orchestration.
• Cloud experience on AWS (Redshift, S3) and Databricks.
• Understanding of data warehousing and ETL design.
• Familiarity with CI/CD pipelines and GitHub Actions is a plus.
• Knowledge of Pharma / Life Sciences domain is nice to have.
• Familiarity with data governance and quality frameworks is desirable.

Responsibilities

• Design, develop, and maintain ETL/ELT pipelines using SQL, Python, and Spark.
• Build data workflows using Apache Airflow for orchestration.
• Develop scalable solutions using AWS services (S3, Redshift, Glue, EMR, Lambda).
• Implement and manage Databricks pipelines (Delta Lake, notebooks).
• Monitor, troubleshoot, and optimize production data pipelines.
• Collaborate with analytics and product teams to deliver data solutions.
• Ensure data quality and performance across pipelines.

Skills

SQL

Python

Spark (PySpark)

Data warehousing

ETL design

Tools

Apache Airflow

AWS (S3, Redshift)

Databricks

Docker

Kubernetes

GitHub Actions

Job description

Responsibilities

• Design, develop, and maintain ETL/ELT pipelines using SQL, Python, and Spark
• Build and manage data workflows using Apache Airflow for orchestration and scheduling
• Develop scalable and optimized solutions using AWS services (S3, Glue, Redshift, EMR, Lambda, etc.)
• Implement and manage data processing pipelines in Databricks (Delta Lake, notebooks, workflows, Unit Catalog)
• Ensure data quality, reliability, and performance across pipelines
• Collaborate with analytics, product, and business teams to deliver data solutions
• Monitor, troubleshoot, and optimize production pipelines

Requirements

• Strong proficiency in SQL and Python
• Hands‑on experience with Apache Spark (PySpark preferred)
• Experience working with Apache Airflow for workflow orchestration
• Solid experience with AWS cloud platform. Redshift performance optimization skills
• Hands‑on experience in Databricks
• Understanding of data warehousing, data modeling, and ETL design
• Experience with CI/CD pipelines and GitHub Actions (Good to Have)
• Knowledge of Pharmaceutical / Life Sciences domain (Good to Have)
• Familiarity with data governance and quality frameworks (Good to Have)
• Exposure to Docker, Kubernetes, or similar technologies (Good to Have)