Lead Data Engineer – AWS

Jobtailor; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Jobtailor Bengaluru seeks an experienced Data Engineer to design scalable data pipelines using Spark/PySpark on Databricks and to optimize ETL/ELT workflows for large-scale processing.

You will work with AWS EMR/S3 and Hadoop, build data lakes/warehouses, and implement API data ingestion and orchestration with Airflow/Autosys. Strong Python and SQL skills are essential.

Qualifications

• Minimum 5+ years of Data Engineering experience.
• Strong hands-on experience in Databricks + Big Data stack.
• Proven experience working on large-scale distributed systems.
• Strong expertise in Apache Spark / PySpark.
• Databricks mandatory.
• SQL knowledge (PostgreSQL or similar).
• Hands-on with AWS EMR, S3; Hadoop/Hive ecosystem.
• Programming: Python mandatory; Scala a plus.
• UNIX / Shell scripting knowledge.
• ETL pipelines and data engineering fundamentals.
• Experience with Elasticsearch for storage/retrieval.
• Workflow orchestration: Airflow, Autosys.
• API development & integration exposure.
• Version control: Git / SVN.
• Basic HTML know-how for web/data tasks.

Responsibilities

• Design and build scalable data pipelines using Spark / PySpark on Databricks.
• Develop and optimize ETL/ELT workflows for large-scale data processing.
• Work with AWS EMR, S3, and Hadoop ecosystem for distributed data processing.
• Build and maintain data lake and data warehouse solutions.
• Develop and integrate APIs for data ingestion and consumption.
• Implement data processing workflows using Airflow / Autosys.
• Optimize data performance using partitioning, caching, and query tuning.
• Handle structured and semi-structured data from multiple sources.
• Ensure data quality, governance, and reliability of pipelines.
• Collaborate with cross-functional teams (Analytics, Product, Engineering).

Skills

Spark/PySpark

Databricks

Big Data

SQL

Python

UNIX/Shell

Airflow

Autosys

APIs

Git/SVN

Elasticsearch

Hadoop/Hive

Tools

AWS EMR

S3

Job description

Responsibilities

• Design and build scalable data pipelines using Spark / PySpark on Databricks
• Develop and optimize ETL/ELT workflows for large-scale data processing
• Work with AWS EMR, S3, and Hadoop ecosystem for distributed data processing
• Build and maintain data lake and data warehouse solutions
• Develop and integrate APIs for data ingestion and consumption
• Implement data processing workflows using Airflow / Autosys
• Optimize data performance using partitioning, caching, and query tuning
• Handle structured and semi-structured data from multiple sources
• Ensure data quality, governance, and reliability of pipelines
• Collaborate with cross-functional teams (Analytics, Product, Engineering)

Requirements

• Minimum 5+ years of Data Engineering experience
• Strong hands-on experience in Databricks + Big Data stack
• Proven experience working on large-scale distributed systems
• Strong expertise in Apache Spark / PySpark
• Databricks (mandatory)
• SQL (PostgreSQL or similar)
• Hands-on experience with AWS EMR, S3
• Hadoop, Hive ecosystem
• Programming skills: Python (must-have), Scala (good exposure)
• Strong knowledge of UNIX / Shell scripting
• ETL pipelines and data engineering fundamentals
• Experience with Elasticsearch (data storage & retrieval)
• Workflow orchestration tools (Airflow, Autosys)
• Exposure to API development & integration
• Version control tools (Git / SVN)
• Basic knowledge of HTML (for web/data tasks)