Senior Data Specialist

Jobtailor; Bengaluru; On-site; INR 1,500,000 - 2,600,000; Full time

Job summary

Jobtailor is seeking a Data Engineer in Bengaluru to design, build, and maintain data pipelines across cloud services and big data tech to support analytics and product needs.

You will design scalable pipelines with Spark / PySpark on Databricks, optimize ETL/ELT workflows, and work with AWS EMR, S3, Hadoop, Hive. Strong Python, SQL, and API skills are required.

Qualifications

• Minimum 5+ years of Data Engineering experience.
• Strong hands-on Databricks + Big Data stack experience.
• Proven experience with large-scale distributed systems.
• Expertise in Spark / PySpark and SQL (PostgreSQL or similar).
• Hands-on AWS EMR, S3, Hadoop, Hive ecosystem experience.
• Programming: Python; Scala is a plus.
• UNIX / Shell scripting and ETL fundamentals.
• Elasticsearch, Airflow, Autosys experience.
• APIs development & integration; version control with Git/SVN.
• HTML basics for web/data tasks.

Responsibilities

• Design and build scalable data pipelines using Spark / PySpark on Databricks.
• Develop and optimize ETL/ELT workflows for large-scale data processing.
• Work with AWS EMR, S3, and Hadoop ecosystem for distributed processing.
• Build and maintain data lake and data warehouse solutions.
• Develop and integrate APIs for data ingestion and consumption.
• Implement data processing workflows using Airflow / Autosys.
• Optimize data performance via partitioning, caching, and query tuning.
• Handle structured and semi-structured data from multiple sources.
• Ensure data quality, governance, and reliability of pipelines.
• Collaborate with Analytics, Product, Engineering teams.

Tools

Databricks

Apache Spark / PySpark

SQL (PostgreSQL or similar)

AWS EMR

S3

Hadoop

Hive

Airflow

Autosys

Elasticsearch

Git

SVN

API development

HTML basics

Python

Scala

UNIX / Shell scripting

Job description

The Data Engineer will design, build, and maintain data pipelines, working across cloud services and big data technologies to support analytics and product needs.

Responsibilities

• Design and build scalable data pipelines using Spark / PySpark on Databricks
• Develop and optimize ETL/ELT workflows for large-scale data processing
• Work with AWS EMR, S3, and Hadoop ecosystem for distributed data processing
• Build and maintain data lake and data warehouse solutions
• Develop and integrate APIs for data ingestion and consumption
• Implement data processing workflows using Airflow / Autosys
• Optimize data performance using partitioning, caching, and query tuning
• Handle structured and semi-structured data from multiple sources
• Ensure data quality, governance, and reliability of pipelines
• Collaborate with cross-functional teams (Analytics, Product, Engineering)

Requirements

• Minimum 5+ years of Data Engineering experience
• Strong hands‑on experience in Databricks + Big Data stack
• Proven experience working on large‑scale distributed systems
• Strong expertise in: Apache Spark / PySpark, Databricks (mandatory), SQL (PostgreSQL or similar)
• Hands‑on experience with: AWS EMR, S3, Hadoop, Hive ecosystem
• Programming skills: Python (must‑have), Scala (good exposure)
• Strong knowledge of: UNIX / Shell scripting, ETL pipelines and data engineering fundamentals
• Experience with: Elasticsearch (data storage & retrieval), Workflow orchestration tools (Airflow, Autosys)
• Exposure to: API development & integration, Version control tools (Git / SVN)
• Basic knowledge of HTML (for web/data tasks)