Senior/Lead Big Data Engineer with Hadoop,Hive,Spark,SQL-Bangalore

Tredence

Bengaluru

Hybrid

INR 2,500,000 - 3,600,000

Full time

10 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Tredence is seeking a Senior/Lead Big Data Engineer in Bengaluru (Hybrid) with 9-15 years of experience to architect, build, and optimize large-scale data platforms. You will own batch ETL/ELT pipelines, lead a data engineering team, and ensure data quality and platform reliability.

Responsibilities include end-to-end data workflows, collaboration with analytics and AI teams, and hands-on tuning of Spark, Hive, and Airflow-based pipelines. Prior e-commerce data experience is a plus.

Qualifications

  • BE/B.Tech in CS/IT or related engineering field.
  • 9–15 years of progressive Data Engineering experience with Hadoop ecosystems.
  • Proven leadership in engineering initiatives or team mentoring.
  • Expert in Apache Spark and Hive/HQL.
  • Strong Python and SQL with data modeling skills.
  • Experience designing production-grade batch ETL pipelines.
  • Hands-on with AI data initiatives and business understanding.
  • Proficient in Git for collaborative development.
  • Experience in e-commerce data and high-volume datasets is a plus.
  • Workflow orchestration using Airflow or Oozie; Agile/Scrum familiarity.

Responsibilities

  • Own end-to-end ETL/ELT pipelines on on-prem Hadoop platforms.
  • Architect, build, and optimize large-scale data platforms.
  • Lead data engineering team and mentor junior engineers.
  • Ensure data quality, lineage, and SLA adherence.
  • Collaborate with analytics, product, and AI teams.

Skills

Hadoop Ecosystem (HDFS, YARN)
Hive & HQL
Apache Spark (Core, Spark SQL, PySpark
SQL & Data Modeling
AI Data Engineering
Batch ETL/Data Pipeline Development

Education

BE/B.Tech in CS/IT

Tools

Airflow
Oozie
Git
Hadoop

Job description

Job Title: Senior / Lead Big Data Engineer
Experience: 9 - 15 Years
Location: Bangalore (Hybrid)
Mode of interview: 2 Virtual+1 in-person
# of roles: 2
Notice period: Immediate to 30 Days joiners only

About the Role

We are looking for a seasoned Senior/Lead Data Engineer with 9 to 15 years of experience to architect, build, and optimize large-scale data platforms. In this role, you will bridge the gap between complex big data infrastructure, functional business domains, and enterprise AI initiatives. Beyond taking end-to-end ownership of batch data pipelines that process massive customer and behavioral datasets, you will play a key leadership role in building, mentoring, and scaling a high-performing data engineering team.

Key Responsibilities
  • Platform Optimization & Governance
  • Performance Tuning: Proactively troubleshoot and fine-tune complex Spark and Hive/HQL jobs to reduce latency, optimize memory usage, and manage cluster resource allocation (YARN).
  • Data Quality & Reliability: Establish automated data validation frameworks, lineage tracking, and strict SLA monitoring to guarantee data accuracy and platform reliability.
  • Platform Operations: Oversee on-prem big data platform stability, capacity planning, and workflow orchestration using Airflow.
  • Team Building & Technical Leadership
  • Build & Scale the Team: Partner with leadership to hire, onboard, and build a strong data engineering team from the ground up.
  • Mentorship & Guidance: Provide hands-on technical leadership, conduct architecture and code reviews, and mentor junior and mid-level engineers.
  • Engineering Standards: Establish and enforce best practices around Git version control, code quality, documentation, and Agile delivery methodologies.
Architecture & Pipeline Development
  • End-to-End ETL Ownership: Architect, develop, and maintain scalable, fault-tolerant batch ETL/ELT pipelines across on-premises Hadoop-based data platforms.
  • Large-Scale Data Processing: Design high-throughput data ingestion and transformation workflows for massive customer and behavioral datasets.
  • Data Modeling: Lead dimensional and relational data modeling efforts to support enterprise analytics, reporting, and downstream AI solutions.
  • Functional Alignment & Cross-Team Collaboration
  • Business & Functional Translation: Translate complex functional business requirements into scalable technical data architectures, ensuring data assets directly solve business challenges.
  • Cross-Functional Partnership: Collaborate closely with analytics, product, and AI teams to understand data needs and deliver reliable, analytics-ready datasets.
Qualifications & Skills
  • Educational Qualification: B.E. or B.Tech in Computer Science, Information Technology, or a related engineering discipline.
  • Experience: 9 to 15 years of progressive experience in Data Engineering, with deep hands-on expertise in Hadoop-based big data ecosystems (HDFS, YARN).
  • Leadership: Proven experience leading engineering initiatives and building or mentoring technical teams.
  • Big Data Stack: Expert-level proficiency in Apache Spark (Core, Spark SQL, PySpark) and Apache Hive / HQL.
  • Programming & Querying: Strong command of Python and advanced SQL, paired with solid experience in enterprise data modeling.
  • Pipeline Engineering: Proven track record of designing and maintaining production-grade batch ETL pipelines.
  • AI & Functional Acumen: Practical experience working on data initiatives supporting AI applications, combined with strong functional business understanding.
  • Version Control: Proficient in Git for collaborative software development.
  • Domain Experience: Prior experience working with e-commerce platforms and processing high-volume customer/transactional datasets is an added advantage.
  • Workflow Orchestration: Hands-on experience managing complex DAGs and schedules using Apache Airflow or Oozie.
  • Infrastructure & Tuning: Deep understanding of on-premise big data platform operations and advanced performance tuning for Hive and Spark workloads.
  • Methodology: Familiarity with Agile/Scrum development methodologies.
Required Skills
Mandatory Skill Set:
  • Hadoop Ecosystem (HDFS, YARN)
  • Hive & HQL
  • Apache Spark (Core, Spark SQL, PySpark)
  • SQL & Data Modeling
  • AI Data Engineering
  • Batch ETL/Data Pipeline Development
Functional Acumen: Team Building & Leadership Preferred Domain expertise : Retail / E-commerce / Online Retailing / Online Shopping / Digital Commerce / E-tailing
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Big Data Developer
Big Data Developer

Viraaj HR Solutions • Maharashtra

On-site
INR 1,000,000 - 1,500,000
Collaborative engineering culture
Competitive compensation
Training in cloud and Big Data technologies
Senior Data Engineer (Spark, Scala) | 9 to 13 YoE | US-based large retail organisation | Bengaluru | Immediate/Early joiners
Senior Data Engineer (Spark, Scala) | 9 to 13 YoE | US-based large retail organisation | Bengaluru | Immediate/Early joiners

Geektrust • Bengaluru

On-site
INR 2,500,000 - 4,500,000
Big Data & Hadoop Developer
Big Data & Hadoop Developer

Tata Consultancy Services Limited • Mumbai

On-site
INR 1,200,000 - 2,000,000
Hiring Bigdata SME - Chennai/ Bangalore
Hiring Bigdata SME - Chennai/ Bangalore

Tech Mahindra Limited • Chennai, Bengaluru

Hybrid
INR 3,500,000 - 9,000,000
Big Data Engineer
Big Data Engineer

Global Infovision • Chennai, Bengaluru

Hybrid
INR 1,500,000 - 2,600,000
Big Data Engineer
Big Data Engineer

Infosys • Hyderabad, Pune District, Bengaluru

On-site
INR 1,200,000 - 2,000,000
Big Data Developer
Big Data Developer

Tata Consultancy Services Limited • Bengaluru

On-site
INR 1,800,000 - 2,400,000
Senior Data Engineer
Senior Data Engineer

Distilhire • Bengaluru

Hybrid
INR 2,500,000 - 4,000,000
Senior Data Engineer
Senior Data Engineer

Luxoft India • India

On-site
INR 1,500,000 - 2,000,000
Hiring Bigdata Engineers - Chennai/ Bangalore Location
Hiring Bigdata Engineers - Chennai/ Bangalore Location

Tech Mahindra Limited • Chennai, Bengaluru

Hybrid
INR 900,000 - 1,300,000