Data Engineer - Bangalore

Quess; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Tredence is seeking a Data Engineer to design, build, and scale high-performance data platforms and pipelines in Bengaluru. You will lead real-time and batch processing using Kafka, Spark, and Scala, while mentoring engineers and driving best practices across CI/CD and data quality frameworks.

The role emphasizes distributed systems, big data architectures, security, and collaboration with data scientists and stakeholders to deliver data-driven solutions.

Qualifications

• Apache Kafka (streaming, producers/consumers, topic management)
• Apache Spark (batch & streaming)
• Scala (advanced programming expertise)
• Solid understanding of data engineering concepts, distributed systems, and big data architectures
• Experience working with large-scale data processing and pipeline development
• Excellent comm skills
• Strong problem-solving and leadership skills

Responsibilities

• Lead the design and development of scalable, fault-tolerant data pipelines and streaming architectures.
• Build and maintain real-time and batch data processing systems using Kafka, Spark, and Scala.
• Architect large-scale data solutions ensuring performance, reliability, and security.
• Collaborate with data scientists, analysts, and business stakeholders to deliver data-driven solutions.
• Mentor and guide junior engineers; drive best practices in coding, design, and deployment.
• Optimize data workflows and improve system efficiency and scalability.
• Implement monitoring, logging, and alerting mechanisms for data platforms.
• Drive adoption of modern data engineering practices (CI/CD, data quality frameworks, etc.).

Skills

Kafka

Spark

Scala

Distributed systems

Big data architectures

Data processing

Communication skills

Leadership

Tools

Docker

Kubernetes

Job description

Roles and responsibilities - Tredence is looking for a Data Engineer to design, build, and scale high-performance data platforms and pipelines. The ideal candidate will have strong expertise in Spark, Scala + GCP cloud, with experience leading engineering teams and delivering enterprise-grade data solutions. Exposure to any cloud platform is a plus.

Key Responsibilities

• Lead the design and development of scalable, fault-tolerant data pipelines and streaming architectures.
• Build and maintain real-time and batch data processing systems using Kafka, Spark, and Scala.
• Architect large-scale data solutions ensuring performance, reliability, and security.
• Collaborate with data scientists, analysts, and business stakeholders to deliver data-driven solutions.
• Mentor and guide junior engineers; drive best practices in coding, design, and deployment.
• Optimize data workflows and improve system efficiency and scalability.
• Implement monitoring, logging, and alerting mechanisms for data platforms.
• Drive adoption of modern data engineering practices (CI/CD, data quality frameworks, etc.).

Mandatory Skills

• Apache Kafka (streaming, producers/consumers, topic management)
• Apache Spark (batch & streaming)
• Scala (advanced programming expertise)
• Solid understanding of data engineering concepts, distributed systems, and big data architectures
• Experience working with large-scale data processing and pipeline development
• Excellent comm skills
• Strong problem-solving and leadership skills

Good to Have (Optional Skills)

• Experience with any cloud platform: AWS / Azure / GCP
• Knowledge of data lakes, lakehouse architectures (Delta Lake, Iceberg, etc.)
• Familiarity with containerization tools (Docker, Kubernetes)
• Experience with orchestration tools (Airflow, etc.)