Big Data Engineer- Scala

Capco; Pune District; On-site; INR 1,200,000 - 2,500,000; Full time

Job summary

Capco is hiring a Big Data Engineer specialized in Scala to design and deliver scalable data pipelines for banking and financial services clients. You will work with Spark, Kafka, and Hadoop ecosystems to transform large data sets and enable advanced analytics.

You will collaborate with data engineers, analysts, and cross-functional teams to ensure reliable data delivery, strong data quality, and robust ETL processes. A client-focused, results-driven mindset is essential.

Qualifications

• Proficient in Scala programming with hands-on development experience.
• Experience with Apache Spark / Spark SQL.
• Strong SQL skills and data modeling ability.
• Experience building large-scale ETL/data pipelines.
• Exposure to Hadoop ecosystem (HDFS, Hive).
• Familiarity with Kafka or similar messaging systems.
• Understanding data formats like Parquet, Avro, JSON.
• Knowledge of performance tuning for Spark jobs.
• Awareness of SDLC and version control.

Responsibilities

• Design, develop, and maintain scalable Big Data applications and data pipelines.
• Develop high-performance data-processing solutions using Scala and Spark.
• Build and optimize ETL/ELT pipelines for large volumes of data.
• Work with Hadoop, HDFS, Hive, Kafka, and Spark SQL.
• Perform Spark job tuning and optimization.
• Develop reusable data-processing frameworks and tooling.
• Troubleshoot production data pipeline issues and errors.
• Implement data-quality checks, monitoring, and alerting.
• Collaborate with data architects, analysts, and business teams.
• Follow best practices in software engineering, including Git and CI/CD.

Skills

Scala

Apache Spark

Spark SQL

SQL

Hadoop

HDFS

Hive

Kafka

ETL

Data Engineering

Big Data

Distributed computing

Tools

Hadoop

HDFS

Hive

Kafka

Spark

Spark SQL

Parquet

Avro

JSON

Docker

Kubernetes

CI/CD

Airflow

Databricks

Delta Lake

Job description

MAKE AN IMPACT

Job Description: Big Data Engineer – Scala

Job Title: Big Data Engineer - Scala

About Us

“Capco, a Wipro company, is a global technology and management consulting firm. Awarded with Consultancy of the year in the British Bank Award and has been ranked Top 100 Best Companies for Women in India 2022 by Avtar & Seramount. With our presence across 32 cities across globe, we support 100+ clients across banking, financial and Energy sectors. We are recognized for our deep transformation execution and delivery.

WHY JOIN CAPCO?

You will work on engaging projects with the largest international and local banks, insurance companies, payment service providers and other key players in the industry. The projects that will transform the financial services industry.

MAKE AN IMPACT

Innovative thinking, delivery excellence and thought leadership to help our clients transform their business. Together with our clients and industry partners, we deliver disruptive work that is changing energy and financial services.

#BEYOURSELFATWORK

Capco has a tolerant, open culture that values diversity, inclusivity, and creativity.

CAREER ADVANCEMENT

With no forced hierarchy at Capco, everyone has the opportunity to grow as we grow, taking their career into their own hands.

DIVERSITY & INCLUSION

We believe that diversity of people and perspective gives us a competitive advantage.

MAKE AN IMPACT

Job Description: Big Data Engineer – Scala

Key Responsibilities

• Design, develop, and maintain scalable Big Data applications and data pipelines.
• Develop high-performance data-processing solutions using Scala and Apache Spark.
• Build and optimize ETL/ELT pipelines for large volumes of structured and unstructured data.
• Work with technologies such as Hadoop, HDFS, Hive, Kafka, and Spark SQL.
• Perform Spark job performance tuning and optimization.
• Develop reusable and maintainable data-processing frameworks.
• Troubleshoot production issues related to data pipelines and distributed applications.
• Implement data-quality checks, monitoring, and error‑handling mechanisms.
• Collaborate with data architects, analysts, business teams, and other engineering teams.
• Follow software engineering best practices, including Git, code reviews, testing, and CI/CD.

Required Skills

• Strong hands-on programming experience in Scala.
• Strong experience with Apache Spark / Spark SQL.
• Good understanding of Big Data and distributed computing concepts.
• Experience with Hadoop ecosystem technologies, including Hadoop/HDFS and Hive.
• Experience building large-scale ETL/data engineering pipelines.
• Strong SQL skills.
• Experience working with Kafka or similar messaging/streaming technologies.
• Understanding of data formats such as Parquet, Avro, and JSON.
• Experience with performance optimization and troubleshooting of Spark applications.
• Good understanding of software development lifecycle and version-control systems.

Good to Have

• Experience with AWS, Azure, or GCP.
• Knowledge of Databricks.
• Experience with Airflow or similar orchestration tools.
• Exposure to Delta Lake or other modern lakehouse technologies.
• Knowledge of Docker, Kubernetes, and CI/CD pipelines.
• Experience working in Agile/Scrum environments.

Core Skills

Scala | Apache Spark | Spark SQL | SQL | Hadoop | HDFS | Hive | Kafka | ETL | Data Engineering | Big Data