Regular Data Engineer

Luxoft; Bengaluru; On-site; INR 1,200,000 - 1,600,000; Full time

Job summary

Luxoft is seeking a Data Engineer for a Banking/Financial Services data platform transformation on AWS. Design, build, and optimize scalable data pipelines using Spark, Scala, Snowflake, and Hadoop.

Collaborate with analytics and business teams to deliver secure, high‑performing data solutions supporting regulatory, risk, and fraud use cases. Key tech includes Spark, Scala, Hadoop (HDFS/Hive), Snowflake, SQL, and AWS data services.

Qualifications

• Overall 5-8 years of total experience and 3-5 years of hands‑on experience in Data Engineering.
• Strong hands‑on experience with Spark and Scala for ETL development and data pipeline implementation.
• Experience working with the Hadoop ecosystem, including HDFS and Hive.
• Strong expertise in SQL, including query optimization, performance tuning, and handling large-volume datasets.
• Proven experience in building and maintaining large-scale data pipelines and batch processing solutions.
• Hands‑on experience with Snowflake for data warehousing, data integration, and performance optimization.
• Exposure to AI/ML‑driven data solutions on AWS cloud platforms and integration of data pipelines with analytics and machine learning workloads.
• Strong end‑to‑end Banking domain knowledge combined with extensive Data Engineering experience in large-scale enterprise environments.
• Hands‑on experience with AWS data services such as S3, Glue, Athena, and Redshift.
• Experience with GitHub, GitHub Actions, and TeamCity (or similar CI/CD tools).
• Proficiency in Shell scripting.
• Experience in production support, including incident management and root cause analysis.
• Strong understanding of clean coding standards, secure coding practices, and code review processes.
• Strong problem‑solving and communication skills.

Responsibilities

• Design, develop, and maintain scalable ETL processes and large-scale data pipelines using Spark and Scala.
• Build and support batch data processing solutions across enterprise data platforms.
• Develop and optimize data solutions using Snowflake for warehousing, integration, and performance tuning.
• Work with the Hadoop ecosystem (HDFS, Hive) for data storage and processing.
• Design and optimize SQL queries for high-volume datasets and ensure efficient data retrieval.
• Develop and manage data solutions on AWS using services such as S3, Glue, Athena, and Redshift.
• Integrate data pipelines with AI/ML and analytics workloads on AWS.
• Collaborate with business and technology teams to deliver end-to-end data engineering solutions within the Banking domain.
• Implement and support CI/CD pipelines using GitHub, GitHub Actions, TeamCity, or similar tools.
• Develop automation scripts using Shell Scripting.
• Provide production support, including incident resolution, troubleshooting, and root cause analysis.
• Follow clean coding standards, secure coding practices, and code review processes.
• Participate in performance optimization, release deployments, and ongoing platform improvements.
• Communicate effectively with stakeholders and contribute to problem‑solving across the data engineering lifecycle.

Skills

Spark

Scala

Hadoop

HDFS

Hive

SQL

Snowflake

AWS

GitHub

GitHub Actions

TeamCity

Shell scripting

Production support

Data pipelines

Banking domain

Tools

Snowflake

Hadoop

Job description

We are seeking a highly skilled Data Engineer to join a large-scale Banking and Financial Services transformation program focused on building enterprise-grade data platforms, analytics solutions, and AI/ML-enabled data products on AWS. The role involves designing, developing, and supporting high-performance data pipelines that process and integrate large volumes of structured and unstructured data across multiple banking systems.

You need to have strong hands‑on experience with Spark, Scala, Hadoop (HDFS/Hive), Snowflake, SQL, and AWS data services, along with proven expertise in building scalable ETL frameworks and batch processing solutions. The role requires close collaboration with business, analytics, and technology teams to deliver reliable, secure, and optimized data solutions that support reporting, regulatory, risk, fraud, and advanced analytics use cases.

Key responsibilities include developing and maintaining end-to-end data pipelines, optimizing data warehouse performance in Snowflake, integrating data platforms with AI/ML workloads on AWS, supporting production environments, performing root cause analysis, and driving CI/CD and automation best practices using tools such as GitHub, GitHub Actions, and TeamCity. Experience in Banking, Financial Crime, AML, or Fraud domains will be highly valued.

This is an excellent opportunity to work on cutting-edge cloud-based data engineering initiatives within a complex enterprise banking ecosystem while contributing to data-driven business outcomes and innovation.

Responsibilities:

• Design, develop, and maintain scalable ETL processes and large-scale data pipelines using Spark and Scala.
• Build and support batch data processing solutions across enterprise data platforms.
• Develop and optimize data solutions using Snowflake for warehousing, integration, and performance tuning.
• Work with the Hadoop ecosystem (HDFS, Hive) for data storage and processing.
• Design and optimize SQL queries for high-volume datasets and ensure efficient data retrieval.
• Develop and manage data solutions on AWS using services such as S3, Glue, Athena, and Redshift.
• Integrate data pipelines with AI/ML and analytics workloads on AWS.
• Collaborate with business and technology teams to deliver end-to-end data engineering solutions within the Banking domain.
• Implement and support CI/CD pipelines using GitHub, GitHub Actions, TeamCity, or similar tools.
• Develop automation scripts using Shell Scripting.
• Provide production support, including incident resolution, troubleshooting, and root cause analysis.
• Follow clean coding standards, secure coding practices, and code review processes.
• Participate in performance optimization, release deployments, and ongoing platform improvements.
• Communicate effectively with stakeholders and contribute to problem‑solving across the data engineering lifecycle.

Mandatory Skills Description:

• Overall 5-8 years of total experience and 3-5 years of hands‑on experience in Data Engineering.
• Strong hands‑on experience with Spark and Scala for ETL development and data pipeline implementation.
• Experience working with the Hadoop ecosystem, including HDFS and Hive.
• Strong expertise in SQL, including query optimization, performance tuning, and handling large-volume datasets.
• Proven experience in building and maintaining large-scale data pipelines and batch processing solutions.
• Hands‑on experience with Snowflake for data warehousing, data integration, and performance optimization.
• Exposure to AI/ML‑driven data solutions on AWS cloud platforms and integration of data pipelines with analytics and machine learning workloads.
• Strong end‑to‑end Banking domain knowledge combined with extensive Data Engineering experience in large-scale enterprise environments.
• Hands‑on experience with AWS data services such as S3, Glue, Athena, and Redshift.
• Experience with GitHub, GitHub Actions, and TeamCity (or similar CI/CD tools).
• Proficiency in Shell scripting.
• Experience in production support, including incident management and root cause analysis.
• Strong understanding of clean coding standards, secure coding practices, and code review processes.
• Strong problem‑solving and communication skills.

Nice-to-Have Skills Description:

• Experience in Banking, Financial Crime, AML (Anti‑Money Laundering), or Fraud domains.
• Experience with Oracle Database and Oracle SQL Developer.
• Exposure to AutoSys or other scheduling/orchestration tools such as Apache Airflow.
• Experience in release management and production deployments.
• Exposure to AI/ML use cases and integration of data pipelines with machine learning workloads.
• Experience working in large‑scale enterprise cloud transformation programs.
• Knowledge of DevOps and CI/CD best practices beyond GitHub Actions and TeamCity.