Senior Data Engineer - AWS & Big Data

Commonwealth Bank of Australia; Bengaluru; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

Commonwealth Bank of Australia (CommBank) in Bangalore is seeking an experienced Data Engineer to build and operate scalable data platforms supporting security analytics and AI workflows. You will own end‑to‑end data pipelines, work with AWS services, Python, and DevSecOps practices, collaborating with data scientists and software engineers to productionise models.

The role requires 5–8 years of data engineering experience, strong AWS/PySpark skills, and a pragmatic approach to delivering

Qualifications

• 5–8 years of Data Engineering experience in a data‑intensive environment.
• Proficiency in AWS cloud services (EC2, S3, Lambda, Athena, Kinesis, Redshift, Glue, EMR, DynamoDB, IAM).
• Excellent Python‑based framework development skills and experience with PySpark/Spark.
• Experience with distributed data processing and large‑scale data platforms.
• Proficient in SQL for complex queries and performance optimization.
• Experience with Terraform and CI/CD/DevOps practices; Docker/Kubernetes considered.

Responsibilities

• Design, build, and maintain scalable data pipelines (batch and real‑time) for AI/ML and security use cases.
• Develop and manage data models and data products to enable analytics and detection capabilities.
• Ensure data quality, reliability, and observability across pipelines and platforms.
• Collaborate with data scientists and software engineers to productionise AI/ML models.
• Lead end‑to‑end data engineering lifecycle from requirements to deployment in a DevSecOps process.

Skills

AWS

Python

Data pipelines

Big Data

SQL

Terraform

Docker

Kubernetes

PySpark

Security

Education

Bachelor's/Master's in CS/IT

Tools

Kafka

Redshift

Snowflake

DBT

Glue

S3

EC2

IAM

Job description

Data Engineer Organization: At CommBank, we never lose sight of the role we play in other people’s financial wellbeing. Our focus is to help people and businesses move forward to progress. To make the right financial decisions and achieve their dreams, targets, and aspirations. Regardless of where you work within our organisation, your initiative, talent, ideas, and energy all contribute to the impact that we can make with our work. Together we can achieve great things.

Job Title: Data Engineer Location: Bangalore

Business & Team: Technology Team is responsible for the world leading application of technology and operations across every aspect of CommBank, from innovative product platforms for our customers to essential tools within our business. We also use technology to drive efficient and timely processing, an essential component of great customer service.

Impact & contribution:- To ensure the Group achieves a sustainable competitive advantage through data engineering, you will play a key role in supporting and executing the Group's data strategy. We are looking for an experienced Data Engineer to join our Group Security Team, which is part of the wider Cyber Security Engineering practice. In this role, you will be responsible for setting up the Group Security Data Platform to ingest data from various organizations' security telemetry data, along with additional data assets and data products. This platform will provide

Roles & Responsibilities

• You will be expected to perform the following tasks in a manner consistent with CBA’s Values and People Capabilities.
• Possesses hands‑on technical experience working in AWS. The individual should have knowledge about AWS services like EC2, S3, Lambda, Athena, Kinesis, Redshift, Glue, EMR, DynamoDB, IAM, SecretManager, KMS, Step functions, SQS, SNS, Cloud Watch.
• The individual should possess a robust set of technical and soft skills and be an excellent AWS Data Engineer with a focus on complex Automation and Engineering Framework development.
• Being well‑versed in Python is mandatory, and experience in developing complex frameworks using Python is required.
• Passionate about Cloud/DevSecOps/Automation and possess a keen interest in solving complex problems systematically.
• Drive the development and implementation of scalable data solutions and data pipelines using various AWS services.
• Possess the ability to work independently and collaborate closely with team members and technology leads.
• Exhibit a proactive approach, constantly seeking innovative solutions to complex technical challenges.
• Can take responsibility for nominated technical assets related to areas of expertise, including roadmaps and technical direction.
• Can own and develop technical strategy, overseeing medium to complex engineering initiatives.
• Design, build, and maintain scalable data pipelines (batch and real‑time) to support AI/ML and security use cases.
• Develop and manage data models and data products that enable analytics and detection capabilities.
• Ensure data quality, reliability, and observability across pipelines and platforms.
• Collaborate closely with data scientists and software engineers to productionise AI models.
• Integrate data solutions into enterprise platforms (e.g. ServiceNow SecOps and other security tooling).
• Contribute to DevOps practices including CI/CD, automation, and infrastructure‑as‑code.

Essential Skills

• About 5‑8 years of experience as a Data Engineering professional in a data‑intensive environment. The individual should have strong analytical and reasoning skills in the relevant area.
• Proficiency in AWS cloud services, specifically EC2, S3, Lambda, Athena, Kinesis, Kafka, Redshift, Snowflake, DBT, Glue, EMR, DynamoDB, IAM, SecretManager, Step functions, SQS, SNS, Cloud Watch.
• Excellent skills in Python‑based framework development are mandatory.
• Experience with distributed data processing and large‑scale data platforms.
• Proficiency in SQL for efficient querying, managing databases, handling complex queries, and optimizing query performance.
• Experience with Terraform, Jenkins, Terraform is essential as part of DevOps.
• Additionally, Docker and Kubernetes are also considered.
• Strong understanding of streaming data processing concepts, including event‑driven architectures.
• Expertise in data pipeline development using various data formats/types.
• Mandatory knowledge and experience in big data processing using PySpark/Spark and performance optimizations of applications.
• Hands on with various Gen AI tools like Claude, GH Copilot, VS Code Co‑pilot/Claude, Cline, Roo, ChatGpt.
• Own and lead E2E Data engineering life cycle right from Requirement gathering, Data Modelling & architect design, develop & Build, test, Deployment/DevOPS Practices and support as part of DevSecOPS process.
• Must demonstrate skills and mindset to implement encryption methodologies like SSL/TLS and data encryption at rest and in transit and other data security best practices.

Education Qualification

• A Bachelor's or Master's degree in Engineering, specializing in Computer Science, Information Technology or relevant qualifications.

We’re keen to support you with the next step in your career. We’re aware of some accessibility issues on this site, particularly for screen reader users. We want to make finding your dream job as easy as possible, so if you require additional support please contact HR Direct on 1800 989 696.

Advertising End Date: 29/11/2026