Senior Azure Data Engineer

Cognizant Technology Solutions Corporation; Sydney; On-site; AUD 90,000 - 100,000; Full time

Job summary

Cognizant Technology Solutions Corporation in Sydney is seeking an experienced Data Engineer with 6+ years of end-to-end big data pipeline design, ETL using Apache Spark, Hive, Py-Spark, and real-time pipelines in the Hadoop ecosystem.

You will work in an Agile setting, with global teams, and implement CI/CD via Bitbucket and Jenkins, plus automation and monitoring using Spark Logs. BFSI domain knowledge is a plus.

Qualifications

• 6+ years of experience in end-to-end big data pipeline design.
• Hands-on with Apache Spark, Hive, Py-Spark and real-time data pipelines in Hadoop ecosystem.
• Experience with Agile methodology.
• Knowledge of BFSI domain (Banking, Cards and Payments).

Responsibilities

• Being part of the DDEP team, worked on Data Transformation, Data Completeness, Data Validation, Data Schema Validation, Data duplication and Data accuracy.
• Developing features per client requirements and feasibility discussions.
• Building shell scripts and transformation logic with Py-Spark and Spark SQL for populating records into SCD tables.
• CI/CD pipeline deployment in Jenkins and related DevOps activities.
• Knowledge of SCD type 2 logic for history and active records.
• Analyzing jobs and providing RCA after sprint releases.
• Participation in design walkthroughs and production deployments.
• Creating Control-M jobs for ingestion layer from edge node to RAW/Landing Layer and transformation layer.
• Using Jira, Confluence, and Bitbucket for project management.

Skills

Agile methodology

Tools

Apache Spark

Sqoop

Py-spark

Spark SQL

Sterling

Hive

MySQL

Oracle

Mongo

MS SQL

IBM-DB

Python

Bitbucket

Jenkins

Control-M

Spark Logs

Job description

What makes Cognizant a unique place to work? The combination of rapid growth and an unique place international and innovative environment! This is creating many opportunities for people like YOU — people with an entrepreneurial spirit who want to make a difference in this world.

At Cognizant, together with your colleagues from all around the world, you will collaborate on creating solutions for the world's leading companies and help them become more flexible, more innovative, and successful. Moreover, this is your chance to be part of the success story.

Position Summary

Data Engineer with 6+ years of experience with end-to-end Big Data Pipeline design, ETL pipeline build using Apache Spark, Hive, Py-Spark and also having a hand-on experience in building of real-time data pipeline using Hadoop ecosystem. Also worked on mainly Agile methodology. Working for service-based IT consultancies have provided me the opportunities to work for different clients all over the globe. This has exposed me to different types of enterprise environments, which helped me gain knowledge in terms of technology, people, process and delivery.

Mandatory Skills

1. Skills/Tools: Apache Spark, Sqoop, Py-spark, Spark SQL,Sterling
2. Database: Hive, MySQL, Oracle, Oracle, Mongo, Sqoop, MS SQL,IBM-DB
3. Language and No-SQL : Python,
4. Build Tools: Expertise in setting up and managing continuous integration and continuous deployment pipelines in Bitbucket and Jenkins
5. Job Scheduler: Familiarity with control-M scheduling tool for ordering the Job pipeline.
6. Monitoring and Logging: Skills in using monitoring tools and analysing the root cause using Spark Logs
7. Security and Compliance: Understanding of security best practices and compliance requirements
8. Business Domain Knowledge: Banking (BFSI), Cards and Payments

Roles And Responsibilities

1. Being part of the DDEP team, worked on Data Transformation Data Completeness, Data Validation, Data Schema Validation, Data duplication and Data accuracy to ensure proper data feed to production tables
2. Working on developing features as per client requirement and client requirement and involved in discussing the feasibility of changes to ensure successful run of developed code.
3. Involved in Building shell Scripts for various activities that is not handled by DDEP framework and building transformation logic using Py-spark and Spark SQL and by using same populating the records into SCD Table.
4. Worked in code deployment activities in DEV, SIT and SVP Layer via CICD pipeline using Jenkins.
5. Have good knowledge on SCD type 2 logics to maintain both history and active Records.
6. Have worked on Analyzing the jobs and providing RCA if issues raised post sprint Release.
7. Took part in Implementation and Design Walkthrough with various teams and worked on Production Deployment on change related Activities.
8. Proven Experience in delivering projects successfully through effective planning, prioritizing, team building , task scheduling and tracking scheduled tasks.lnvolved in creating control M Jobs for Ingestion Layer from edge node to RAW/Landing Layer and transformation layer.
9. Utilize Atlassian tools, including Jira, confluence, and Bitbucket, for project management and team collaboration.
10. Involved on doing the Tech PIV and testing the functionality of the tables as part of Production Implementation
11. Working with Sterling and able to get confidence from client to handle the delivery single handedly.

Salary Range: 90k-100k

Date of Posting: 3/11/2026

For a complete list of open opportunities with Cognizant, visit http://www.cognizant.com/careers. Cognizant is committed to providing Equal Employment Opportunities. Successful candidates will be required to undergo a background check.