Senior Azure Data Engineer – Big Data & Real-Time Pipelines

Cognizant; Sydney; On-site; AUD 90,000 - 100,000; Full time

Job summary

Cognizant in Sydney seeks a Data Engineer with 6+ years of experience in end-to-end Big Data pipelines and ETL development using Apache Spark, Hive, and Py-Spark. Real-time data pipelines and batch jobs will be part of your scope.

You will work in an Agile environment, collaborating with global teams to deliver scalable data solutions for BFSI and payments domains, while ensuring security and compliance standards are met.

Qualifications

• Experience with end-to-end Big Data pipelines and ETL in Spark/Hive stacks.
• Hands-on with real-time data pipelines using Hadoop ecosystem.
• Proficient in Python and SQL across multiple databases.
• CI/CD and build/automation tools like Jenkins and Bitbucket.
• Familiar with Control-M scheduling for batch jobs.
• Knowledge of BFSI domain concepts and data security practices.

Responsibilities

• Develop end-to-end data pipelines, data quality checks and schema validation.
• Collaborate with clients to define requirements and feasibility in Agile sprints.
• Create shell scripts and transformation logic using PySpark and Spark SQL.
• Deploy code via CI/CD pipelines across DEV, SIT, and PROD environments.
• Maintain SCD Type 2 history and production data integrity.
• Analyze root causes and provide RCA for post-release issues.
• Participate in design walkthroughs and production deployment activities.
• Coordinate ingestion layer jobs from edge to RAW/Landing and transformation layers.

Skills

Hive

MySQL

Oracle

Mongo

Sqoop

MS SQL

IBM-DB

Python

CI/CD pipelines

Jenkins

Bitbucket

Control-M

Monitoring

Security and Compliance

Banking

Tools

Bitbucket

Jenkins

Control-M

Job description

Cognizant in Sydney seeks a Data Engineer with 6+ years of experience in end-to-end Big Data pipelines and ETL development using Apache Spark, Hive, and Py-Spark. Real-time data pipelines and batch jobs will be part of your scope.

You will work in an Agile environment, collaborating with global teams to deliver scalable data solutions for BFSI and payments domains, while ensuring security and compliance standards are met.