Senior Data Engineer: Spark, Hadoop & Real-Time Pipelines

Cognizant Technology Solutions Corporation; Sydney; On-site; AUD 90,000 - 100,000; Full time

Job summary

Cognizant Technology Solutions Corporation is seeking a Data Engineer in Sydney, Australia, with 6+ years of experience in end-to-end big data pipelines, ETL using Apache Spark, and hands-on Hadoop ecosystem work. You will work in a dynamic, Agile environment on client projects across the globe.

The role requires expertise in Spark, Sqoop, Py-Spark, Spark SQL and toolchains like Bitbucket, Jenkins, and CI/CD processes.

Qualifications

• End-to-end big data pipeline design and ETL using Spark.
• Hands-on experience with real-time data pipelines in Hadoop ecosystem.
• Agile methodologies in service-based IT projects.
• Experience in Banking/ BFSI domain a plus.

Responsibilities

• Transform data and validate quality to ensure production feed accuracy.
• Develop features per client requirements and assess feasibility.
• Build shell scripts and transformation logic with PySpark and Spark SQL.
• Deploy code via CI/CD pipelines using Jenkins in DEV/SIT/SVP.

Skills

Apache Spark

Sqoop

Py-spark

Spark SQL

Sterling

Tools

Bitbucket

Jenkins

Control-M

Jira

Confluence

Job description

Cognizant Technology Solutions Corporation is seeking a Data Engineer in Sydney, Australia, with 6+ years of experience in end-to-end big data pipelines, ETL using Apache Spark, and hands-on Hadoop ecosystem work. You will work in a dynamic, Agile environment on client projects across the globe.

The role requires expertise in Spark, Sqoop, Py-Spark, Spark SQL and toolchains like Bitbucket, Jenkins, and CI/CD processes.