Senior Spark & Data Pipeline Engineer

Cognizant Technology Solutions Corporation; Sydney; Hybrid; AUD 90,000 - 100,000; Full time

Job summary

Cognizant Technology Solutions Corporation in Sydney seeks an experienced Data Engineer to design end-to-end big data pipelines using Apache Spark, Hive, and Py-Spark, with hands-on experience in real-time data processing.

You will work in an Agile environment, collaborating with global teams and deploying CI/CD pipelines using Jenkins and Bitbucket, while ensuring data quality and compliance across banking and payments domains.

Qualifications

• 6+ years of experience building end-to-end big data pipelines.
• Hands-on with Spark, Hive, Py-Spark, and real-time data processing.
• Experience with CI/CD pipelines in Jenkins/Bitbucket.

Responsibilities

• Transform data and ensure data quality for production tables.
• Develop features per client requirements and assess feasibility of changes.
• Deploy code across DEV, SIT, SVP layers via CI/CD pipelines.
• Implement SCD type 2 logic and analyze RCA post-sprint releases.
• Collaborate in design walkthroughs and production deployments.

Skills

Python

Spark

CI/CD

Tools

Apache Spark

Sqoop

Py-spark

Spark SQL

Sterling

Bitbucket

Jenkins

Control-M

Monitoring tools

Job description

Cognizant Technology Solutions Corporation in Sydney seeks an experienced Data Engineer to design end-to-end big data pipelines using Apache Spark, Hive, and Py-Spark, with hands-on experience in real-time data processing.

You will work in an Agile environment, collaborating with global teams and deploying CI/CD pipelines using Jenkins and Bitbucket, while ensuring data quality and compliance across banking and payments domains.