Senior Data Engineer — Big Data & Spark Expert

Cognizant Technology Solutions Corporation; Sydney; On-site; AUD 90,000 - 100,000; Full time

Job summary

Cognizant Technology Solutions Corporation in Sydney is seeking a Data Engineer with 6+ years of experience to design end-to-end big data pipelines, build ETL using Apache Spark and Hive, and contribute to real-time data processing within Hadoop ecosystems. You will work across Agile teams to deliver scalable data solutions for global clients.

The role requires hands-on skills in Spark, Sqoop, Python, and CI/CD tooling (Bitbucket, Jenkins) and familiarity with security and governance in BFSI

Qualifications

• Proficient in end-to-end Big Data pipeline design using Apache Spark, Hive and PySpark.
• Experience with real-time data pipelines in Hadoop ecosystem.
• Strong Python and SQL skills with data quality focus.

Responsibilities

• Participate in Data Transformation, Data Validation, and Data Completeness efforts.
• Develop features per client requirements and assess feasibility of changes.
• Create shell scripts and transformation logic using PySpark and Spark SQL.
• Manage CI/CD pipelines via Jenkins and deployment activities in DEV/SIT/SVP.
• Apply SCD type 2 logic to maintain history and current records.
• Analyze jobs and provide RCA after sprint releases.
• Lead production deployment activities and changes coordination.
• Collaborate with teams using Jira, Confluence and Bitbucket.
• Work with Sterling to ensure delivery confidence.

Skills

Apache Spark

Sqoop

Py-spark

Spark SQL

Sterling

Python

CI/CD pipelines

Jenkins

Bitbucket

Control-M

Monitoring & Logging

Security best practices

BFSI domain knowledge

Cards & Payments

Tools

Bitbucket

Jenkins

Control-M

Hadoop ecosystem

Job description

Cognizant Technology Solutions Corporation in Sydney is seeking a Data Engineer with 6+ years of experience to design end-to-end big data pipelines, build ETL using Apache Spark and Hive, and contribute to real-time data processing within Hadoop ecosystems. You will work across Agile teams to deliver scalable data solutions for global clients.

The role requires hands-on skills in Spark, Sqoop, Python, and CI/CD tooling (Bitbucket, Jenkins) and familiarity with security and governance in BFSI