Senior Azure Big Data Engineer - Spark & Pipelines

Cognizant Technology Solutions Corporation; Sydney; On-site; AUD 90,000 - 100,000; Full time

Job summary

Cognizant Technology Solutions Corporation in Sydney is seeking an experienced Data Engineer with 6+ years of end-to-end big data pipeline design, ETL using Apache Spark, Hive, Py-Spark, and real-time pipelines in the Hadoop ecosystem.

You will work in an Agile setting, with global teams, and implement CI/CD via Bitbucket and Jenkins, plus automation and monitoring using Spark Logs. BFSI domain knowledge is a plus.

Qualifications

• 6+ years of experience in end-to-end big data pipeline design.
• Hands-on with Apache Spark, Hive, Py-Spark and real-time data pipelines in Hadoop ecosystem.
• Experience with Agile methodology.
• Knowledge of BFSI domain (Banking, Cards and Payments).

Responsibilities

• Being part of the DDEP team, worked on Data Transformation, Data Completeness, Data Validation, Data Schema Validation, Data duplication and Data accuracy.
• Developing features per client requirements and feasibility discussions.
• Building shell scripts and transformation logic with Py-Spark and Spark SQL for populating records into SCD tables.
• CI/CD pipeline deployment in Jenkins and related DevOps activities.
• Knowledge of SCD type 2 logic for history and active records.
• Analyzing jobs and providing RCA after sprint releases.
• Participation in design walkthroughs and production deployments.
• Creating Control-M jobs for ingestion layer from edge node to RAW/Landing Layer and transformation layer.
• Using Jira, Confluence, and Bitbucket for project management.

Skills

Agile methodology

Tools

Apache Spark

Sqoop

Py-spark

Spark SQL

Sterling

Hive

MySQL

Oracle

Mongo

MS SQL

IBM-DB

Python

Bitbucket

Jenkins

Control-M

Spark Logs

Job description

Cognizant Technology Solutions Corporation in Sydney is seeking an experienced Data Engineer with 6+ years of end-to-end big data pipeline design, ETL using Apache Spark, Hive, Py-Spark, and real-time pipelines in the Hadoop ecosystem.

You will work in an Agile setting, with global teams, and implement CI/CD via Bitbucket and Jenkins, plus automation and monitoring using Spark Logs. BFSI domain knowledge is a plus.