Cloud Data Engineer — Scalable Pipelines (Azure/PySpark)

Cognizant Technology Solutions Corporation; Sydney; On-site; AUD 80,000 - 90,000; Full time

Job summary

Cognizant Technology Solutions Corporation in Sydney, Australia, is seeking a results-driven Data Engineer to design, build, and optimize scalable data pipelines across cloud and distributed systems, using Python, PySpark, and SQL to deliver clean, reliable data for analytics and reporting.

You will collaborate with cross-functional teams, implement governance and security practices, and contribute to metadata-driven frameworks, ETL enhancements, and cost-efficient data platforms, with a focus

Qualifications

• Experience designing scalable data pipelines across cloud and distributed systems.
• Proficient in PySpark and SQL for data ingestion, transformation and analytics.
• Knowledge of data governance, security practices, and compliance.
• Experience with cloud services (Azure and AWS) and data lake technologies.
• Familiarity with metadata-driven frameworks and ETL enhancements.

Responsibilities

• Build and maintain scalable data pipelines using cloud services for ingestion, transformation, and storage.
• Develop data products that integrate customer data from multiple sources into unified profiles.
• Create metadata-driven frameworks to ensure data quality, consistency, and governance.
• Write PySpark and Spark SQL scripts for complex data transformations, ensuring scalable pipelines.
• Support ETL processes by fixing errors, improving resiliency, and ensuring data integrity across reports.
• Monitor data platforms, respond to alerts, and optimize pipelines for efficiency and cost reduction.
• Perform root cause analysis for job failures and ensure stability after releases.
• Collaborate with onsite and offshore teams to coordinate tasks and deliver projects smoothly.
• Participate in design reviews, implementation walkthroughs, and production deployments.
• Apply governance best practices to maintain security, lineage, and compliance with standards.

Skills

Data engineering

Azure ADLS

PySpark

SQL

Data governance

Monitoring & logging

Security & compliance

Banking domain

Education

Databricks Certified Data Engineer Associate

Tools

Git

Bitbucket

CI/CD

Shell scripting

Postman

Spark SQL

Jira

Confluence

VS Code

Job description

Cognizant Technology Solutions Corporation in Sydney, Australia, is seeking a results-driven Data Engineer to design, build, and optimize scalable data pipelines across cloud and distributed systems, using Python, PySpark, and SQL to deliver clean, reliable data for analytics and reporting.

You will collaborate with cross-functional teams, implement governance and security practices, and contribute to metadata-driven frameworks, ETL enhancements, and cost-efficient data platforms, with a focus