Senior AWS Data Engineer — Pipelines, PySpark, DataOps

Tech Mahindra; Sydney; On-site; AUD 140,000 - 180,000; Full time

Job summary

Tech Mahindra is hiring a senior Data Engineer to design, build, and operate end-to-end data pipelines on AWS. You will work with teams across analytics and data science to deliver reliable, scalable data products in a large enterprise environment. Expertise in SQL, Python, PySpark, and AWS data services is essential.

Strong communication and data warehousing knowledge are required. The role focuses on data ingestion, transformation, and governance, with emphasis on DataOps practices,

Qualifications

• 8–10 years’ experience as a Data Engineer.
• Advanced SQL skills.
• Hands-on experience working with Teradata and Siebel CRM data sets.
• Experience delivering data pipelines in a large-scale enterprise data platform environment.
• Strong hands-on AWS experience with Amazon S3, AWS Glue, Amazon Redshift, Amazon Athena, Amazon EMR and dbt.
• Strong programming capability in Python and strong data transformation experience using PySpark (preferred) and/or Spark.
• Advanced SQL skills (query optimisation, complex joins, window functions, performance tuning).
• Experience with workflow orchestration tools such as Airflow.
• Solid understanding of data warehousing concepts (dimensional modelling, partitioning, incremental loads, CDC concepts).
• Experience implementing monitoring, logging, alerting, and operational support processes.
• Strong communication skills and ability to work with stakeholders to translate requirements into data deliverables.
• Telco Industry Experience is highly desirable.

Responsibilities

• Design, develop, and maintain end-to-end data pipelines (batch and near real-time) on AWS Data Platform
• Build and manage ETL/ELT workflows using AWS services (e.g., AWS Glue, S3, Redshift, Athena, EMR), dbt and orchestration tools such as Airflow
• Implement data ingestion patterns from diverse sources (databases, APIs, files, event streams) into lake/warehouse layers such as raw, cleansed, and curated data layers
• Develop transformation logic using SQL and Python/PySpark for cleansing, enrichment, and standardisation
• Implement robust data quality checks, reconciliation controls, and monitoring/alerting for failures and anomalies
• Collaborate with data analysts/data scientists to model datasets for analytics and machine learning consumption.
• Contribute to DataOps/DevOps practices: version control, CI/CD, automated testing, release management, and operational support.
• Produce and maintain technical documentation (data flows, mappings, job schedules, runbooks, and operational procedures)
• Optimise Data Pipeline performance and Support workflow orchestration and scheduling
• Support production deployments and operations

Skills

Advanced SQL

Python

PySpark

Spark

Communication skills

Tools

Teradata

Siebel CRM data

dbt

Airflow

Amazon S3

AWS Glue

Amazon Redshift

Amazon Athena

Amazon EMR

Job description

Tech Mahindra is hiring a senior Data Engineer to design, build, and operate end-to-end data pipelines on AWS. You will work with teams across analytics and data science to deliver reliable, scalable data products in a large enterprise environment. Expertise in SQL, Python, PySpark, and AWS data services is essential.

Strong communication and data warehousing knowledge are required. The role focuses on data ingestion, transformation, and governance, with emphasis on DataOps practices,