Senior Data Engineer-AWS

Tech Mahindra; Sydney; On-site; AUD 120,000 - 160,000; Full time

Job summary

Tech Mahindra is hiring a Data Engineer to design, build and maintain end-to-end data pipelines on AWS. You will work with Spark/Python, SQL, and Airflow to ingest, transform, and validate data across raw, cleansed, and curated layers. Collaboration with data scientists and analysts is key to delivering analytics-ready datasets.

The role emphasizes DataOps, CI/CD, and comprehensive documentation, with experience in AWS services including S3, Glue, Redshift, Athena and EMR highly valued.

Qualifications

• Minimum experience as a Data Engineer in large-scale environments.
• Strong command of SQL and data modelling concepts.
• Hands-on AWS data services experience (S3, Glue, Redshift, Athena, EMR).
• Proficient in Python and PySpark for data transformation.

Responsibilities

• Design, develop, and maintain end-to-end data pipelines on AWS Data Platform.
• Build and manage ETL/ELT workflows using AWS services, dbt, and Airflow.
• Ingest data from databases, APIs, files, and event streams into lake/warehouse layers.
• Transform data with SQL and Python/PySpark for cleansing and enrichment.
• Implement data quality checks, monitoring, and alerting for issues.
• Collaborate with analysts/data scientists to model datasets for analytics/ML.
• Contribute to DataOps/DevOps incl. version control and CI/CD.
• Produce and maintain data flow documentation and runbooks.
• Optimise pipeline performance and support deployments.

Skills

Data engineering

Advanced SQL

PySpark

Python

Airflow

Data Warehousing

ETL/ELT pipelines

AWS data services

Stakeholder communication

Telco industry experience

Tools

Amazon S3

AWS Glue

Amazon Redshift

Amazon Athena

Amazon EMR

dbt

Airflow

Teradata

Siebel CRM data

Job description

At Tech Mahindra (Tech Mahindra | Connected World, Connected Experiences), we live the philosophy of connected world and connected experiences. We thrive on change that is powered by the intelligent symphony of technology and humans designing meaningful and sustainable experiences. Consumer 'experiences' are driving and disrupting industries like never before. Businesses must build seamless yet simple enterprises that collaborate, synergize, and drive the change. Change that connects us all and empowers us to deliver experiences that span across the digital, the physical, the convergent, and everything in between. That's when truly connected experiences manifest.

Responsibilities:

• Design, develop, and maintain end to end data pipelines (batch and near real time) on AWS Data Platform
• Build and manage ETL/ELT workflows using AWS services (e.g., AWS Glue, S3, Redshift, Athena, EMR), dbt and orchestration tools such as Airflow
• Implement data ingestion patterns from diverse sources (databases, APIs, files, event streams) into lake/warehouse layers such as raw, cleansed, and curated data layers
• Develop transformation logic using SQL and Python/PySpark for cleansing, enrichment, and standardisation
• Implement robust data quality checks, reconciliation controls, and monitoring/alerting for failures and anomalies
• Collaborate with data analysts/data scientists to model datasets for analytics and machine learning consumption.
• Contribute to DataOps/DevOps practices: version control, CI/CD, automated testing, release management, and operational support.
• Produce and maintain technical documentation (data flows, mappings, job schedules, runbooks, and operational procedures)
• Optimise Data Pipeline performance and Support workflow orchestration and scheduling
• Support production deployments and operations

Required Skills & Experience

• Extensive experience as a Data Engineer
• Advanced SQL skills
• Hands on experience working with Teradata and Siebel CRM data sets
• Experience delivering data pipelines in a large-scale enterprise data platform environment
• Strong hands-on AWS experience with common data services such as : Amazon S3, AWS Glue, Amazon Redshift, Amazon Athena, Amazon EMR and dbt
• Strong programming capability in Python and strong data transformation experience using PySpark (preferred) and/or Spark.
• Advanced SQL skills (query optimisation, complex joins, window functions, performance tuning)
• Experience with workflow orchestration tools such as Airflow
• Solid understanding of data warehousing concepts (dimensional modelling, partitioning, incremental loads, CDC concepts).
• Experience implementing monitoring, logging, alerting, and operational support processes.
• Strong communication skills and ability to work with stakeholders to translate requirements into data deliverables
• Telco Industry Experience is highly desirable