Senior Data Engineer: Cloud Data Platform & Spark

XPT Software; Brisbane; Sur place; AUD 120 000 - 160 000; Plein temps

Résumé du poste

XPT Software in Australia is seeking an experienced Senior Data Engineer to join our Brisbane team. You will design and implement data pipelines, profiling, and cloud-based data platforms using Azure, Databricks, Spark, and related tools.

You will collaborate with cross-location teams, manage structured and unstructured data, and optimize data workflows with Airflow and scheduling tools. Strong SQL and Python, plus cloud and big data experience, are essential.

Qualifications

• Minimum 8 years of experience in data engineering or analytics.
• Strong SQL and Python expertise.
• Experience with data profiling and cloud data warehouses.
• Hands-on with Azure Cloud and Databricks.
• Experience with scheduling tools like Airflow or Control-M.
• Familiarity with big data tools: Spark, Hive, Impala, Hue, HDFS, HBase.
• Experience managing structured and unstructured data.

Connaissances

Advanced SQL

Python

Data profiling

Azure Cloud

Databricks

Airflow

Control-M

Spark (Python)

Hive

Impala

Hue

HDFS

HBase

Structured data

Unstructured data

Description du poste

• XPT SoftwareAustralia PTY Ltd, incorporated in 2016, is a Software Services company
  
• XPT works with topclients across Australia in Banking, Insurance, Telecom,Retail, Energy, Mining and Manufacturingdomains.
  
• We have 120+technocrats in Australia working at our clientlocations.
  
• XPT SoftwareAustralia is part of group companies which has globalpresence across India & Europe.
  
• We have served100+ clients globally, fulfilling their onsite-offshoreneeds.
  

Minimum Years of Experience: 8Years

Must Have Skills:

· Hands On with advanced SQL, Python etc.

· Hands On in data profiling

· Hands On in working on Cloud like Azure and Cloud DW like Data bricks

· Hands On experience on Scheduling tools like Airflow, Control M etc.

· Knowledgeable on Big Data tools like Spark (Python/Scala), Hive, Impala,Hue and storage (e.g. HDFS, HBase)

· Knowledgeable managing structured and unstructured data types.

Nice to Have Skills:

· Experience in working with a team of data engineers working fromdifferent locations in different time zones.

· Effective prioritization.

· Pragmatic stakeholder management.CloudPlatform Familiarity: Understanding of cloud services (AWS, Azure, GCP) toanalyze and optimize cloud-based systems.

· Basic Scripting Knowledge: Ability to write simple scripts in Python,SQL, or R for data extraction, transformation, and analysis.

Be careful - Don’t provide your bank or credit card details when applying for jobs. Don't transfer any money or complete suspicious online surveys. If you see something suspicious, report this job ad .