Senior Data Engineer & SDE - Azure, Spark & Data Migrations

Insight Global; Redmond; On-site; AUD 120,000 - 150,000; Full time

Job summary

Insight Global is seeking a Software Engineer with a strong Data Engineering background to support enterprise data migration, platform modernization, and security initiatives. The role centers on migrating data between systems, improving data quality, and building secure, scalable data solutions on Azure.

You will design pipelines using Python, Spark, and Azure services such as Data Factory, Blob Storage, and Data Lake Gen2, while ensuring data reliability and governance.

Qualifications

• 3+ years of experience in Software Engineering or Data Engineering.
• Proficient in Python development.
• Experience with Apache Spark for large-scale data processing.
• Experience building data pipelines with Azure Data Factory (ADF).
• Experience with Azure Blob Storage / Azure Data Lake Storage Gen2.
• Experience migrating data between systems using SQL databases.
• Understanding of CI/CD concepts and deployment automation.
• Familiarity with cloud security best practices and network concepts.
• Ability to collaborate within a team and learn new technologies quickly.

Responsibilities

• Support ongoing data migration efforts including moving data between SQL-based systems and Azure Data Explorer (Kusto).
• Design, develop, and maintain data pipelines and integrations across Azure-based platforms.
• Partner with engineering teams to understand existing systems and drive platform improvements.
• Ensure data quality, consistency, and reliability during migration and modernization.
• Develop and maintain data processing using Python and Spark.
• Build and support data workflows with ADF, Azure Blob Storage, and Data Lake Gen2.
• Implement and enhance CI/CD pipelines for secure deployments.
• Apply security best practices across the software development lifecycle.

Skills

Python

Problem solving

Team collaboration

Security practices

Tools

Apache Spark

Azure Data Factory (ADF)

Azure Blob Storage

Azure Data Lake Storage Gen2

Azure Data Explorer (Kusto)

SQL databases

Job description

Insight Global is seeking a Software Engineer with a strong Data Engineering background to support enterprise data migration, platform modernization, and security initiatives. The role centers on migrating data between systems, improving data quality, and building secure, scalable data solutions on Azure.

You will design pipelines using Python, Spark, and Azure services such as Data Factory, Blob Storage, and Data Lake Gen2, while ensuring data reliability and governance.