Azure Data Engineer: Scalable Pipelines & Cross-Cloud Ops

DXC Luxoft; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

Luxoft is seeking an Azure Data Engineer / Integration Developer to own the delivery and operation of complex data pipelines in a hybrid Azure/AWS landscape. You will lead automated reconciliation, governance, and secure data migrations while collaborating in an agile team.

You will design scalable data architectures, optimize PySpark and Spark SQL, and implement IaC and CI/CD with Azure DevOps. 5+ years of data engineering experience in Azure is expected, with strong SQL expertise and API

Qualifications

• Extensive commercial experience orchestrating complex multi-step pipelines in Azure Data Factory and Azure Databricks.
• Proficiency in PySpark and Spark SQL with performance optimization.
• Expertise in Azure SQL / T-SQL including stored procedures, indexing, and dimensional modeling.
• Experience with cross-cloud integrations (Azure and AWS) and secure cloud workflows.

Responsibilities

• Pipeline Engineering & Delivery: Design, build, schedule, and maintain enterprise-grade batch and micro-batch data pipelines using Azure Data Factory and Azure Databricks.
• Transformations & Processing: Write clean, modular transformation logic in PySpark and Spark SQL across Bronze/Silver/Gold delta lake layers.
• Database Engineering & Data Modelling: Architect and optimize Azure SQL database solutions, dimensional/relational models.
• Heterogeneous Integrations: Build end-to-end file, API, and database integrations across hybrid environments.
• Production Operations & Failure Recovery: Lead troubleshooting, RCA, failure recovery, and reconciliation checks.
• Cross-Cloud Data Migration: Support data migration across AWS and Azure.
• DevOps & IaC Automation: Maintain Terraform-based IaC, CI/CD via Azure DevOps YAML.
• Security & Governance: Implement credential-less authentication, Key Vault, RBAC.

Skills

Azure Data Factory

Azure Databricks

PySpark

Spark SQL

Azure SQL / T-SQL

REST APIs

Data Integration

Terraform

Azure DevOps (YAML)

Git

AWS/Azure cross-cloud

Tools

Terraform

Azure DevOps

Git

S3/Blob storage integrations

Job description

Luxoft is seeking an Azure Data Engineer / Integration Developer to own the delivery and operation of complex data pipelines in a hybrid Azure/AWS landscape. You will lead automated reconciliation, governance, and secure data migrations while collaborating in an agile team.

You will design scalable data architectures, optimize PySpark and Spark SQL, and implement IaC and CI/CD with Azure DevOps. 5+ years of data engineering experience in Azure is expected, with strong SQL expertise and API