Azure Data Engineer/Integration Developer

DXC Luxoft; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

Luxoft is seeking an Azure Data Engineer / Integration Developer to own the delivery and operation of complex data pipelines in a hybrid Azure/AWS landscape. You will lead automated reconciliation, governance, and secure data migrations while collaborating in an agile team.

You will design scalable data architectures, optimize PySpark and Spark SQL, and implement IaC and CI/CD with Azure DevOps. 5+ years of data engineering experience in Azure is expected, with strong SQL expertise and API

Qualifications

• Extensive commercial experience orchestrating complex multi-step pipelines in Azure Data Factory and Azure Databricks.
• Proficiency in PySpark and Spark SQL with performance optimization.
• Expertise in Azure SQL / T-SQL including stored procedures, indexing, and dimensional modeling.
• Experience with cross-cloud integrations (Azure and AWS) and secure cloud workflows.

Responsibilities

• Pipeline Engineering & Delivery: Design, build, schedule, and maintain enterprise-grade batch and micro-batch data pipelines using Azure Data Factory and Azure Databricks.
• Transformations & Processing: Write clean, modular transformation logic in PySpark and Spark SQL across Bronze/Silver/Gold delta lake layers.
• Database Engineering & Data Modelling: Architect and optimize Azure SQL database solutions, dimensional/relational models.
• Heterogeneous Integrations: Build end-to-end file, API, and database integrations across hybrid environments.
• Production Operations & Failure Recovery: Lead troubleshooting, RCA, failure recovery, and reconciliation checks.
• Cross-Cloud Data Migration: Support data migration across AWS and Azure.
• DevOps & IaC Automation: Maintain Terraform-based IaC, CI/CD via Azure DevOps YAML.
• Security & Governance: Implement credential-less authentication, Key Vault, RBAC.

Skills

Azure Data Factory

Azure Databricks

PySpark

Spark SQL

Azure SQL / T-SQL

REST APIs

Data Integration

Terraform

Azure DevOps (YAML)

Git

AWS/Azure cross-cloud

Tools

Terraform

Azure DevOps

Git

S3/Blob storage integrations

Job description

Project description

Luxoft is seeking an Azure Data Engineer / Integration Developer to join our enterprise Data & Analytics practice. In this role, you will take technical ownership of the delivery, support, and operational resilience of complex data pipelines, enterprise transformations, and cross-platform integrations across a hybrid Azure/AWS landscape. You will work within an agile team delivering robust data integration frameworks for financial close systems, regulatory compliance, and enterprise reporting—leading operational failure recovery, automated source-to-target reconciliation, and cloud data migration initiatives.

Responsibilities

• Pipeline Engineering & Delivery: Design, build, schedule, and maintain enterprise-grade batch and micro-batch data pipelines using Azure Data Factory (ADF) and Azure Databricks.Transformations & Processing: Write clean, modular, and optimized transformation logic in PySpark and Spark SQL across bronze, silver, and gold delta lake architecture layers.Database Engineering & Data Modelling: Architect and optimize Azure SQL database solutions, dimensional/relational data models (Star/Snowflake schema), complex stored procedures, views, and indexing strategies.Heterogeneous Integrations: Build and maintain end-to-end file, API, and database integrations across hybrid environments (REST/SOAP APIs, SFTP, Blob/S3 storage drops, database CDC, and messaging queues).Production Operations & Failure Recovery: Lead proactive troubleshooting, incident root-cause analysis (RCA), failure recovery, and automated source-to-target reconciliation checks (row counts, checksums, hash comparisons, and schema drift handling).Cross-Cloud Data Migration: Support data migration activities and workload cutovers across the AWS and Azure landscape where required, ensuring secure network transit and data consistency.DevOps & IaC Automation: Maintain infrastructure as code using Terraform, build automated CI/CD deployment pipelines via Azure DevOps (YAML), and enforce branch policies in Git.Security & Governance: Implement credential-less authentication and secrets management using Azure Managed Identities, Azure Key Vault, Azure RBAC, and secure private networking.

SKILLS

Must have

• Azure Cloud Data Core (Databricks — Mandatory): Extensive commercial experience orchestrating complex multi-step pipelines in Azure Data Factory (ADF) and building distributed production workloads on Azure Databricks (mandatory).Big Data & Distributed Computing: Advanced proficiency in PySpark and Spark SQL, including transformation logic, query optimization, partitioning, caching, and shuffle tuning.Relational Engineering & Data Modelling: Deep expertise in Azure SQL / T-SQL, including writing and tuning complex stored procedures, query execution plan analysis, indexing strategies, and relational/dimensional data modelling.Full-Spectrum Integration: Proven hands-on experience implementing file, API, and database integrations across diverse data formats (JSON, Parquet, Avro, Delta, CSV) utilizing REST APIs, database connectors, and cloud storage endpoints.Operational Reliability & Reconciliation: Demonstrated track record in pipeline troubleshooting, failure recovery, idempotent reprocessing, and automated source-to-target reconciliation mechanisms.DevOps, CI/CD & IaC: Commercial experience automating infrastructure deployment with Terraform, managing version control in Git, and creating automated release gates using Azure DevOps YAML pipelines.Cloud Security Standards: Direct experience architecting secure cloud workflows using Azure Managed Identities, Azure Key Vault, Service Principals, and least-privilege RBAC.AWS Infrastructure & Networking: Practical working knowledge of core AWS infrastructure and networking concepts (e.g., Amazon S3, IAM policies/roles, VPCs, subnets, security groups, and routing).Experience Level: 5+ years of dedicated commercial experience in Data Engineering / ETL Integration, with at least 3+ years focused specifically on the Microsoft Azure ecosystem.

Nice to have

Practical experience supporting or executing cloud and data migrations involving AWS.Domain background in financial close processes, reconciliation accounting, or regulatory reporting workflows.Experience handling sensitive or restricted data environments (PII/PCI compliance, data masking, encryption) and implementing enterprise data governance frameworks (e.g., Microsoft Purview, Unity Catalog).Relevant Microsoft or AWS Certifications (e.g., DP-203: Azure Data Engineer Associate, AWS Certified Data Engineer / Solutions Architect).