Azure Databricks Data Engineer - Real-Time Analytics

Luxoft; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

Luxoft is seeking an experienced Azure Databricks Developer to design and implement scalable cloud-based data solutions on Azure. You will build and optimize data pipelines, handle real-time streaming, and collaborate with data engineers, architects, and business teams to deliver analytics across trading platforms.

Strong Python, PySpark, SQL, and DevOps skills are required, with a focus on secure, compliant cloud architectures.

Qualifications

• 10+ years experience in development3+ years of hands-on experience with Azure cloud services, including Databricks and DevOps.
• Strong proficiency Python and Pyspark or SQL for data transformation and analysis.
• Experience with real-time data streaming and event-driven architectures.
• Deep understanding of ETL/ELT workflows and big data processing frameworks.
• Knowledge of Azure DevOps and CI/CD pipelines for automated deployments.
• Familiarity with data security, governance, and compliance best practices in cloud environments.
• Excellent problem-solving skills and ability to work in an agile development environment.
• Strong communication and collaboration skills.

Responsibilities

• Design, implement, and maintain data pipelines and ETL processes using Azure Databricks.
• Design and manage Azure Databricks for optimized data storage and retrieval.
• Collaborate with cross-functional teams to define and implement data integration solutions across multiple sources and platforms.
• Implement best practices for data governance, security, and compliance in cloud-based solutions.
• Monitor and troubleshoot performance issues in Azure-based applications and data pipelines.
• Automate deployment and monitoring processes using CI/CD pipelines.
• Stay updated with emerging Azure technologies and provide recommendations for improvements.
• Producing Architectural design diagram/Technical Documentation

Skills

Azure Databricks

Azure DevOps

Python

PySpark

SQL

Real-time data streaming

ETL/ELT

CI/CD pipelines

Data governance

Tools

Azure Databricks

Azure DevOps

Azure Log Analytics

Azure Functions

Power BI

Job description

Luxoft is seeking an experienced Azure Databricks Developer to design and implement scalable cloud-based data solutions on Azure. You will build and optimize data pipelines, handle real-time streaming, and collaborate with data engineers, architects, and business teams to deliver analytics across trading platforms.

Strong Python, PySpark, SQL, and DevOps skills are required, with a focus on secure, compliant cloud architectures.