Azure Big Data Platform Support Engineer

Luxoft; Kraków; Hybrid; PLN 140,000 - 210,000; Full time

Job summary

Luxoft is seeking a Senior Big Data Engineer to work on ACQA, a cloud-based valuation and risk platform built on Microsoft Azure. The role involves supporting multiple teams to build scalable analytics, managing the Databricks environment, and driving platform upgrades with best practices.

You will contribute to the design and operation of CI/CD pipelines, containerisation, and cloud-native tooling to ensure efficient, reliable data workflows across Front Office, Finance and Risk.

Qualifications

• Experience developing big data solutions with Spark in production.
• Proficient in Java/Scala for complex production systems.
• Prior platform engineering on a major cloud provider, preferably Azure.
• Experience building and operating CI/CD pipelines with modern tools.
• Proficiency with containerisation and Kubernetes in development and runtime.

Responsibilities

• Collaborate with federated teams to enable BigData solutions.
• Manage the ACQA Databricks platform, including monitoring and tooling.
• Provide best-practices for runtime upgrades and platform improvements.

Skills

Big data techs (Spark)

Java/Scala prod systems

Azure cloud platform engineering

CI/CD pipelines (GitLab)

Kubernetes containerisation

Tools

Databricks

Terraform

GitLab CI/CD

Kubernetes

Job description

Project description

ACQA is built on Microsoft Azure cloud computing technology. It aims to deliver:Scalable cost-efficient infrastructure, using cloud PaaS components.Single core platform, open architecture, designed for change, itemised $cost metrics, automated data lineage.Shared across Front Office, Finance and Risk, improving regulatory compliance.One-Platform / One-Experience -fast to train, easy to operate, retaining talent.The ACQA platform is made up of a series of components providing the next generation valuation and risk management services.

Responsibilities

• Work with multiple federated teams to enable them to build BigData solutionsManage the ACQA Databricks platform – provide monitoring and other toolingProvide best-practices for runtime upgrades

SKILLS

Must have

• Development of big data technologies like Apache SparkProgramming complex production systems in Java and ScalaExperience in a platform engineering role on a major cloud provider (ideally Microsoft Azure)Development and/or operations of CI/CD pipelines using modern cloud-friendly build systems like GitLabExperience with containerisation in development, build and runtime environments including experience of Kubernetes7+ years of general experience

Nice to have

Production experience with Terraform on a major public cloud provider (ideally Microsoft Azure)Databricks experiencePython experienceTesting and instrumentation practices, observability and alerting toolingUnderstanding of information modelling, data structures and algorithmsTechnical architecture and low-level designAgile development and planning practicesAbility to collaborate effectively in a large global team and influence key architectural decisions