Full-Stack Data Engineer (Databricks, FastAPI & Next.js)

Celebal Technologies; Brisbane; Hybrid; AUD 120,000 - 160,000; Part time

Job summary

Celebal Technologies is seeking a highly capable Full Stack Data Engineer to design cloud-native data platforms leveraging Databricks, PySpark, Python backends, and Next.js frontends. The role spans Databricks compute, data pipelines, API services, and secure cloud integrations.

The successful candidate will contribute to enterprise-grade analytics integrations, AI/ML workflows, and scalable web applications while working with Azure cloud services and modern CI/CD practices.

Qualifications

• Databricks data engineering and ETL development.
• PySpark notebooks and data pipelines development.
• Backend APIs using FastAPI/Flask/Django.
• REST API development and authentication (OAuth2/JWT).
• CI/CD pipelines and version control with Git.
• Agile software development practices.
• Unit and integration testing.
• Secure coding practices and deployment.
• Experience with Azure cloud services.

Responsibilities

• Develop and maintain Databricks data processing workloads.
• Create PySpark notebooks and data transformation workflows.
• Build backend services with Python and REST APIs.
• Develop authentication and authorization mechanisms (OAuth2, JWT).
• Integrate backend with Databricks, Azure services, and third-party APIs.
• Implement asynchronous processing and background jobs.
• Write reusable, well-tested code with logging and monitoring.
• Collaborate on CI/CD pipelines and code reviews.

Skills

Databricks

PySpark

FastAPI

REST APIs

OAuth2/JWT

Agile

Unit Testing

CI/CD

Cloud-native

REST API Development

Tools

Azure Cloud Services

Git

CI/CD Pipelines (Azure DevOps / GitHub Actions)

Delta Lake

Job description

Location: REMOTE or Hybrid from Brisbane QLD 4000 (for Local Candidates)

Duration: 3+ Months Initial (Extendable up to 6+ months)

Must Have: Unrestricted working rights in Australia.

Job Summary

We are seeking a highly motivated Full Stack Data Engineer to build modern, scalable, cloud-native applications that leverage Databricks for data processing, Python for backend services and APIs, and Next.js for responsive web applications.

The ideal candidate will have experience developing enterprise-grade applications that integrate with analytics platforms, data pipelines, AI/ML services, and cloud-native technologies. The role requires strong backend engineering skills, modern frontend development capabilities, and experience working with Databricks as the compute platform.

Key Responsibilities

Databricks & Compute

• Develop and maintain data processing workloads using Databricks.
• Build PySpark notebooks and workflows for data transformation and analytics.
• Integrate applications with Databricks Jobs, Workflows, SQL Warehouses, and REST APIs.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Work with Delta Lake, Unity Catalog, and Medallion Architecture.
• Develop data APIs and services that consume Databricks datasets.
• Design and develop scalable backend services using Python (FastAPI, Flask, or Django).
• Build RESTful APIs and microservices.
• Develop authentication and authorization mechanisms (OAuth2, JWT, SSO).
• Integrate backend services with Databricks, Azure services, and third-party APIs.
• Implement asynchronous processing and background jobs.
• Write reusable, well-tested, and maintainable code.
• Build logging, monitoring, and error-handling capabilities.

Frontend Development (Next.js)

• Develop responsive web applications using Next.js and React.
• Create reusable UI components and modern dashboards.
• Integrate frontend applications with backend APIs.
• Implement authentication and role-based access control.
• Optimize application performance and user experience.

Integration & Cloud

• Integrate applications with Azure cloud services.
• Consume APIs from enterprise applications and external systems.
• Implement secure API communication and data exchange.
• Participate in cloud deployment and infrastructure discussions.

DevOps & CI/CD

• Use Git for source control and collaborative development.
• Build and maintain CI/CD pipelines using Azure DevOps or GitHub Actions.
• Support automated testing and deployment processes.
• Participate in code reviews and enforce coding standards.

Quality & Security

• Develop unit and integration tests.
• Ensure applications meet enterprise security and compliance requirements.
• Implement secure coding practices and vulnerability remediation.
• Troubleshoot production issues and support release activities.

Must Have

Data Engineering core concepts

• Databricks
• ETL Pipelines, Medallion Architecture, PySpark
• Databricks Automation Bundles
• FastAPI (Must)
• REST API Development
• AuthN Basics (JWT, OAuth2)
• Agile
• Unit Testing
• CI/CD

Good to Have

• Next.js or similar
• Html/css framework
• MLOps

ADDITIONAL REQUIREMENTS

• Unrestricted working rights in Australia.