Remote Data Engineer - Databricks, FastAPI & Next.js

Celebal Technologies; Brisbane; Hybrid; AUD 120,000 - 160,000; Part time

Job summary

Celebal Technologies is seeking a highly capable Full Stack Data Engineer to design cloud-native data platforms leveraging Databricks, PySpark, Python backends, and Next.js frontends. The role spans Databricks compute, data pipelines, API services, and secure cloud integrations.

The successful candidate will contribute to enterprise-grade analytics integrations, AI/ML workflows, and scalable web applications while working with Azure cloud services and modern CI/CD practices.

Qualifications

• Databricks data engineering and ETL development.
• PySpark notebooks and data pipelines development.
• Backend APIs using FastAPI/Flask/Django.
• REST API development and authentication (OAuth2/JWT).
• CI/CD pipelines and version control with Git.
• Agile software development practices.
• Unit and integration testing.
• Secure coding practices and deployment.
• Experience with Azure cloud services.

Responsibilities

• Develop and maintain Databricks data processing workloads.
• Create PySpark notebooks and data transformation workflows.
• Build backend services with Python and REST APIs.
• Develop authentication and authorization mechanisms (OAuth2, JWT).
• Integrate backend with Databricks, Azure services, and third-party APIs.
• Implement asynchronous processing and background jobs.
• Write reusable, well-tested code with logging and monitoring.
• Collaborate on CI/CD pipelines and code reviews.

Skills

Databricks

PySpark

FastAPI

REST APIs

OAuth2/JWT

Agile

Unit Testing

CI/CD

Cloud-native

REST API Development

Tools

Azure Cloud Services

Git

CI/CD Pipelines (Azure DevOps / GitHub Actions)

Delta Lake

Job description

Celebal Technologies is seeking a highly capable Full Stack Data Engineer to design cloud-native data platforms leveraging Databricks, PySpark, Python backends, and Next.js frontends. The role spans Databricks compute, data pipelines, API services, and secure cloud integrations.

The successful candidate will contribute to enterprise-grade analytics integrations, AI/ML workflows, and scalable web applications while working with Azure cloud services and modern CI/CD practices.