Databricks Application Engineer

N2S.Global; Sydney; On-site; AUD 120,000 - 160,000; Full time

Job summary

N2S.Global in Australia is seeking a Databricks Application Engineer to design, develop, optimize, and support scalable data solutions on the Databricks platform. You will collaborate with data engineers, data scientists, and business stakeholders to deliver high-performance pipelines.

The role requires strong Spark and Databricks expertise, cloud experience, Delta Lake, and CI/CD practices. You will build notebooks, workflows, and automated jobs while monitoring data quality and governance.

Qualifications

• Bachelor's degree in CS, IT, engineering or related field.
• 3+ years of data engineering or big data experience.
• Hands-on Databricks experience.
• Strong programming in Python, PySpark or Scala.
• Strong SQL and data warehousing concepts.

Responsibilities

• Design, develop, optimize, and maintain scalable data pipelines on Databricks.
• Build and optimize ETL/ELT processes for large-scale data.
• Develop data ingestion from APIs, databases, files, and streaming sources.
• Implement Delta Lake architecture and data management best practices.
• Optimize Spark jobs for performance, scalability, and cost.
• Collaborate with business and technical teams to translate requirements into solutions.
• Develop notebooks, workflows, and automated jobs within Databricks.
• Monitor and troubleshoot production data pipelines and workflows.
• Implement data quality checks, governance, and security standards.
• Support CI/CD deployment processes and DevOps practices.
• Participate in code reviews and contribute to design discussions.
• Ensure compliance with data management policies and standards.

Skills

Databricks

PySpark

Python

SQL

Spark

ETL/ELT

Education

Bachelor's degree in Computer Science / IT / Engineering

Tools

Delta Lake

Unity Catalog

Databricks Workflows

Azure Databricks

Git CI/CD

Job description

We are seeking a skilled Databricks Application Engineer to design, develop, optimize, and support scalable data engineering and analytics solutions on the Databricks platform. The ideal candidate will have strong expertise in Spark, Databricks, cloud platforms, and data pipeline development. This role involves working closely with data engineers, architects, data scientists, and business stakeholders to deliver high-performance data solutions.

Key Responsibilities:

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
• Build and optimize ETL/ELT processes for large-scale structured and unstructured data.
• Develop data ingestion frameworks from various sources including APIs, databases, flat files, and streaming platforms.
• Implement Delta Lake architecture and data management best practices.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Collaborate with business and technical teams to gather requirements and translate them into technical solutions.
• Develop notebooks, workflows, and automated jobs within Databricks.
• Monitor and troubleshoot production data pipelines and workflows.
• Implement data quality checks, governance, and security standards.
• Support CI/CD deployment processes and DevOps practices.
• Participate in code reviews and contribute to technical design discussions.
• Ensure compliance with organizational data management policies and standards.

Required Qualifications:

• Bachelor's degree in Computer Science, Information Technology, Engineering, or related field.
• 3+ years of experience in Data Engineering or Big Data technologies.
• Hands‑on experience with Databricks platform.
• Strong programming skills in Python, PySpark, or Scala.
• Experience with Apache Spark and distributed data processing.
• Strong SQL development and database concepts.
• Experience with Delta Lake, Databricks Workflows, and Unity Catalog.
• Knowledge of Data Warehousing concepts and dimensional modeling.
• Experience working with cloud platforms:Azure Databricks (Preferred)
• AWS Databricks
• GCP Databricks
• Experience with Git, CI/CD pipelines, and version control systems.
• Strong analytical and problem‑solving skills.

Preferred Qualifications:

• Databricks Certified Associate or Professional certification.
• Experience with Azure Data Factory, Azure Synapse, or Microsoft Fabric.
• Knowledge of Kafka, Event Hubs, or streaming data architectures.
• Experience with Terraform or Infrastructure as Code (IaC).
• Familiarity with Airflow, DBT, or orchestration tools.
• Exposure to machine learning workflows within Databricks.
• Experience in Agile/Scrum environments.

Technical Skills:

Must Have:

• Databricks
• PySpark
• Python
• SQL
• ETL/ELT

Good to Have:

• Microsoft Fabric
• Kafka
• Airflow
• DBT
• Terraform
• GitHub Actions / Azure DevOps

Nice to Have:

• Machine Learning Operations (MLOps)
• Data Governance
• Unity Catalog
• Data Security & Compliance

Soft Skills:

• Strong communication and stakeholder management.
• Ability to work independently and collaboratively.
• Excellent troubleshooting and analytical skills.
• Strong documentation and knowledge-sharing mindset.