Databricks Developer

Insight Global; Vancouver; On-site; CAD 110,000 - 150,000; Full time

Job summary

Insight Global is seeking a Databricks Developer to design, build, and optimize high-performance data pipelines in a modern Azure cloud environment.

You will collaborate with engineers, analysts, and architects to deliver trusted datasets powering enterprise analytics and reporting.

Qualifications

• 3+ years in data engineering or big data development.
• Strong Databricks + Apache Spark experience (PySpark, Spark SQL, Delta Lake).
• Azure Data Factory + Azure Data Lake expertise.
• SQL & Python proficiency for transformation and validation.

Responsibilities

• Design, build, and maintain scalable data pipelines and workflows using Databricks (SQL, PySpark, Delta Lake).
• Develop efficient ETL/ELT pipelines for structured and semi-structured data using Azure Data Factory (ADF) and Databricks notebooks/jobs.
• Collaborate with Data Scientists, Data Analysts, and Data Architects to deliver trusted datasets.

Skills

Databricks + Spark

Azure Data Factory

SQL

Python

APIs integration

Delta Lake

ETL/ELT

Git CI/CD

Data governance

dbt

Azure Synapse

ML/IoT

Tools

Databricks notebooks

HTTP requests

CI/CD tooling

Job description

Job Description

Insight Global is seeking a Databricks Developer for a large maritime transportation enterprise. This role is a hands-on technical contributor responsible for building high-performance data pipelines, optimizing Spark workloads, and enabling scalable data products across a modern Azure cloud environment. The Databricks Developer will work closely with data engineers, analysts, and architects to deliver reliable, trusted datasets that power enterprise analytics and reporting.

Responsibilities

• Design, build, and maintain scalable data pipelines and workflows using Databricks (SQL, PySpark, Delta Lake).
• Develop efficient ETL/ELT pipelines for structured and semi-structured data using Azure Data Factory (ADF) and Databricks notebooks/jobs.
• Integrate and transform large-scale datasets from multiple sources into unified, analytics-ready outputs.
• Optimize Spark jobs and manage Delta Lake performance using techniques such as partitioning, Z-ordering, broadcast joins, and caching.
• Design and implement data ingestion pipelines for RESTful APIs, transforming JSON responses into Spark tables.
• Apply best practices in data modeling and data warehousing concepts.
• Perform data validation and quality checks.
• Work with various data formats, including JSON, Parquet, and Avro.
• Build and manage data orchestration pipelines, including linked services and datasets for ADLS, Databricks, and SQL Server.
• Create parameterized and dynamic ADF pipelines, and trigger Databricks notebooks from ADF.
• Collaborate closely with Data Scientists, Data Analysts, Business Analysts, and Data Architects to deliver trusted, high-quality datasets.
• Contribute to data governance, metadata documentation, and ensure adherence to data quality standards.
• Use version control tools (e.g., Git) and CI/CD pipelines to manage code deployment and workflow changes.
• Develop real-time and batch processing pipelines for streaming data sources such as MQTT, Kafka, and Event Hub.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to [e-mail].

Skills and Requirements

• 3+ years in data engineering or big data development
• Strong Databricks + Apache Spark experience (PySpark, Spark SQL, Delta Lake)
• Azure Data Factory + Azure Data Lake expertise
• SQL & Python proficiency for transformation and validation
• API integration experience using requests/http
• Delta Lake architecture knowledge including performance tuning
• Experience with real-time and batch design patterns
• Comfort with Git, CI/CD, and Agile delivery Unity Catalog governance
• dbt, Azure Synapse, or Microsoft Fabric
• Azure/Databricks certifications
• ML/IoT familiarity for anomaly detection and predictive modeling