Azure Databricks Data Engineer - Oracle to Modern Pipelines

Insight Global; San Francisco (CA); On-site; USD 120,000 - 180,000; Full time

Job summary

Insight Global is seeking a Data Engineer to modernize enterprise data platforms using Databricks on Microsoft Azure. You will migrate Oracle PL/SQL ETL processes to Databricks pipelines and design scalable data workflows with data quality in mind.

You will work with stakeholders to gather requirements, translate them into technical solutions, and ensure smooth migration while supporting production ETL operations and ongoing enhancements.

Qualifications

• 4–6 years of ETL, database development, or data engineering.
• 4+ years advanced SQL development and optimization.
• 3+ years Databricks on Microsoft Azure development.
• Experience migrating legacy ETL to cloud data platforms.

Responsibilities

• Migrate Oracle ETL to Databricks pipelines.
• Design, test, deploy scalable data pipelines on Azure Databricks.
• Validate migrated data with business users during migration.
• Implement incremental processing with Delta Lake.
• Support production ETL during migration and optimization.
• Collaborate on source control, CI/CD, and release management.

Skills

ETL development

SQL optimization

Data pipeline design

Python/PySpark

Education

Bachelor's degree in CS/IS/Engineering

Tools

Databricks on Azure

Databricks SQL

Spark SQL

PySpark

Delta Lake

Azure Data Lake Storage

Git / CI/CD

Job description

Insight Global is seeking a Data Engineer to modernize enterprise data platforms using Databricks on Microsoft Azure. You will migrate Oracle PL/SQL ETL processes to Databricks pipelines and design scalable data workflows with data quality in mind.

You will work with stakeholders to gather requirements, translate them into technical solutions, and ensure smooth migration while supporting production ETL operations and ongoing enhancements.