Data Engineer

Compunnel, Inc.; Pleasanton, Northern (CA, KY); Hybrid; USD 120,000 - 160,000; Full time

Job summary

Compunnel, Inc. is seeking a Data Engineer with strong hands-on Databricks experience to support migrating Oracle PL/SQL ETL processes to Microsoft Azure Databricks in Pleasanton, CA.

You will design and operate Databricks data platforms and contribute to architecture decisions, not just rebuild pipelines and data transformations. You will develop scalable data pipelines, migrate complex Oracle ETL logic, ensure data quality, and support modern enterprise data platforms in a collaborative,

Qualifications

• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field.
• 4–6 years of experience in Databricks data engineering.
• 3+ years of experience developing data pipelines using Databricks on Microsoft Azure.
• 4+ years of advanced SQL development and query optimization.
• 3+ years of Oracle PL/SQL development.
• Hands-on experience migrating Oracle or other relational database ETL processes to cloud-based data platforms.
• Strong experience with Spark SQL, PySpark, Delta Lake, and Azure Data Lake Storage (ADLS).
• Hands-on experience with the Databricks platform and its architecture and capabilities.
• Experience making technical or architectural decisions within Databricks environments.
• Experience with Git, source control, and CI/CD deployment processes.
• Strong understanding of data pipeline development, data transformation, validation, and reconciliation.
• Strong analytical, troubleshooting, and problem-solving skills.
• Excellent communication skills and ability to work independently and collaboratively.

Responsibilities

• Analyze existing Oracle SQL and PL/SQL ETL processes and migrate them to Databricks using Spark SQL and/or PySpark.
• Design, develop, test, deploy, and maintain scalable data pipelines on Microsoft Azure Databricks.
• Analyze and convert complex Oracle PL/SQL transformation logic, including packages, procedures, and functions, into scalable Databricks pipelines.
• Build robust data pipelines to process, validate, transport, collate, aggregate, and distribute data.
• Design workflows that enable analysts and data teams to efficiently access and use data.
• Implement incremental data processing strategies using Databricks and Delta Lake.
• Perform data validation and reconciliation between Oracle source systems and Databricks target environments.
• Validate migrated data with business users and subject matter experts to ensure business-level accuracy.
• Develop data transformation and movement processes while balancing performance, scalability, and cost objectives.
• Establish processes and controls to maintain data integrity throughout data pipelines and warehouses.
• Develop automated tests to validate data transfer integrity, quality, and efficiency.
• Support existing production ETL processes during the migration and provide post-migration support.
• Monitor, troubleshoot, and enhance Databricks data pipelines and production processes.
• Optimize pipeline performance and perform root cause analysis for production issues.
• Contribute to Databricks architecture and platform decisions and provide technical recommendations.
• Collaborate with software developers, database architects, data analysts, data scientists, and other data engineers.
• Work with business users and stakeholders to gather requirements and translate them into technical solutions.
• Participate in source control, CI/CD, release management, and production support activities.
• Create and maintain technical design documents, data mappings, and system documentation.
• Take ownership of assigned projects, manage priorities, and deliver high-quality solutions on schedule.

Skills

Databricks

Spark SQL

PySpark

Azure Databricks

Oracle PL/SQL

ETL pipelines

SQL development

Delta Lake

ADLS

Git

CI/CD

Data architecture

Communication

Education

Bachelor's degree

Tools

Git

CI/CD

Delta Lake

ADLS

Job description

Job Summary
We are seeking a Data Engineer with strong hands-on Databricks experience to support the migration of enterprise Oracle PL/SQL-based ETL processes to Microsoft Azure Databricks. The ideal candidate will have experience designing and operating Databricks data platforms and be able to work in a consultative capacity, contributing to architecture and platform decisions rather than only rebuilding pipelines and data transformations. The role will focus on developing scalable data pipelines, migrating complex Oracle ETL logic, ensuring data quality, and supporting modern enterprise data platforms.

Key Responsibilities

• Analyze existing Oracle SQL and PL/SQL ETL processes and migrate them to Databricks using Spark SQL and/or PySpark.
• Design, develop, test, deploy, and maintain scalable data pipelines on Microsoft Azure Databricks.
• Analyze and convert complex Oracle PL/SQL transformation logic, including packages, procedures, and functions, into scalable Databricks pipelines.
• Build robust data pipelines to process, validate, transport, collate, aggregate, and distribute data.
• Design workflows that enable analysts and data teams to efficiently access and use data.
• Implement incremental data processing strategies using Databricks and Delta Lake.
• Perform data validation and reconciliation between Oracle source systems and Databricks target environments.
• Validate migrated data with business users and subject matter experts to ensure business-level accuracy.
• Develop data transformation and movement processes while balancing performance, scalability, and cost objectives.
• Establish processes and controls to maintain data integrity throughout data pipelines and warehouses.
• Develop automated tests to validate data transfer integrity, quality, and efficiency.
• Support existing production ETL processes during the migration and provide post-migration support.
• Monitor, troubleshoot, and enhance Databricks data pipelines and production processes.
• Optimize pipeline performance and perform root cause analysis for production issues.
• Contribute to Databricks architecture and platform decisions and provide technical recommendations.
• Collaborate with software developers, database architects, data analysts, data scientists, and other data engineers.
• Work with business users and stakeholders to gather requirements and translate them into technical solutions.
• Participate in source control, CI/CD, release management, and production support activities.
• Create and maintain technical design documents, data mappings, and system documentation.
• Take ownership of assigned projects, manage priorities, and deliver high-quality solutions on schedule.

Required Qualifications

• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field.
• 4–6 years of experience in Databricks data engineering.
• 3+ years of experience developing data pipelines using Databricks on Microsoft Azure.
• 4+ years of advanced SQL development and query optimization.
• 3+ years of Oracle PL/SQL development.
• Hands-on experience migrating Oracle or other relational database ETL processes to cloud-based data platforms.
• Strong experience with Spark SQL, PySpark, Delta Lake, and Azure Data Lake Storage (ADLS).
• Hands-on experience with the Databricks platform and its architecture and capabilities.
• Experience making technical or architectural decisions within Databricks environments.
• Experience with Git, source control, and CI/CD deployment processes.
• Strong understanding of data pipeline development, data transformation, validation, and reconciliation.
• Strong analytical, troubleshooting, and problem-solving skills.
• Excellent communication skills and ability to work independently and collaboratively.