Azure Data Engineer: Spark, Databricks & Cloud Pipelines

Compunnel, Inc.; San Jose (CA); On-site; USD 120,000 - 180,000; Full time

Job summary

Compunnel, Inc. is seeking an experienced Data Engineer to design, implement, and optimize cloud-based data solutions on Azure.

This role focuses on scalable pipelines, modern data platforms, and distributed computing with Spark, Databricks, and SQL. You will build and maintain Azure Databricks workflows, Data Lake architectures, and data models to support enterprise analytics, while producing architecture diagrams and documentation and collaborating with cross-functional teams in San Jose,

Qualifications

• Bachelor's degree in CS or related field.
• 5+ years hands-on data engineering with Spark, MapReduce, or Databricks.
• Experience designing cloud-based data solutions on Microsoft Azure.
• Strong Python and SQL skills.
• Experience with Azure Databricks, Data Lake, and cloud-native services.
• Ability to create architecture diagrams and documentation.

Responsibilities

• Design scalable cloud-based data pipelines, data warehouses, and data lakes.
• Develop cloud-native data architectures aligned with business objectives.
• Integrate Azure data services, including Azure Databricks and Data Lake.
• Build Spark-based ETL pipelines in Databricks.
• Develop data models to support analytics and reporting.
• Produce architecture diagrams and technical documentation.
• Monitor and optimize data pipeline performance and reliability.
• Collaborate with cross-functional teams to deliver scalable cloud-based data solutions.
• Ensure data security and governance best practices.

Skills

Distributed computing

Data modeling

Cloud architecture

Python

SQL

Documentation

Education

Bachelor's degree in Computer Science or related field

Tools

Azure Databricks

Databricks

Spark

Azure Data Lake

Hadoop

Visio/Lucidchart

Job description

Compunnel, Inc. is seeking an experienced Data Engineer to design, implement, and optimize cloud-based data solutions on Azure.

This role focuses on scalable pipelines, modern data platforms, and distributed computing with Spark, Databricks, and SQL. You will build and maintain Azure Databricks workflows, Data Lake architectures, and data models to support enterprise analytics, while producing architecture diagrams and documentation and collaborating with cross-functional teams in San Jose,