Databricks Engineer

iLink Digital; Milpitas, Northern (CA, KY); Hybrid; USD 120,000 - 160,000; Full time

Job summary

iLink Digital in Milpitas, CA is seeking a skilled Databricks Engineer to design, build, and optimize data pipelines on the Databricks Lakehouse Platform. You will implement Medallion Architecture and work across data sources to deliver scalable ETL/ELT solutions.

The role requires PySpark, Spark SQL, Python, and SQL expertise, with collaboration across data architects, scientists, and stakeholders. Experience with CI/CD, data governance, and cloud platforms is highly valued.

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
• Experience designing and optimizing data pipelines on Databricks Lakehouse.
• Proficiency with PySpark, Spark SQL, Python and SQL.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks.
• Build ETL/ELT workflows for batch and streaming data processing.
• Develop solutions using PySpark, Spark SQL, and Delta Lake.
• Implement Medallion Architecture (Bronze, Silver, Gold) for data transformation.
• Integrate data from various sources including relational databases, APIs, cloud storage, and streaming platforms.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Collaborate with Data Architects, Data Scientists, BI developers, and business stakeholders.
• Implement CI/CD pipelines and deployment automation for Databricks workloads.
• Ensure data quality, security, governance, and compliance.
• Monitor, troubleshoot, and optimize production data pipelines.
• Document technical solutions and follow engineering best practices.

Education

Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field

Tools

Databricks Lakehouse Platform

PySpark

Spark SQL

Python

SQL

Delta Lake

Delta Live Tables

Unity Catalog

Databricks Workflows

Terraform

MLflow

Azure/AWS/GCP

Job description

Milpitas, United States | Posted on 07/22/2026

We are looking for a skilled Databricks Engineer with strong expertise in designing, developing, and optimizing modern data engineering solutions on the Databricks Lakehouse Platform. The ideal candidate should have experience building scalable ETL/ELT pipelines, working with large-scale data, and leveraging Apache Spark to deliver high-performance data solutions.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks.
  
• Build ETL/ELT workflows for batch and streaming data processing.
  
• Develop solutions using PySpark, Spark SQL, and Delta Lake.
  
• Implement Medallion Architecture (Bronze, Silver, Gold) for data transformation.
  
• Integrate data from various sources including relational databases, APIs, cloud storage, and streaming platforms.
  
• Optimize Spark jobs for performance, scalability, and cost efficiency.
  
• Collaborate with Data Architects, Data Scientists, BI developers, and business stakeholders.
  
• Implement CI/CD pipelines and deployment automation for Databricks workloads.
  
• Ensure data quality, security, governance, and compliance.
  
• Monitor, troubleshoot, and optimize production data pipelines.
  
• Document technical solutions and follow engineering best practices.
  

Required Skills

Core Technologies

• Databricks Lakehouse Platform
  
• PySpark
  
• Spark SQL
  
• Python
  
• SQL
  

Cloud Platforms (one or more)

• AWS
  
• Data Warehousing
  
• Data Modeling
  
• ETL/ELT Development
  
• Batch Processing
  
• Data Lake Architecture
  
• Git
  

Preferred Qualifications

• Experience with Unity Catalog.
  
• Knowledge of Databricks Workflows and Jobs.
  
• Hands-on experience with Delta Live Tables (DLT).
  
• Exposure to MLflow is an added advantage.
  
• Experience with data governance and security best practices.
  
• Familiarity with Infrastructure as Code (Terraform) is a plus.
  

Educational Qualification

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
  

Preferred Certifications

Good to Have

• Experience with real-time analytics.
  
• Knowledge of Lakehouse architecture.
  
• Experience with Agile/Scrum methodologies.
  
• Strong analytical and problem-solving skills.
  
• Excellent communication and stakeholder management abilities.
  

Mandatory Skills

• Databricks
  
• PySpark
  
• Spark SQL
  
• Python
  
• SQL
  
• Azure/AWS/GCP (at least one cloud platform)