Databricks Engineer: Scale Data Pipelines on Lakehouse

iLink Digital; Milpitas, Northern (CA, KY); Hybrid; USD 120,000 - 160,000; Full time

Job summary

iLink Digital in Milpitas, CA is seeking a skilled Databricks Engineer to design, build, and optimize data pipelines on the Databricks Lakehouse Platform. You will implement Medallion Architecture and work across data sources to deliver scalable ETL/ELT solutions.

The role requires PySpark, Spark SQL, Python, and SQL expertise, with collaboration across data architects, scientists, and stakeholders. Experience with CI/CD, data governance, and cloud platforms is highly valued.

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
• Experience designing and optimizing data pipelines on Databricks Lakehouse.
• Proficiency with PySpark, Spark SQL, Python and SQL.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks.
• Build ETL/ELT workflows for batch and streaming data processing.
• Develop solutions using PySpark, Spark SQL, and Delta Lake.
• Implement Medallion Architecture (Bronze, Silver, Gold) for data transformation.
• Integrate data from various sources including relational databases, APIs, cloud storage, and streaming platforms.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Collaborate with Data Architects, Data Scientists, BI developers, and business stakeholders.
• Implement CI/CD pipelines and deployment automation for Databricks workloads.
• Ensure data quality, security, governance, and compliance.
• Monitor, troubleshoot, and optimize production data pipelines.
• Document technical solutions and follow engineering best practices.

Education

Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field

Tools

Databricks Lakehouse Platform

PySpark

Spark SQL

Python

SQL

Delta Lake

Delta Live Tables

Unity Catalog

Databricks Workflows

Terraform

MLflow

Azure/AWS/GCP

Job description

iLink Digital in Milpitas, CA is seeking a skilled Databricks Engineer to design, build, and optimize data pipelines on the Databricks Lakehouse Platform. You will implement Medallion Architecture and work across data sources to deliver scalable ETL/ELT solutions.

The role requires PySpark, Spark SQL, Python, and SQL expertise, with collaboration across data architects, scientists, and stakeholders. Experience with CI/CD, data governance, and cloud platforms is highly valued.