Senior Databricks Architect: Lakehouse & AI Pipelines

Siri InfoSolutions Inc; Malvern (Chester County); On-site; USD 140,000 - 210,000; Full time

Job summary

Siri InfoSolutions Inc. is seeking a Senior Databricks Engineer in Malvern, PA to lead end-to-end data engineering on the Databricks Lakehouse. You will build batch and streaming pipelines, optimize Spark/SQL code, and manage lakehouse governance.

Candidate will design resilient ETL/ELT processes, collaborate with data scientists and BI teams, and optimize compute costs on AWS using Spot Instances and CI/CD practices.

Qualifications

• Mastery of Spark SQL and the DataFrame API for distributed data processing.
• Experience with Delta Lake medallion architecture and ACID transactions.
• Proficient with Delta Live Tables, Workflows, and ELT/ETL pipelines.
• Strong knowledge of Unity Catalog for data governance and security.
• Advanced Python (PySpark) and SQL skills for data processing.
• Experience reading/writing to S3 and integrating with AWS data warehouses.

Responsibilities

• Develop batch and streaming data pipelines from multiple sources.
• Optimize PySpark/SQL code and troubleshoot distributed workloads.
• Manage Delta Lake lakehouse layers Bronze/Silver/Gold.
• Automate workflows using Databricks Jobs/Delta Live Tables.
• Implement data governance and security controls with Unity Catalog.
• Collaborate with data scientists, analysts, and ML engineers.

Skills

Spark SQL

DataFrame API

Delta Lake

DLT/Pipelines

Databricks Workflows

Python (PySpark)

SQL

S3 / AWS integration

IAM & KMS

Cluster cost optimization

Git & CI/CD

MLflow

GenAI / Vector search

Tools

Databricks Lakehouse

Delta Live Tables

Unity Catalog

AWS Glue

Kinesis

Job description

Siri InfoSolutions Inc. is seeking a Senior Databricks Engineer in Malvern, PA to lead end-to-end data engineering on the Databricks Lakehouse. You will build batch and streaming pipelines, optimize Spark/SQL code, and manage lakehouse governance.

Candidate will design resilient ETL/ELT processes, collaborate with data scientists and BI teams, and optimize compute costs on AWS using Spot Instances and CI/CD practices.