Databricks Lakehouse Architect & Platform Lead

Valiant Technologies LLC; Boston (MA); On-site; USD 180,000 - 260,000; Full time

Job summary

Valiant Technologies LLC seeks an experienced Databricks expert to design scalable, secure data and AI platforms on the Databricks Lakehouse. The role will define target architecture, establish engineering standards, and guide data ingestion, transformation, governance, performance optimization, and production operations.

The candidate will lead architecture reviews, drive cost optimization, and establish CI/CD and monitoring best practices while partnering with stakeholders and mentoring

Qualifications

• 12–15 years of data engineering, integration, or cloud/data architecture.
• Hands-on Databricks experience with cloud architectures.
• Strong knowledge of Spark, Delta Lake, Unity Catalog, SQL and Python.
• Experience designing enterprise-scale data and AI platforms.
• Security, governance, performance tuning, and cost optimization familiarity.
• CI/CD and production monitoring experience.
• Excellent communication and stakeholder management.

Responsibilities

• Design scalable and secure Databricks Lakehouse architectures.
• Define target architecture, engineering standards, and reusable frameworks.
• Lead data ingestion, transformation, governance, and production operations.
• Conduct architecture reviews and technology evaluations.
• Drive performance and cost optimization across the platform.
• Establish CI/CD, monitoring, and operational best practices.
• Partner with client stakeholders, data engineers, analysts, security teams, and vendors to translate business requirements into technical solutions.
• Provide technical leadership and mentoring to engineering teams.

Skills

Databricks

Cloud architecture

Apache Spark

Delta Lake

Unity Catalog

SQL

Python

Data governance

Performance tuning

CI/CD

Production monitoring

Stakeholder management

Job description

Valiant Technologies LLC seeks an experienced Databricks expert to design scalable, secure data and AI platforms on the Databricks Lakehouse. The role will define target architecture, establish engineering standards, and guide data ingestion, transformation, governance, performance optimization, and production operations.

The candidate will lead architecture reviews, drive cost optimization, and establish CI/CD and monitoring best practices while partnering with stakeholders and mentoring