DataBricks Solution Architect

Veriipro; Charlotte (NC); On-site; USD 120,000 - 160,000; Full time

Job summary

Veriipro is seeking a Databricks Solutions Architect to assess and optimize enterprise data pipelines. You will review the current Databricks architecture and how applications consume data, identifying bottlenecks and opportunities to improve performance, cost, security, and governance.

You will deliver an optimization roadmap, perform deep code reviews of PySpark, Spark SQL, notebooks, SQL queries, and Delta Lake implementations, and help shape scalable data solutions across streaming and batch

Qualifications

• Experience reviewing Databricks architecture.
• Ability to optimize performance, cost, and governance.
• Proven experience with data consumption patterns.
• Able to deliver an optimization roadmap.

Responsibilities

• Architecture assessment: review Databricks architecture and data flows.
• Understand how applications consume Databricks data.
• Identify bottlenecks and unnecessary complexity.
• Code review: PySpark, Spark SQL, notebooks and SQL queries.
• Evaluate Delta Lake implementation and streaming code.

Skills

Data consumption

Architecture review

Performance optimization

Security governance

Roadmap delivery

Databricks architect

Tools

Databricks

Spark SQL

Delta Lake

Job description

Required Skills:

• Understanding how application consume data
• Need someone who can come in look at the architecture and code and offer optimization solution
• Focus on performance, cost, security and governance
• Deliver an optimization roadmap
• DataBricks Solutions architect on the application side.
• Someone who has experience with consumption of data

Responsibilities:

1. Architecture assessment

• Review the existing Databricks architecture
• Understand how applications consume Databricks data
• Review data flows, integrations, APIs, SQL Warehouses, clusters, Delta tables, etc.
• Identify architectural bottlenecks and unnecessary complexity

2. Code review

This is critical. They should be comfortable getting into the actual:

• PySpark code
• Spark SQL
• Databricks notebooks
• SQL queries
• ETL/ELT pipelines
• Delta Lake implementation
• Streaming code,