Databricks Architect

VDart Inc; United States; Hybrid; USD 120,000 - 180,000; Full time

Job summary

VDart Inc. is seeking an experienced Databricks Architect in Bolingbrook, IL with remote options for a 12-month engagement.

The role focuses on designing, implementing, and governing enterprise-scale data platforms using Databricks to support real-time data processing and analytics initiatives. The candidate will lead data architecture, cloud integration, governance, security, and platform optimization, while mentoring data engineering teams and collaborating with BI tools such as Power BI and

Qualifications

• 8+ years of experience in data engineering, data architecture, or enterprise data platforms.
• 3+ years of hands-on experience designing and architecting Databricks solutions.
• Experience designing data lakehouse architectures and enterprise data governance.
• Hands-on experience with cloud platforms: Azure Databricks, AWS Databricks, or GCP Databricks.
• Experience with streaming technologies: Kafka, Pub/Sub, Event Hubs, or Kinesis.

Responsibilities

• Design and implement scalable data lakehouse architectures using Databricks.
• Define enterprise data models, architecture standards, governance frameworks, and security controls.
• Lead the migration of legacy data warehouses, ETL processes, and data workloads to Databricks.
• Architect and implement batch and real-time data processing solutions using Apache Spark and PySpark.
• Design producer/consumer architectures for streaming and event-driven data platforms.
• Evaluate and define appropriate streaming technologies (Kafka, Pub/Sub, Event Hubs, Kinesis).
• Optimize data platforms for performance, scalability, reliability, and cost efficiency.
• Develop data integration and orchestration solutions; establish CI/CD and DataOps practices.

Skills

Databricks

Apache Spark

PySpark

SQL

Delta Lake

Azure Databricks

AWS Databricks

GCP Databricks

Streaming

Kafka

Power BI

Tableau

Data governance

Education

Bachelor’s degree in a related field

Master’s degree preferred

Job description

Role: Databricks Architect

Location: Bolingbrook, IL Remote

Duration: 12 Months

Job Summary:

• We are seeking an experienced Databricks Architect to design, implement, and govern enterprise-scale data platforms using Databricks. The ideal candidate will have strong expertise in Databricks, Apache Spark, PySpark, SQL, Delta Lake, cloud platforms, data lakehouse architecture, and real-time data processing.
• The Databricks Architect will provide technical leadership across data architecture, cloud integration, data engineering, governance, security, analytics enablement, and platform optimization.

Key Responsibilities:

• Design and implement scalable data lakehouse architectures using Databricks.
• Define enterprise data models, architecture standards, governance frameworks, and security controls.
• Lead the migration of legacy data warehouses, ETL processes, and data workloads to Databricks.
• Architect and implement batch and real-time data processing solutions using Apache Spark and PySpark.
• Design producer/consumer architectures for streaming and event-driven data platforms.
• Evaluate and define appropriate streaming technologies, including Apache Kafka, GCP Pub/Sub, Azure Event Hubs, and AWS Kinesis.
• Optimize data platforms for performance, scalability, reliability, and cost efficiency.
• Develop and implement data integration and orchestration solutions.
• Establish CI/CD, DataOps, DevOps, monitoring, logging, and operational best practices.
• Implement data security, access management, governance, and compliance controls.
• Collaborate with business stakeholders, data engineers, data scientists, cloud architects, and analytics teams.
• Provide technical leadership and mentorship to data engineering teams.
• Support analytics and BI initiatives using platforms such as Power BI and Tableau.
• Evaluate emerging Databricks and cloud data technologies and recommend appropriate solutions.

Required Skills & Qualifications:

• 8+ years of experience in data engineering, data architecture, or enterprise data platforms.
• 3+ years of hands-on experience designing and architecting Databricks solutions.

Strong expertise in:

• Databricks
• Apache Spark
• PySpark
• SQL
• Delta Lake
• Hands-on experience with at least one major cloud platform:
• Azure / Azure Databricks
• AWS / AWS Databricks
• GCP / GCP Databricks
• Strong experience with real-time and streaming data architectures.
• Experience with one or more streaming technologies:
• Apache Kafka
• GCP Pub/Sub
• Azure Event Hubs
• AWS Kinesis

• Strong understanding of producer/consumer architecture, scalability, reliability, and technology trade-offs.
• Experience designing data lakehouse architectures and enterprise data governance.
• Hands-on experience with data integration tools and orchestration frameworks.
• Strong understanding of data security, identity/access management, compliance, and governance.
• Experience with CI/CD and automated deployment processes for data platforms.
• Strong communication, architecture documentation, problem-solving, and technical leadership skills.

Preferred Skills:

• Databricks Certified Data Engineer Professional certification or equivalent experience.
• Experience with Unity Catalog.
• Experience with Delta Live Tables (DLT).
• Experience with Databricks Workflows.
• Experience with Power BI, Tableau, or other BI platforms.
• Knowledge of DataOps, DevOps, and MLOps practices.
• Experience with cloud-native data services and enterprise data modernization initiatives.

Education:

• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field.
• Master's degree preferred.