Databricks Subject Matter Expert

Virtue America; New Jersey; On-site; USD 180,000 - 240,000; Full time

Job summary

Virtue America in New Jersey seeks an experienced Databricks Subject Matter Expert to design, develop, and optimize enterprise-scale Lakehouse architectures and cloud-based data platforms. The role emphasizes Databricks, Apache Spark, Delta Lake, Unity Catalog, and data engineering across cloud environments, governance, and MLOps.

The ideal candidate has 10+ years in data engineering or architecture and 5+ years hands-on Databricks experience.

Qualifications

• 10+ years of experience in data engineering, analytics, or platform architecture.
• 5+ years of hands-on experience with Databricks.
• Strong expertise in Apache Spark, Delta Lake, Unity Catalog, PySpark/Scala, and SQL.
• Experience designing and implementing enterprise-scale Lakehouse and cloud data platforms.
• Knowledge of data governance, data quality, lineage, and performance optimization.
• Experience with Databricks Workflows, MLflow, MLOps, and model deployment.
• Hands-on experience with at least one major cloud platform: Azure, AWS, or GCP.
• Strong client-facing communication, technical leadership, and problem-solving skills.

Responsibilities

• Design and implement Lakehouse architectures using Delta Lake, Unity Catalog, and Bronze/Silver/Gold medallion architecture.
• Develop and optimize ETL/ELT pipelines using Apache Spark, PySpark, Scala, SQL, and Databricks Workflows.
• Implement data governance, security, data quality, lineage, and access control policies.
• Optimize Spark workloads, data processing performance, and resource utilization.
• Support cloud-based data solutions using Azure Databricks, AWS, or Google Cloud Platform (GCP).
• Implement MLflow, MLOps workflows, model deployment, and production data solutions.
• Establish best practices for data architecture, pipeline development, monitoring, and troubleshooting.
• Lead architecture reviews, technical discussions, and complex problem-solving initiatives.
• Collaborate with clients, data engineers, architects, and business stakeholders to deliver scalable data solutions.
• Mentor engineering teams and provide technical guidance on Databricks technologies and platform best practices.

Skills

Databricks

Apache Spark

Delta Lake

Unity Catalog

Data engineering

Cloud environments

Data governance

MLOps

PySpark

Scala

SQL

MLflow

Tools

Databricks Workflows

MLflow

Azure Databricks

AWS

GCP

Job description

Job Title: Databricks Subject Matter Expert (SME)

Location: New Jersey, NJ

Experience: 10+ Years

Databricks Experience: 5+ Years

Job Summary:

We are seeking an experienced Databricks SME to design, develop, and optimize enterprise-scale Lakehouse architectures and cloud-based data platforms. The ideal candidate will have strong expertise in Databricks, Apache Spark, Delta Lake, Unity Catalog, and data engineering, along with experience in cloud environments, data governance, and MLOps.

Key Responsibilities:

• Design and implement Lakehouse architectures using Delta Lake, Unity Catalog, and Bronze/Silver/Gold medallion architecture.
• Develop and optimize ETL/ELT pipelines using Apache Spark, PySpark, Scala, SQL, and Databricks Workflows.
• Implement data governance, security, data quality, lineage, and access control policies.
• Optimize Spark workloads, data processing performance, and resource utilization.
• Support cloud-based data solutions using Azure Databricks, AWS, or Google Cloud Platform (GCP).
• Implement MLflow, MLOps workflows, model deployment, and production data solutions.
• Establish best practices for data architecture, pipeline development, monitoring, and troubleshooting.
• Lead architecture reviews, technical discussions, and complex problem-solving initiatives.
• Collaborate with clients, data engineers, architects, and business stakeholders to deliver scalable data solutions.
• Mentor engineering teams and provide technical guidance on Databricks technologies and platform best practices.

Required Qualifications:

• 10+ years of experience in data engineering, analytics, or platform architecture.
• 5+ years of hands-on experience with Databricks.
• Strong expertise in Apache Spark, Delta Lake, Unity Catalog, PySpark/Scala, and SQL.
• Experience designing and implementing enterprise-scale Lakehouse and cloud data platforms.
• Knowledge of data governance, data quality, lineage, and performance optimization.
• Experience with Databricks Workflows, MLflow, MLOps, and model deployment.
• Hands-on experience with at least one major cloud platform: Azure, AWS, or GCP.
• Strong client-facing communication, technical leadership, and problem-solving skills.

Preferred Qualifications:

• Experience with large-scale data migration and platform modernization.
• Knowledge of CI/CD, infrastructure automation, and automated data testing.
• Experience implementing secure, scalable, and production-ready enterprise data solutions.