Senior Databricks Data Engineer

Virtue America; New York (NY); On-site; USD 140,000 - 190,000; Full time

Job summary

Virtue America in New York City is seeking a Senior Databricks Data Engineer to design, build, and optimize scalable data pipelines and Lakehouse solutions across Azure, AWS, or GCP.

You will work with Databricks, Spark, PySpark, Python, SQL, Delta Lake, and Unity Catalog to enable reliable batch and streaming processing, strong data governance, and trusted analytics.

Qualifications

• Experience with Databricks and enterprise data engineering.
• Strong Spark, PySpark, Python, and SQL skills.
• Experience with Delta Lake, Unity Catalog, and Lakehouse architecture.
• Experience with cloud platforms (Azure, AWS, or GCP).

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines.
• Build batch and real-time streaming data workflows.
• Implement Lakehouse architecture using Delta Lake and Unity Catalog.
• Develop data transformations using Python and SQL.
• Model data for analytics, reporting, and BI.
• Ensure data quality, monitoring, and performance optimization.
• Govern data access with Unity Catalog and data lineage.
• Deploy solutions across cloud environments (Azure/AWS/GCP).
• Implement versioning and CI/CD deployments.
• Collaborate with architects, analysts, and cross-functional teams.

Skills

Databricks

Apache Spark

PySpark

Python

SQL

Delta Lake

Unity Catalog

ETL/ELT

Data modeling

Cloud platforms

Tools

Git

CI/CD pipelines

Azure

AWS

GCP

Job description

Job Title: Senior Databricks Data Engineer

Location: New York City, NY

Job Summary:

We are seeking an experienced Senior Databricks Data Engineer to design, develop, and optimize scalable data pipelines and cloud-based Lakehouse solutions. The ideal candidate will have strong expertise in Databricks, Apache Spark, PySpark, Python, SQL, Delta Lake, and Unity Catalog, with experience building reliable batch and streaming data processing solutions.

Key Responsibilities:

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, Apache Spark, and PySpark.
• Build and optimize batch and real-time streaming data processing workflows.
• Implement Lakehouse architecture using Delta Lake and Unity Catalog.
• Develop data transformation and integration solutions using Python and SQL.
• Design and maintain data models to support analytics, reporting, and business intelligence.
• Implement data quality checks, validation, monitoring, and performance optimization.
• Manage data governance, access controls, and data lineage using Unity Catalog.
• Deploy and maintain data engineering solutions across Azure, AWS, or GCP environments.
• Implement version control and automated deployment processes using Git and CI/CD pipelines.
• Troubleshoot pipeline failures and improve data reliability, scalability, and processing efficiency.
• Collaborate with data architects, analysts, and cross-functional teams to deliver business data solutions.

Required Qualifications:

• Strong hands-on experience with Databricks and enterprise data engineering.
• Proficiency in Apache Spark, PySpark, Python, and SQL.
• Experience with Delta Lake, Unity Catalog, and Lakehouse architecture.
• Strong knowledge of ETL/ELT processes and data pipeline development.
• Experience with batch processing and streaming data architectures.
• Understanding of data modeling, data warehousing, and data integration.
• Hands-on experience with at least one cloud platform: Azure, AWS, or GCP.
• Experience with Git, CI/CD pipelines, and software development best practices.
• Strong analytical, problem-solving, and communication skills.

Preferred Qualifications:

• Knowledge of data governance, security, and data quality frameworks.
• Familiarity with workflow orchestration, automated testing, and production monitoring.
• Experience supporting enterprise analytics and cloud data platform modernization.