Remote Big Data Engineer: Spark & Databricks Lakehouse

Visionaire Partners, LLC.; Atlanta (GA); Hybrid; USD 110,000 - 140,000; Full time

Benefits offered by this job

401(k) match

Medical insurance

Dental insurance

Vision insurance

Life insurance

Short/long-term disability

Hospital indemnity

Accident coverage

Medical/Dependent Care FSAs

Job summary

Visionaire Partners, LLC. seeks a Big Data Engineer to join a dynamic team optimizing a Lakehouse-focused analytics platform.

You will design, build, and govern scalable streaming and ingestion pipelines, collaborating with cross-functional partners to translate requirements into robust architectures. Remote-friendly for non-local candidates (EST hours) with an Atlanta-based hybrid option, the role offers a six-plus month auto-renewing contract and a comprehensive benefits package for

Qualifications

• 5+ years in Data Engineering; strong proficiency in SQL, Spark, and Databricks.
• Hands-on experience with Delta Lake, Delta Live Tables, Unity Catalog, Kafka, and AWS analytics services.
• Background in Agile delivery and translating complex requirements into reliable models.
• Preferred: Exposure to Snowflake, NoSQL databases (e.g., Cassandra), or enterprise messaging systems.

Responsibilities

• Design, build, and maintain production ETL/ELT pipelines using Databricks, Apache Spark (PySpark/Spark SQL), and Delta Lake.
• Implement Delta Live Tables (DLT) for streaming/batch workflows and set up data governance via Unity Catalog.
• Integrate high-volume event streams using Apache Kafka and leverage AWS cloud infrastructure (S3, IAM).
• Cleanse, transform, and curate large structured and semi-structured datasets to support BI reporting and machine learning needs.
• Enforce data quality, operational observability, and engineering best practices in an Agile environment.

Skills

Data engineering

SQL

Spark

Databricks

Delta Lake

Delta Live Tables

Unity Catalog

Apache Kafka

AWS analytics services

Agile delivery

Tools

Kafka

Snowflake

Cassandra

Job description

Visionaire Partners, LLC. seeks a Big Data Engineer to join a dynamic team optimizing a Lakehouse-focused analytics platform.

You will design, build, and govern scalable streaming and ingestion pipelines, collaborating with cross-functional partners to translate requirements into robust architectures. Remote-friendly for non-local candidates (EST hours) with an Atlanta-based hybrid option, the role offers a six-plus month auto-renewing contract and a comprehensive benefits package for