Big Data Engineer (Live Anywhere)

Visionaire Partners, LLC.; Atlanta (GA); Hybrid; USD 110,000 - 140,000; Full time

Benefits offered by this job

401(k) match

Medical insurance

Dental insurance

Vision insurance

Life insurance

Short/long-term disability

Hospital indemnity

Accident coverage

Medical/Dependent Care FSAs

Job summary

Visionaire Partners, LLC. seeks a Big Data Engineer to join a dynamic team optimizing a Lakehouse-focused analytics platform.

You will design, build, and govern scalable streaming and ingestion pipelines, collaborating with cross-functional partners to translate requirements into robust architectures. Remote-friendly for non-local candidates (EST hours) with an Atlanta-based hybrid option, the role offers a six-plus month auto-renewing contract and a comprehensive benefits package for

Qualifications

• 5+ years in Data Engineering; strong proficiency in SQL, Spark, and Databricks.
• Hands-on experience with Delta Lake, Delta Live Tables, Unity Catalog, Kafka, and AWS analytics services.
• Background in Agile delivery and translating complex requirements into reliable models.
• Preferred: Exposure to Snowflake, NoSQL databases (e.g., Cassandra), or enterprise messaging systems.

Responsibilities

• Design, build, and maintain production ETL/ELT pipelines using Databricks, Apache Spark (PySpark/Spark SQL), and Delta Lake.
• Implement Delta Live Tables (DLT) for streaming/batch workflows and set up data governance via Unity Catalog.
• Integrate high-volume event streams using Apache Kafka and leverage AWS cloud infrastructure (S3, IAM).
• Cleanse, transform, and curate large structured and semi-structured datasets to support BI reporting and machine learning needs.
• Enforce data quality, operational observability, and engineering best practices in an Agile environment.

Skills

Data engineering

SQL

Spark

Databricks

Delta Lake

Delta Live Tables

Unity Catalog

Apache Kafka

AWS analytics services

Agile delivery

Tools

Kafka

Snowflake

Cassandra

Job description

Big Data Engineer Join a dynamic team optimizing a modern, Lakehouse-focused analytics platform. This position blends strategic business insight with hands-on technical execution. You will collaborate with cross-functional partners to gather requirements, design pragmatic architectures, and deploy scalable streaming and ingestion pipelines end-to-end.

Big Data Engineer Join a dynamic team optimizing a modern, Lakehouse-focused analytics platform. This position blends strategic business insight with hands-on technical execution. You will collaborate with cross-functional partners to gather requirements, design pragmatic architectures, and deploy scalable streaming and ingestion pipelines end-to-end.

Responsibilities

• Design, build, and maintain production ETL/ELT pipelines using Databricks, Apache Spark (PySpark/Spark SQL), and Delta Lake.
• Implement Delta Live Tables (DLT) for streaming/batch workflows and set up data governance via Unity Catalog.
• Integrate high-volume event streams using Apache Kafka and leverage AWS cloud infrastructure (S3, IAM).
• Cleanse, transform, and curate large structured and semi-structured datasets to support BI reporting and machine learning needs.
• Enforce data quality, operational observability, and engineering best practices in an Agile environment.

Qualifications

• 5+ years in Data Engineering; strong proficiency in SQL, Spark, and Databricks.
• Hands-on experience with Delta Lake, Delta Live Tables, Unity Catalog, Kafka, and AWS analytics services.
• Background in Agile delivery and translating complex requirements into reliable models.
• Preferred: Exposure to Snowflake, NoSQL databases (e.g., \"Cassandra\"), or enterprise messaging systems.

Location & Work Schedule:

This is a 6+ month auto-renewing contract. Open to 100% remote work for non-local candidates willing to work EST hours. Atlanta-based candidates will work a hybrid schedule (3 days in-office).Through Visionaire Partners, full-time W2 contractors gain access to a full benefits package (for contractors, spouses, domestic partners, and dependents), including a 401(k) with up to 4% match, medical, dental, vision, life insurance, short/long-term disability, critical illness, hospital indemnity, accident coverage and Medical/Dependent Care FSAs.