Databricks Architect/ Senior Data Engineer

OZ; Boca Raton (FL); On-site; USD 120,000 - 160,000; Full time

Benefits offered by this job

Competitive compensation

Comprehensive benefits package

Remote work options

Job summary

A leading technology consultancy is looking for a Databricks Senior Data Engineer in Boca Raton, Florida. The ideal candidate should have over 8 years of experience in data engineering with a strong focus on Databricks and Apache Spark. Responsibilities include architecting scalable data platforms, optimizing data pipelines, and leading technical initiatives. If you're passionate about data and want to work in a collaborative environment, this is your chance to make an impact in a thriving organization.

Qualifications

• 8+ years of experience in data engineering with strong hands-on expertise in Databricks and Apache Spark.
• Proven experience designing and implementing scalable ETL/ELT pipelines in cloud environments.
• Hands-on experience with Databricks Lakehouse and Delta Lake.

Responsibilities

• Architect and design scalable data platforms and complex ETL/ELT workflows for Databricks Lakehouse.
• Write, test, and optimize code in Python, PySpark, and SQL.
• Implement CI/CD and automation for data pipelines.

Skills

Databricks

Python

SQL

Data Engineering

Apache Spark

Data Architecture

Education

Bachelor’s degree in Computer Science, Information Technology, Engineering

Tools

Azure Data Factory

Kafka

Job description

Overview

Note: Only applications from U.S. citizens or lawful permanent residents (Green Card Holders) will be considered.

We believe work should be innately rewarding and a team-building venture. Working with our teammates and clients should be an enjoyable journey where we can learn, grow as professionals, and achieve amazing results. Our core values revolve around this philosophy. We are relentlessly committed to helping our clients achieve their business goals, leapfrog the competition, and become leaders in their industry. What drives us forward is the culture of creativity combined with a disciplined approach, passion for learning & innovation, and a ‘can-do’ attitude!

What We're Looking For

We are seeking a highly experienced Databricks professional with deep expertise in data engineering, distributed computing, and cloud-based data platforms. The ideal candidate is both an architect and a hands-on engineer who can design scalable data solutions while actively contributing to development, optimization, and deployment.

This role requires strong technical leadership, a deep understanding of modern data architectures, and the ability to implement best practices in DataOps, performance optimization, and data governance.

Experience with modern AI/GenAI-enabled data platforms and real-time data processing environments is highly desirable.

Position Overview

The Databricks Senior Data Engineer will play a critical role in designing, implementing, and optimizing enterprise-scale data platforms using the Databricks Lakehouse architecture. This role combines architecture leadership with hands-on engineering, focusing on building scalable, secure, and high-performance data pipelines and platforms. The ideal candidate will establish coding standards, define data architecture frameworks such as the Medallion Architecture, and guide the end-to-end development lifecycle of modern data solutions.

This individual will collaborate with cross-functional stakeholders, including data engineers, BI developers, analysts, and business leaders, to deliver robust data platforms that enable advanced analytics, reporting, and AI-driven decision-making.

Key Responsibilities

• Architecture & Design: Architect and design scalable, reliable data platforms and complex ETL/ELT and streaming workflows for the Databricks Lakehouse Platform (Delta Lake, Spark).
• Hands-On Development: Write, test, and optimize code in Python, PySpark, and SQL for data ingestion, transformation, and processing.
• DataOps & Automation: Implement CI/CD, monitoring, and automation (e.g., with Azure DevOps, DBX) for data pipelines.
• Stakeholder Collaboration: Work with BI developers, analysts, and business users to define requirements and deliver data-driven solutions.
• Performance Optimization: Tune delta tables, Spark jobs, and SQL queries for maximum efficiency and scalability.
• GenAI Applications Development: It is a big plus to have experience in GenAI application development

Requirements

• 8+ years of experience in data engineering, with strong hands-on expertise in Databricks and Apache Spark.
• Proven experience designing and implementing scalable ETL/ELT pipelines in cloud environments.
• Strong programming skills in Python and SQL; experience with PySpark required.
• Hands-on experience with Databricks Lakehouse, Delta Lake, and distributed data processing.
• Experience working with cloud platforms such as Microsoft Azure, AWS, or GCP (Azure preferred).
• Experience with CI/CD pipelines, Git, and DevOps practices for data engineering.
• Strong understanding of data architecture, data modeling, and performance optimization.
• Experience working with cross-functional teams to deliver enterprise data solutions.
• Tackles complex data challenges, ensuring data quality and reliable delivery.

Qualifications

• Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field.
• Experience with data integration tools such as Azure Data Factory or similar platforms.
• Familiarity with cloud data warehouses such as Databricks, Snowflake, or Azure Fabric.
• Experience supporting analytics, reporting, or AI/ML workloads is highly desirable.
• Databricks, Azure, or cloud certifications are preferred.
• Strong problem-solving, communication, and technical leadership skills.

Technical Proficiency

• Python, SQL, Scala (preferred)
• Cloud platforms: Azure (preferred), AWS, or GCP
• Azure Data Factory, Kafka, and modern data integration tools
• Data warehousing: Databricks, Snowflake, or Azure Fabric
• Data architecture, ETL/ELT design, and performance optimization

What You’re Looking For

Join a fast-growing organization that thrives on innovation and collaboration. You’ll work alongside talented, motivated colleagues in a global environment, helping clients solve their most critical business challenges. At OZ, your contributions matter – you’ll have the chance to be a key player in our growth and success. If you’re driven, bold, and eager to push boundaries, we invite you to join a company where you can truly make a difference.

About Us

OZ is a 28-year-old global technology consulting, services, and solutions leader specializing in creating business-focused solutions for our clients by leveraging disruptive digital technologies and innovation.

OZ is committed to creating a continuum between work and life by allowing people to work remotely. We offer competitive compensation and a comprehensive benefits package. You’ll enjoy our work style within an incredible culture. We’ll give you the tools you need to succeed so you can grow and develop with us and become part of a team that lives by its core values.