Lead Data Engineer - Databricks Lakehouse & AI Platforms

Brown & Brown; Plano (TX); Vor Ort; USD 150.000 - 210.000; Vollzeit

Benefits dieser Stelle

Health benefits

401k matching

Paid time off

Zusammenfassung

Brown & Brown in Plano, TX is seeking a Lead Data Engineer to advance our Data & AI platform by building scalable data solutions for analytics and AI applications. This role blends hands-on engineering with technical leadership, partnering with product owners, architects, and stakeholders to translate business needs into production-ready data products.

You will design, deploy, and operate end-to-end data pipelines, ensure governance, reliability, and cost efficiency, and mentor engineers while

Qualifikationen

• Bachelor’s degree in a STEM field or equivalent education and experience.
• 8+ years of data engineering experience building and supporting production data solutions.
• Strong hands-on experience with Databricks, Apache Spark/PySpark, Delta Lake, Python, and SQL.
• Experience designing reliable batch and streaming data pipelines and operating Databricks capabilities (Lakeflow, Jobs, Auto Loader, Structured Streaming, Delta Lake, Unity Catalog).
• Experience in data governance with Unity Catalog (catalogs, schemas, tables, views, credentials, lineage, permissions).
• Experience with Terraform and IaC approaches, Git, and CI/CD practices.
• Strong troubleshooting of Spark workloads using Spark UI, logs, query plans; knowledge of partitioning and optimization.
• Understanding of data modeling (conceptual, logical, relational, dimensional, physical).
• Proven ability to deliver ETL/ELT through whole engineering lifecycle and balance quality, speed, and cost.

Aufgaben

• Lead design, development, deployment, and operations of scalable batch and streaming data solutions.
• Build Databricks-native data engineering solutions using Delta Lake and Unity Catalog.
• Develop production-grade ingestion, transformation, and data-processing pipelines with Python, PySpark, Spark, and SQL.
• Define engineering standards and reusable patterns to improve scalability and maintainability.
• Build governed data assets supporting AI, ML, analytics, APIs, and AI agents.
• Mentor engineers and drive engineering excellence and accountability.
• Drive architecture discussions, code reviews, and unblock teams.
• Improve platform adoption and collaborate with product owners, architects, and stakeholders.
• Implement data-quality validation, monitoring, observability, and remediation.

Kenntnisse

Databricks

Apache Spark/PySpark

Python

SQL

Leadership

Agile

Data governance

Ausbildung

Bachelor’s degree in computer science, Engineering, Mathematics, Information Systems or related STEM field

Tools

Databricks

Delta Lake

Lakeflow Declarative Pipelines

Databricks Jobs

Auto Loader

Structured Streaming

Unity Catalog

Terraform

Databricks Asset Bundles

Git

CI/CD

Spark UI

Jobbeschreibung

Brown & Brown in Plano, TX is seeking a Lead Data Engineer to advance our Data & AI platform by building scalable data solutions for analytics and AI applications. This role blends hands-on engineering with technical leadership, partnering with product owners, architects, and stakeholders to translate business needs into production-ready data products.

You will design, deploy, and operate end-to-end data pipelines, ensure governance, reliability, and cost efficiency, and mentor engineers while