Lead Data Engineer — Databricks Lakehouse & AI Platform

Brown & Brown, Inc.; Plano (TX); In loco; USD 150.000 - 210.000; Tempo pieno

Vantaggi offerti da questo lavoro

Health benefits

401(k) plan

Paid time off

Professional development

Descrizione del lavoro

Brown & Brown, Inc. is seeking a Lead Data Engineer in Plano, TX to advance our Data & AI platform by designing and operating scalable data solutions powering AI, analytics, and enterprise insights.

You will lead design, implement batch and streaming pipelines with Databricks, Delta Lake, PySpark, and SQL, while mentoring engineers and setting governance standards across the data lifecycle.

Competenze

• Bachelor’s degree in computer science, Engineering, Mathematics, Information Systems, or a related STEM field, or equivalent education and professional experience.
• 8+ years of experience in data engineering or related software/data engineering disciplines, including significant hands-on experience building and supporting production data solutions.
• Strong hands-on experience with Databricks, Apache Spark/PySpark, Delta Lake, Python, and SQL.
• Experience designing and operating production-grade batch and streaming data pipelines.
• Experience with Databricks capabilities such as Lakeflow Declarative Pipelines, Databricks Jobs, Auto Loader, Structured Streaming, Delta Lake, and Unity Catalog.
• Experience designing reliable workflow orchestration, including scheduling, dependencies, parameters, retries, notifications, and operational monitoring.
• Experience implementing data governance and access controls with Unity Catalog, including catalogs, schemas, tables, views, external locations, storage credentials, lineage, and fine-grained permissions.
• Experience with Terraform, Databricks Asset Bundles, or comparable Infrastructure as Code approaches, along with Git and CI/CD practices.
• Strong experience troubleshooting and optimizing Spark workloads using tools such as Spark UI, job logs, query plans, partitioning, join optimization, and Adaptive Query Execution.
• Strong understanding of data modeling and ETL/ELT lifecycles.
• Demonstrated technical leadership, mentoring, and ability to balance quality, speed, scalability, and cost.

Mansioni

• Lead the design, development, deployment, and operational support of scalable batch and streaming data solutions.
• Design and build Databricks-native data engineering solutions using Delta Lake and Unity Catalog.
• Develop production-grade ingestion, transformation, and data processing pipelines using Python, PySpark, Spark, and SQL.
• Define engineering standards and reusable patterns to improve scalability and developer productivity.
• Build governed data solutions to improve data quality, accessibility, and trust across enterprise data assets.
• Mentor engineers and drive architectural alignment, code reviews, and unblock teams.
• Implement data governance capabilities and integrate with enterprise metadata, lineage, and access controls.
• Collaborate with product owners, architects, and stakeholders to expand platform adoption.

Conoscenze

Databricks Lakehouse

Python

SQL

Spark

Data modeling

Leadership

Agile

Mentoring

Problem solving

Formazione

Bachelor’s degree in CS/Engineering/Math or related

Strumenti

Databricks

Delta Lake

Unity Catalog

Terraform

Git

CI/CD

Descrizione del lavoro

Brown & Brown, Inc. is seeking a Lead Data Engineer in Plano, TX to advance our Data & AI platform by designing and operating scalable data solutions powering AI, analytics, and enterprise insights.

You will lead design, implement batch and streaming pipelines with Databricks, Delta Lake, PySpark, and SQL, while mentoring engineers and setting governance standards across the data lifecycle.