Lead Data Engineer - Google Cloud & Azure Databricks (Kansas City)

Burns & McDonnell

Kansas City (MO)

On-site

USD 150,000 - 210,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Burns & McDonnell is seeking a senior data engineer who marries an architecture mindset with hands-on production experience. You will own end-to-end pipelines across Google Cloud and Azure Databricks, guiding the data product roadmap and mentoring the team.

You will collaborate with product and business stakeholders to translate strategic goals into robust, secure data solutions, while establishing standards, CI/CD, testing, and governance across the platform.

Qualifications

  • Bachelor degree in CS, MIS, or related field plus 8 years experience.
  • Strong programming skills and coding principles.
  • Expert in web development framework.
  • Excellent written and verbal communication.
  • Analytical thinker with problem-solving skills.
  • Ability to obtain applicable certifications.

Responsibilities

  • Architect, build, and ship scalable pipelines to production.
  • Design data pipelines on Azure Databricks with Spark.
  • Own end-to-end data product lifecycle and SLAs.
  • Provide technical leadership and code reviews.
  • Lead multi-cloud data platform architecture and cost management.
  • Collaborate with product managers and leadership on data requirements.
  • Define data engineering standards including CI/CD and testing.
  • Monitor pipelines and troubleshoot production systems.
  • Evaluate emerging technologies for governance and AI integrations.

Skills

Python
Java
SQL
Analytical thinking
Communication
Mentoring
Team leadership

Education

Bachelor's degree in CS/IS/Engineering

Tools

BigQuery
Dataflow
Databricks
Spark
Airflow
Unity Catalog
Delta Lake

Job description

Overview:

We\'re looking for a senior engineer who combines an architecture mindset with deep hands‑on experience - someone who can design scalable systems and personally build and ship them into production. As our lead data engineer, you will own pipelines end‑to‑end across Google Cloud and Azure Databricks, from whiteboard to running code, while setting the technical direction for our enterprise data products, mentoring the team, and serving as the bridge between business stakeholders and engineering to turn strategic goals into robust, secure data solutions.


Responsibilities:


  • Architect, build, and ship scalable batch and streaming data pipelines into production using BigQuery Dataform, Google Cloud Dataflow, Confluent Kafka, and Pub/Sub.

  • Design and build data pipelines on Azure Databricks using Apache Spark, Delta Lake, and Unity Catalog, keeping engineering patterns consistent with our Google Cloud platform.

  • Manage the end‑to‑end data product lifecycle, serving as the technical owner for critical data assets and establishing clear service level objectives.

  • Provide technical leadership and mentorship to junior and senior engineers, conducting code reviews and fostering a culture of technical excellence.

  • Lead the engineering team across Google Cloud and Azure Databricks environments, setting multi‑cloud architecture standards for data movement, interoperability, security, and cost management.

  • Partner with product managers, business analysts, and leadership to define data product requirements, align roadmaps, and manage engineering delivery schedules.

  • Define, document, and enforce data engineering standards, including CI/CD practices, robust testing frameworks, and comprehensive technical documentation.

  • Own the monitoring, cost optimization, and troubleshooting of production pipelines to maintain reliable business operations.

  • Evaluate emerging technologies, particularly generative AI integrations and advanced governance tools, to continuously improve data platform capabilities.


Required:


  • Bachelor Degree in Computer Science, Management Information Systems, Engineering or related field and 8 years Required

  • Expert knowledge in programming languages and coding principles and procedures.

  • Expert in web development framework.

  • Strong attention to detail, facilitation, team building, collaboration, and problem‑solving skills.

  • Excellent verbal and written communication skills.

  • Ability to work methodically and analytically in a quantitative problem‑solving environment.

  • Demonstrated critical thinking skills.

  • Expert in Microsoft Office Suite (Word, Excel, and PPT).

  • Strong attention to detail, facilitation, team building and collaboration.

  • Ability to obtain applicable certifications.

  • Strongly Preferred.


Strongly Preferred:


  • Ten or more years of experience in data engineering, data warehousing, or data architecture, with at least five years in a technical leadership, mentoring, or consulting role.

  • Deep technical expertise with Google Cloud, BigQuery, Dataflow, and related cloud data services.

  • Hands‑on production experience with Azure Databricks, including Apache Spark (PySpark), Delta Lake, and Unity Catalog.

  • Proven experience architecting and leading delivery on multi‑cloud data platforms, ideally spanning Google Cloud and Azure.

  • Production experience designing and deploying real‑time streaming architectures using messaging platforms like Confluent Kafka or Google Pub/Sub.

  • Strong expertise in workflow orchestration using scheduling tools like Airflow.

  • Advanced SQL skills and extensive experience with transformation frameworks like Dataform or dbt.

  • Production experience with at least one transactional database, such as Oracle, SQL Server, or PostgreSQL.

  • Strong hands‑on coding skills in Python or Java, with a track record of personally shipping production‑grade data pipelines.

  • Demonstrated experience managing data product delivery, defining schemas, and establishing data quality baselines.

  • Strong communication skills with a proven track record of collaborating with cross‑functional stakeholders and translating business needs into technical specifications.

  • Strong problem‑solving abilities and experience leading engineering teams in fast‑paced environments.


Additional Preferred:


  • Experience coding with agentic tooling as part of generative AI workflows.

  • Direct experience implementing data mesh principles, managing data products, and using tools like Knowledge Catalog (Dataplex) or Entropy Data for contract management.

  • Advanced knowledge of data security, compliance, and governance tooling such as Immuta and native BigQuery security controls.

  • Experience operating across hybrid or multi‑data platform environments, including Palantir.

  • Google Cloud Professional Data Engineer or Professional Cloud Architect certification.

  • Databricks Certified Data Engineer Professional or Microsoft Azure data certifications, plus certifications in supporting technologies such as Power BI.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Lead Data Engineer
Lead Data Engineer

Good co India • United States

Remote
USD 120,000 - 180,000
Lead Databricks Data Engineer
Lead Databricks Data Engineer

Fa Ewjt Saasfaprod1 • New York (NY)

On-site
USD 140,000 - 170,000
Big Data Architect (GCP)
Big Data Architect (GCP)

SoftServe Inc. • Town of Poland (NY)

On-site
USD 120,000 - 190,000
Lead Databricks Data Engineer
Lead Databricks Data Engineer

EXL Talent Acquisition Team • New York (NY)

On-site
USD 180,000 - 240,000
Senior Databricks Engineer
Senior Databricks Engineer

Remote • San Francisco (CA)

On-site
USD 150,000 - 190,000
Data Engineer - Seattle, WA
Data Engineer - Seattle, WA

AHU Technologies Inc. • United States

Remote
USD 90,000 - 130,000
Data Engineering Team Lead
Data Engineering Team Lead

UNAVAILABLE • McLean (VA)

On-site
USD 150,000 - 190,000
Lead Data Engineer (GCP) REMOTE - #35373
Lead Data Engineer (GCP) REMOTE - #35373

Manila Recruitment • United States

Remote
USD 140,000 - 190,000
Databricks Data Engineer
Databricks Data Engineer

EXL Talent Acquisition Team • New York (NY)

On-site
USD 170,000 - 230,000
Data Engineer-Data Platforms-Google - 3
Data Engineer-Data Platforms-Google - 3

International Business Machines Corporation • San Francisco (CA)

Remote
USD 120,000 - 180,000