Azure Databricks Data Platform Lead

FlexSteel USA, LLC.

Houston (TX)

On-site

USD 140,000 - 190,000

Full time

9 days ago

Get more replies from employers

Send a job-specific resume in minutes.

Job summary

FlexSteel USA, LLC. is seeking a Data Platform Lead in Houston to own the Azure Databricks Lakehouse, orchestrate Bronze/Silver/Gold data layers, and establish Unity Catalog governance for enterprise data assets.

You will lead ingestion from ERP, PostgreSQL, SaaS, APIs, and files, partner with cybersecurity and DevOps, set standards, ensure data quality, and drive production readiness across analytics, reporting, automation, and AI use cases in a collaborative, on-site team.

Qualifications

7+ years of experience in data engineering, data architecture, cloud data platforms, or enterprise data integration.
Hands-on experience designing and supporting production data pipelines in cloud environments.
Experience with modern Lakehouse architecture, preferably Azure Databricks and Delta Lake.
Experience working with enterprise source systems such as ERP, CRM, SaaS applications, PostgreSQL, SQL Server, Oracle, or similar relational databases.
Experience reviewing vendors or contractor technical deliverables and enforcing engineering standards.

Responsibilities

Own the technical architecture for the Azure Databricks Lakehouse, including workspace structure, catalogs, schemas, compute patterns, storage strategy, and environment separation.
Define and maintain bronze, silver, and gold layer standards, including naming conventions, table ownership, audit columns, refresh patterns, and production readiness criteria.
Implement and govern Unity Catalog standards for access control, lineage, data classification, catalog/schema organization, and least‑privilege access.
Partner with cybersecurity, infrastructure, and DevOps teams to align Databricks with enterprise identity, networking, secrets management, monitoring, and compliance expectations.
Establish cost controls, cluster policies, job standards, and usage monitoring to ensure the platform is reliable, scalable, and cost‑effective.
Design, build, and oversee production‑grade data pipelines using Databricks, Spark, Python/PySpark, SQL, Delta Lake, and approved orchestration patterns.
Lead ingestion from ERP systems, product PostgreSQL databases, SaaS platforms, APIs, files, and other enterprise data sources into the Lakehouse.
Define engineering patterns for full loads, incremental loads, CDC where applicable, reprocessing, error handling, logging, reconciliation, and pipeline recovery.
Ensure every production pipeline includes source‑to‑target mapping, ownership, data quality rules, monitoring, alerting, and operational handover documentation.
Review vendor and contractor deliverables for technical quality, maintainability, security, performance, and production readiness.