Director Of Engineering – Data Platform

Zilker Partners; Austin (TX); On-site; USD 180,000 - 240,000; Full time

Job summary

Director of Engineering, Data Platform at Zilker Partners, based in Austin, leads the architecture and delivery of petabyte-scale data ingestion, processing, and analytics for AI-driven cybersecurity products. You will build distributed teams, drive platform strategy, and enable self-service tooling across data science and product squads.

You will shape engineering practices, mentor leaders, and partner with Product, AI, and Security teams to ensure reliable, scalable platforms that support ML

Qualifications

• 10+ years in software engineering with leadership experience
• Proven track record building large-scale data platforms in production
• Deep expertise in distributed data processing and storage at scale
• Hands-on with Databricks and Spark-based pipelines
• Experience running ML/AI workloads in production environments
• Strong reliability and SRE mindset across platforms
• Background in security analytics or cybersecurity domain a plus
• Ability to optimize cloud costs and performance at scale across AWS/Azure/GCP
• Experience with real-time data processing and streaming analytics a plus

Responsibilities

• Build, grow, and mentor engineering teams across geographies
• Develop engineering managers and senior leaders through coaching
• Foster inclusive, trust-based culture with ownership and accountability
• Create execution plans with clear priorities, rhythms, and meaningful work
• Align data platform priorities with product, AI, infrastructure, and customer teams
• Communicate platform strategy, progress, risks, and trade-offs to executives
• Partner with customer-facing teams to understand evolving data needs
• Shape company direction on data, AI, and platform capabilities
• Lead large-scale platform initiatives balancing innovation, reliability and speed
• Define engineering best practices for data processing and data quality
• Own production operations, incident management, and continuous improvement
• Ensure predictable delivery with customer impact in mind
• Engage architecture discussions for scalable, AI-ready infrastructure
• Champion platform-first mindset with self-service APIs and tooling
• Drive adoption of features like feature stores and vector storage
• Promote AI to boost productivity and incident response

Skills

Leadership

Data platform

Distributed systems

Databricks

Spark

ML/AI workloads

SRE principles

LLM-based platforms

Cloud cost optimization

Cybersecurity familiarity

Tools

Databricks

Spark

Kubernetes

Job description

Job Overview:

Our Client is seeking a Director of Engineering to lead the Data Platform organization at the core of its AI-driven cybersecurity products. This role owns the ingestion, processing, storage, and analysis of petabytes of security telemetry—spanning network packets, logs, identity signals, and security events—and advances the platform to power modern AI workloads, autonomous security workflows, and next-generation analytics.

This is an organizational leadership role that demands deep technical credibility. You will build and develop high-performing, distributed teams while engaging directly in architectural decisions, challenging assumptions, and de-risking designs. Partnering closely with Product, Security Research, AI, Infrastructure, and Customer teams, you will establish a platform-first strategy that accelerates Data Science, AI/ML, and product delivery through self-service capabilities, strong observability, and best-in-class reliability.

Job Responsibilities:

• Build, grow, and mentor high-performing engineering teams across multiple geographies


• Develop engineering managers and senior technical leaders through coaching, sponsorship, and career development


• Foster an inclusive culture grounded in trust, ownership, and accountability


• Create conditions for strong execution with clear priorities, healthy team rhythms, and focus on meaningful work


• Align priorities and investments with Product, Security Research, AI, Infrastructure, and Customer teams


• Communicate platform strategy, progress, risks, and trade-offs clearly to executive leadership


• Partner with customer-facing teams to anticipate evolving data requirements and scaling challenges


• Influence company-wide direction on data, AI, and platform capabilities


• Lead delivery of large-scale platform initiatives, balancing innovation, reliability, and speed


• Define and enforce engineering best practices across data processing, schema evolution, data quality, and service-level objectives


• Own production operations, incident management, and continuous improvement


• Ensure predictable delivery while maintaining focus on customer impact and long-term platform health


• Engage deeply on architecture for scalable, reliable, AI-ready infrastructure and pressure-test designs with platform architects


• Champion a platform-first mindset with self-service APIs, robust developer tooling, observability, and clear documentation


• Drive adoption of modern capabilities including feature stores, vector storage, and model serving infrastructure


• Champion the use of AI to improve engineering productivity, software delivery, and incident response

Job Requirements:

• 10+ years of software engineering experience, including 5+ years leading engineering organizations


• Proven track record building and operating large-scale data platforms in production


• Deep expertise in large-scale data infrastructure: distributed systems, ingestion pipelines, stream and batch processing, and storage at scale (object stores, columnar databases, time-series systems)


• Hands-on experience with Databricks, Spark-based processing, and the Databricks platform ecosystem


• Experience running ML or AI workloads in production, including platforms, feature stores, and inference infrastructure


• Operational mindset with strong grounding in SRE principles, incident response, and reliability practices


• Experience building or operating platforms that support LLM-based applications, including retrieval, embeddings, and model serving


• Demonstrated ability to champion and apply AI to improve engineering productivity, software delivery, and incident response


• Preferred: Background in cybersecurity, NDR, SIEM, XDR, observability, or related domains


• Preferred: Proven ability to optimize cloud costs and performance at scale on AWS, Azure, or GCP


• Preferred: Experience with real-time detection systems or streaming analytics.