EDL Tech Lead

Accion Labs; Dallas (TX); On-site; USD 140,000 - 200,000; Full time

Job summary

Accion Labs seeks an Enterprise Data Lake Technical Lead to own the design and delivery of a scalable, secure data lake platform across analytics, reporting, and operations. You will lead architectural decisions, mentor teams, and collaborate across security, product, and data engineering to enable self-serve data discovery and governed data workflows.

Responsibilities include building real-time and batch ingestion pipelines, optimizing Delta Lake and Spark, and deploying containerized services

Qualifications

• 5+ years with Apache Kafka (streaming, Connect, Schema Registry).
• 3+ years with Databricks (Delta Lake, Spark optimization, Unity Catalog).
• Deep understanding of data lake architecture patterns.
• 3+ years with DataHub or similar metadata platforms.
• 3+ years with Kubernetes and Terraform.
• 5+ years with enterprise RDBMS (SQL Server, PostgreSQL).
• 5+ years with Azure or AWS cloud services.
• Strong SQL skills and data integration experience.
• CDC patterns and implementation experience.

Responsibilities

• Own the design and implementation of enterprise-scale data lake solutions on Azure/AWS.
• Provide technical mentorship to data engineering and platform teams.
• Drive governance, ingestion, storage, processing, and security layers.
• Lead architectural alignment with Architecture and Product Management.
• Develop data catalog capabilities and metadata-driven data operations.
• Ensure platform reliability, observability, and cost optimization.

Skills

C#

Go

Python

Bash

Leadership

Architecture

SQL

Tools

Kafka

Databricks

DataHub

Terraform

Kubernetes

Azure

AWS

Job description

The Enterprise Data Lake (EDL) Technical Lead is responsible for owning the design, implementation, and engineering leadership of the enterprise data lake platform. This role ensures the platform is scalable, reliable, secure, and consumable across analytics, reporting, and operational workloads.

The Tech Lead drives technical decisions and collaborates closely with Architecture, Product Management, Data Engineering, Security, and downstream consumers to deliver robust data platform capabilities.

Key Responsibilities

Technical Leadership

• Own the design and implementation of enterprise-scale data lake solutions on Azure/AWS Cloud
• Define technical standards and best practices for data platform components
• Drive technical decision-making across data ingestion, storage, processing, and governance layers
• Provide technical mentorship and guidance to data engineering and platform teams
• Collaborate with Architecture team on overall architectural alignment and strategy
• Partner with Product Management, Data Engineering, Security, SRE, and downstream consumers

Data Governance & Catalog Management (Critical)

• Own the design and implementation of enterprise data catalog and metadata management layer
• Build and operate data catalog using DataHub or equivalent metadata platform
• Design and implement automated metadata ingestion, lineage tracking, and data discovery capabilities
• Design and implement data governance policies, data quality rules, and compliance frameworks or Chrono data Orchestrator
• Enable self-service data discovery and access management for downstream consumers
• Establish data ownership, stewardship models, and metadata standards across the enterprise
• Build APIs and services for metadata-driven data operations and workflows
• Design and implement real-time and batch data ingestion pipelines using Apache Kafka
• Optimize data processing workflows on Databricks platform (Delta Lake, Spark optimization, Unity Catalog)
• Design integration patterns between RDBMS sources and data lake (CDC, batch ETL, replication)
• Optimize database performance, query tuning, and indexing strategies across relational and distributed systems
• Implement infrastructure as code using Terraform for automated provisioning and management
• Design and deploy containerized data services on Kubernetes clusters
• Develop data platform services and tooling using C# and Go
• Build database migration tools, data validation frameworks, and connectivity services

Operations & Reliability

• Ensure platform scalability, reliability, and security across all data lake components
• Implement monitoring, logging, and observability solutions for data infrastructure
• Optimize Linux-based systems for data processing workloads
• Ensure platform security, compliance, and data governance standards (GDPR, SOC2, etc.)
• Drive cost optimization and performance tuning initiatives
• Implement database backup/recovery strategies and disaster recovery planning

Required Qualifications

Technical Expertise

Data Streaming & Processing

• 5+ years with Apache Kafka (streaming architecture, Kafka Connect, Schema Registry, stream processing)
• 3+ years with Databricks (Delta Lake, Apache Spark optimization, Unity Catalog, cluster management)
• Deep understanding of data lake architecture patterns (Bronze/Silver/Gold, medallion architecture)

Data Governance & Cataloging

• 3+ years with DataHub or similar metadata management platforms (Alation, Collibra, Apache Atlas)
• Deep experience building and operating enterprise data catalog systems
• Expertise in automated metadata extraction, lineage tracking, and impact analysis
• Experience with data quality frameworks and metadata-driven data operations
• Knowledge of data governance policies, data classification, and compliance automation
• Understanding of data discovery, access control, and self-service analytics enablement

Relational Database Systems

• 5+ years with enterprise RDBMS platforms including:
• SQL Server (T-SQL, SSIS, SSRS, replication, Always On Availability Groups)
• PostgreSQL (advanced query optimization, partitioning, extensions, streaming replication)
• Strong SQL skills (complex queries, stored procedures, window functions, query optimization)
• Change Data Capture (CDC) patterns and implementation (Debezium, Azure Data Factory, AWS DMS, custom solutions)
• 5+ years with Azure or AWS Cloud including:
• AWS: S3, MSK/Kinesis, EKS, RDS/Aurora, Secrets Manager, IAM, CloudWatch
• Cloud-native data services, networking, security, and IAM
• 3+ years with Kubernetes (deployment strategies, scaling, monitoring, service mesh, Helm)
• 3+ years with Terraform (modules, state management, multi-environment deployments, multi-cloud)

Programming & Development

• Strong proficiency in any programming languages like C#, Go
• Experience with Python for data engineering tasks (preferred)
• Familiarity with Shell scripting (Bash)

Leadership & Experience

• 3+ years in technical leadership capacity (Tech Lead, Principal Engineer)
• Proven track record of delivering large-scale data infrastructure projects
• Experience leading teams of 5-10+ engineers
• Strong architectural design and system thinking capabilities
• Experience migrating legacy RDBMS workloads to modern data lake architectures
• Demonstrated ability to balance technical excellence with business needs

Preferred Qualifications

Additional Technologies

• Experience with cloud ETL services (Azure Data Factory, AWS Glue, Azure Stream Analytics, AWS Lambda)
• Knowledge of managed database features (Azure SQL elastic pools/hyperscale, AWS RDS/Aurora serverless)
• Knowledge of additional streaming technologies (Apache Flink)
• Experience with database sharding and horizontal partitioning strategies at scale
• Familiarity with NoSQL databases (Cosmos DB, DynamoDB, MongoDB, Cassandra, Redis)