Senior Data Engineer - Databricks & AI-Driven Data Platforms

KOUD; United States; Hybrid; USD 140,000 - 190,000; Full time

Job summary

KOUD is seeking a Senior Data Engineer to design, build, and optimize modern data platforms enabling AI, analytics, and BI across enterprise systems. The role emphasizes Databricks, Delta Lake, and cloud-native architectures, with technical leadership to set standards and mentor engineers.

The ideal candidate will drive data quality, governance, and scalable pipelines using Spark, PySpark, and related tooling, while collaborating with data science teams to operationalize ML solutions.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or related field.
• Strong experience in data engineering, data warehousing, or data platform development.
• Hands-on experience with Databricks and Apache Spark.
• Experience designing and implementing enterprise-scale cloud data platforms.
• Experience leading technical initiatives and mentoring engineering teams.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks, Spark, and cloud-native technologies.
• Build and optimize batch and real-time data ingestion frameworks.
• Develop reusable engineering patterns, frameworks, and accelerators for enterprise data solutions.
• Implement data quality, observability, lineage, and governance capabilities across the data ecosystem.
• Support large-scale data modernization and cloud migration initiatives.
• Design Delta Lake architectures and Medallion data models.
• Build optimized ETL/ELT pipelines using Databricks notebooks, workflows, and Delta Live Tables.
• Establish performance tuning, cost optimization, and workload management best practices.
• Leverage Unity Catalog for data governance, security, and metadata management.
• Implement CI/CD pipelines and IaC for Databricks deployments.
• Design cloud-native data architectures across Azure and AWS environments.
• Integrate enterprise data sources including SQL Server, Oracle, APIs, SaaS platforms, and streaming platforms.
• Build resilient and secure data services supporting high availability and disaster recovery requirements.
• Partner with infrastructure and security teams to implement cloud security controls and compliance standards.
• Build and optimize data products supporting AI, ML, and generative AI workloads.
• Develop feature engineering pipelines and curated datasets for model training and inference.
• Support vector databases, semantic search, RAG, and LLM-powered applications.
• Collaborate with Data Scientists and AI Engineers to operationalize ML solutions.
• Implement MLOps and DataOps practices to support scalable AI delivery.

Skills

Databricks

Apache Spark

Delta Lake

Unity Catalog

Python

SQL

Data Modeling

MLOps

Cloud: Azure/AWS

CI/CD

IaC

Data Warehousing

Education

Bachelor's degree in CS/IT/Engineering

Master's degree (preferred)

Tools

Databricks Workflows

Delta Live Tables

Terraform

GitHub

Azure DevOps

Job description

KOUD is seeking a Senior Data Engineer to design, build, and optimize modern data platforms enabling AI, analytics, and BI across enterprise systems. The role emphasizes Databricks, Delta Lake, and cloud-native architectures, with technical leadership to set standards and mentor engineers.

The ideal candidate will drive data quality, governance, and scalable pipelines using Spark, PySpark, and related tooling, while collaborating with data science teams to operationalize ML solutions.