Senior Data Engineer - Databricks & AI Platforms (Hybrid)

KOUD; United States; Hybrid; USD 64,000 - 104,000; Full time

Benefits offered by this job

Hybrid on-site in Pinheiros

Job summary

KOUD is seeking a Senior Data Engineer to design, build, and optimize scalable data platforms and pipelines that support analytics, ML, and AI-driven solutions using Databricks, Spark, Python, and SQL.

You will partner with data scientists, AI engineers, architects, and business stakeholders to deliver trusted data products with robust governance, security, and modern DataOps practices, including CI/CD.

The role is hybrid, with on-site collaboration in Pinheiros, São Paulo, Brazil.

Qualifications

• Bachelor's degree in Computer Science, Information Systems, Engineering, Mathematics, or a related field.
• Advanced experience in data engineering, ETL development, or large-scale data platform engineering.
• Years of hands-on experience with Databricks and Apache Spark.
• Strong proficiency in Python, SQL, and distributed data processing frameworks.
• Experience building cloud-based data lakes, data warehouses, and Lakehouse architectures.
• Experience supporting AI, machine learning, or advanced analytics initiatives.
• Strong understanding of data modeling, data governance, and enterprise data management practices.
• Experience developing and optimizing large-scale data pipelines in AWS, Azure, or Google Cloud.
• English Fluent.

Responsibilities

• Design, develop, and maintain scalable data pipelines and data products using Databricks, Spark, Python, and SQL.
• Build and optimize batch, streaming, and real-time data integration solutions.
• Implement Databricks Lakehouse architectures utilizing Delta Lake and patterns.
• Develop data products that support analytics, ML, and Generative AI applications.
• Collaborate with Data Scientists and AI Engineers to prepare, transform, and govern data for AI/ML use cases.
• Design feature engineering pipelines and support ML lifecycle processes.
• Develop data solutions for RAG, vector search, semantic search, and LLM-based apps.
• Optimize Spark jobs, SQL workloads, and data processing frameworks for performance and cost.
• Implement data quality, observability, lineage, governance, and monitoring.
• Ensure compliance with data privacy, security, and responsible AI standards.
• Contribute to CI/CD, Infrastructure-as-Code, and DataOps across the platform.
• Mentor junior engineers and promote engineering best practices.

Skills

Databricks

Apache Spark

Python

SQL

Data modeling

Data governance

Cloud platforms

MLOps

CI/CD

Education

Bachelor's degree in CS/IS/Engineering/Math

Tools

Databricks Delta Lake

Unity Catalog

Delta Live Tables

MLflow

Mosaic AI

Databricks Workflows

Kafka

Job description

KOUD is seeking a Senior Data Engineer to design, build, and optimize scalable data platforms and pipelines that support analytics, ML, and AI-driven solutions using Databricks, Spark, Python, and SQL.

You will partner with data scientists, AI engineers, architects, and business stakeholders to deliver trusted data products with robust governance, security, and modern DataOps practices, including CI/CD.

The role is hybrid, with on-site collaboration in Pinheiros, São Paulo, Brazil.