Senior Data Engineer (Databricks & AI Platforms)

KOUD; São Paulo; Hybrid; BRL 280,000 - 400,000; Full time

Job summary

KOUD is seeking a Senior Data Engineer to design, build, and optimize scalable data platforms that support analytics, ML, and AI-driven solutions. You will partner with data scientists, AI engineers, and business stakeholders to deliver trusted data products using cloud technologies.

The role emphasizes enterprise AI enablement, governance, data quality, and observability, with opportunities to mentor engineers and contribute to CI/CD and DataOps across the data platform.

Qualifications

• Bachelor's degree in CS/IS/Engineering or related field.
• Extensive experience with Databricks and Spark.
• Proficiency in Python, SQL, and cloud-based data architectures.
• Experience building data pipelines and Lakehouse architectures.
• Strong knowledge of data governance and enterprise data management.

Responsibilities

• Design, maintain scalable data pipelines and data products with Databricks.
• Build batch, streaming, and real-time data integrations.
• Implement Delta Lake-based architectures and governance.
• Collaborate with data scientists and AI engineers on data preparation for AI/ML use cases.
• Mentor junior engineers and promote engineering best practices.

Skills

Databricks

Spark

Python

SQL

Data governance

Education

Bachelor's degree in Computer Science/Engineering/Information Systems

Tools

Delta Lake

MLflow

Job description

Koud is looking for a Senior Senior Data Engineer (Databricks & AI Platforms) to work with one of our clients.The Senior Data Engineer is responsible for designing, building, and optimizing scalable data platforms and pipelines that support analytics, business intelligence, machine learning (ML), and AI-driven solutions. This role partners with data scientists, AI engineers, architects, and business stakeholders to deliver trusted, high-quality data products using Databricks, cloud technologies, and modern data engineering practices. The position also serves as a technical leader in enabling enterprise AI and Generative AI initiatives through robust, secure, and governed data platforms.

Key Responsibilities:

• Design, develop, and maintain scalable data pipelines and data products using Databricks, Spark, Python, and SQL;


• Build and optimize batch, streaming, and real-time data integration solutions from enterprise and third-party data sources;


• Implement Databricks Lakehouse architectures utilizing Delta Lake and Medallion design patterns;


• Develop and maintain data products that support analytics, predictive modeling, machine learning, and Generative AI applications;


• Collaborate with Data Scientists and AI Engineers to prepare, transform, and govern data for AI and ML use cases;


• Design and implement feature engineering pipelines and support ML lifecycle processes;


• Develop data solutions that support Retrieval Augmented Generation (RAG), vector search, semantic search, and LLM-based applications;


• Optimize Spark jobs, SQL workloads, and data processing frameworks for performance, scalability, and cost efficiency;


• Implement data quality, observability, lineage, governance, and monitoring capabilities;


• Ensure compliance with data privacy, security, and responsible AI standards;


• Contribute to CI/CD, Infrastructure-as-Code, and DataOps practices across the data platform;


• Mentor junior engineers and promote engineering best practices across the organization.

Required Qualifications / Skills:

• Bachelor's degree in Computer Science, Information Systems, Engineering, Mathematics, or a related field;


• Advanced experience in data engineering, ETL development, or large-scale data platform engineering;


• Years of hands-on experience with Databricks and Apache Spark;


• Strong proficiency in Python, SQL, and distributed data processing frameworks;


• Experience building cloud-based data lakes, data warehouses, and Lakehouse architectures;


• Experience supporting AI, machine learning, or advanced analytics initiatives;


• Strong understanding of data modeling, data governance, and enterprise data management practices;


• Experience developing and optimizing large-scale data pipelines in AWS, Azure, or Google Cloud;


• English Fluent.

Preferred:

• Experience with Databricks Delta Lake, Unity Catalog, Delta Live Tables, MLflow, Mosaic AI, and Databricks Workflows;


• Hands-on experience supporting Generative AI, Large Language Models (LLMs), RAG architectures, vector databases, or AI-powered applications;


• Familiarity with AI frameworks such as LangChain, Semantic Kernel, OpenAI APIs, Hugging Face, or similar technologies;


• Experience with feature stores, model training pipelines, and machine learning operationalization (MLOps);


• Experience with Kafka, Event Hub, or other streaming technologies;


• Databricks Certified Data Engineer Professional or equivalent cloud certification.

Additional Information:

• Hybrid position, with 3 days per week working on-site in Pinheiros, São Paulo.