Senior Data Engineer (Databricks & AI Platforms)

KOUD; United States; Hybrid; USD 64,000 - 104,000; Full time

Benefits offered by this job

Hybrid on-site in Pinheiros

Job summary

KOUD is seeking a Senior Data Engineer to design, build, and optimize scalable data platforms and pipelines that support analytics, ML, and AI-driven solutions using Databricks, Spark, Python, and SQL.

You will partner with data scientists, AI engineers, architects, and business stakeholders to deliver trusted data products with robust governance, security, and modern DataOps practices, including CI/CD.

The role is hybrid, with on-site collaboration in Pinheiros, São Paulo, Brazil.

Qualifications

• Bachelor's degree in Computer Science, Information Systems, Engineering, Mathematics, or a related field.
• Advanced experience in data engineering, ETL development, or large-scale data platform engineering.
• Years of hands-on experience with Databricks and Apache Spark.
• Strong proficiency in Python, SQL, and distributed data processing frameworks.
• Experience building cloud-based data lakes, data warehouses, and Lakehouse architectures.
• Experience supporting AI, machine learning, or advanced analytics initiatives.
• Strong understanding of data modeling, data governance, and enterprise data management practices.
• Experience developing and optimizing large-scale data pipelines in AWS, Azure, or Google Cloud.
• English Fluent.

Responsibilities

• Design, develop, and maintain scalable data pipelines and data products using Databricks, Spark, Python, and SQL.
• Build and optimize batch, streaming, and real-time data integration solutions.
• Implement Databricks Lakehouse architectures utilizing Delta Lake and patterns.
• Develop data products that support analytics, ML, and Generative AI applications.
• Collaborate with Data Scientists and AI Engineers to prepare, transform, and govern data for AI/ML use cases.
• Design feature engineering pipelines and support ML lifecycle processes.
• Develop data solutions for RAG, vector search, semantic search, and LLM-based apps.
• Optimize Spark jobs, SQL workloads, and data processing frameworks for performance and cost.
• Implement data quality, observability, lineage, governance, and monitoring.
• Ensure compliance with data privacy, security, and responsible AI standards.
• Contribute to CI/CD, Infrastructure-as-Code, and DataOps across the platform.
• Mentor junior engineers and promote engineering best practices.

Skills

Databricks

Apache Spark

Python

SQL

Data modeling

Data governance

Cloud platforms

MLOps

CI/CD

Education

Bachelor's degree in CS/IS/Engineering/Math

Tools

Databricks Delta Lake

Unity Catalog

Delta Live Tables

MLflow

Mosaic AI

Databricks Workflows

Kafka

Job description

Koud is looking for a Senior Senior Data Engineer (Databricks & AI Platforms) to work with one of our clients.The Senior Data Engineer is responsible for designing, building, and optimizing scalable data platforms and pipelines that support analytics, business intelligence, machine learning (ML), and AI-driven solutions. This role partners with data scientists, AI engineers, architects, and business stakeholders to deliver trusted, high-quality data products using Databricks, cloud technologies, and modern data engineering practices. The position also serves as a technical leader in enabling enterprise AI and Generative AI initiatives through robust, secure, and governed data platforms.

Key Responsibilities:

• Design, develop, and maintain scalable data pipelines and data products using Databricks, Spark, Python, and SQL;


• Build and optimize batch, streaming, and real-time data integration solutions from enterprise and third-party data sources;


• Implement Databricks Lakehouse architectures utilizing Delta Lake and Medallion design patterns;


• Develop and maintain data products that support analytics, predictive modeling, machine learning, and Generative AI applications;


• Collaborate with Data Scientists and AI Engineers to prepare, transform, and govern data for AI and ML use cases;


• Design and implement feature engineering pipelines and support ML lifecycle processes;


• Develop data solutions that support Retrieval Augmented Generation (RAG), vector search, semantic search, and LLM-based applications;


• Optimize Spark jobs, SQL workloads, and data processing frameworks for performance, scalability, and cost efficiency;


• Implement data quality, observability, lineage, governance, and monitoring capabilities;


• Ensure compliance with data privacy, security, and responsible AI standards;


• Contribute to CI/CD, Infrastructure-as-Code, and DataOps practices across the data platform;


• Mentor junior engineers and promote engineering best practices across the organization.

Required Qualifications / Skills:

• Bachelor's degree in Computer Science, Information Systems, Engineering, Mathematics, or a related field;


• Advanced experience in data engineering, ETL development, or large-scale data platform engineering;


• Years of hands-on experience with Databricks and Apache Spark;


• Strong proficiency in Python, SQL, and distributed data processing frameworks;


• Experience building cloud-based data lakes, data warehouses, and Lakehouse architectures;


• Experience supporting AI, machine learning, or advanced analytics initiatives;


• Strong understanding of data modeling, data governance, and enterprise data management practices;


• Experience developing and optimizing large-scale data pipelines in AWS, Azure, or Google Cloud;


• English Fluent.

Preferred:

• Experience with Databricks Delta Lake, Unity Catalog, Delta Live Tables, MLflow, Mosaic AI, and Databricks Workflows;


• Hands-on experience supporting Generative AI, Large Language Models (LLMs), RAG architectures, vector databases, or AI-powered applications;


• Familiarity with AI frameworks such as LangChain, Semantic Kernel, OpenAI APIs, Hugging Face, or similar technologies;


• Experience with feature stores, model training pipelines, and machine learning operationalization (MLOps);


• Experience with Kafka, Event Hub, or other streaming technologies;


• Databricks Certified Data Engineer Professional or equivalent cloud certification.

Additional Information:

• Hybrid position, with 3 days per week working on-site in Pinheiros, São Paulo.