Databricks Solutions Architect

Insight Global; Salvador; Remote; BRL 180,000 - 300,000; Full time

Job summary

Insight Global is seeking a senior data engineer to design, build, and support scalable data and analytics solutions on Databricks. You will develop pipelines, Delta Lake tables, notebooks, workflows, and Databricks SQL assets to empower business users.

The role emphasizes ML model development, AI tooling integration, and collaboration with U.S. teams, requiring strong Python/SQL skills, Databricks expertise, and a collaborative mindset.

Qualifications

• 5+ years in data engineering, analytics, software development, or data architecture.
• 3+ years hands-on Databricks experience.
• Strong Python, SQL, PySpark, and Spark performance‑tuning skills.
• Experience with Delta Lake, Databricks Workflows, data modeling, and cloud‑based data platforms.
• Hands‑on experience building and deploying ML models, preferably using MLflow and MLOps practices.
• Experience with generative AI, LLM APIs, prompt engineering, RAG, embeddings, and vector search.
• Familiarity with AI tools such as Claude, OpenAI, Azure OpenAI, or similar platforms.
• Strong communication, problem‑solving, and stakeholder‑management skills.
• Experience with Azure Databricks, AWS, or GCP.
• Experience integrating data platforms with enterprise applications and BI tools.
• Knowledge of Unity Catalog, CI/CD, Git, Terraform, and data governance.
• Databricks, cloud, or AI/ML certifications.

Responsibilities

• Design, build, and support scalable data and analytics solutions on Databricks.
• Develop data pipelines, Delta Lake tables, notebooks, workflows, and Databricks SQL assets.
• Create data models, forecasting capabilities, scenario analysis, and reporting solutions.
• Integrate Databricks with enterprise applications, databases, APIs, cloud storage, and BI tools.
• Develop and operationalize machine learning models for predictive analytics.
• Evaluate and implement generative AI solutions using Claude, OpenAI, Azure OpenAI, or similar tools.
• Build AI-enabled applications using LLM APIs, retrieval-augmented generation (RAG), embeddings, and vector search.
• Partner with business and technical stakeholders to translate requirements into production-ready solutions.
• Establish standards for security, governance, testing, documentation, deployment, and performance optimization.
• Provide technical guidance and support for Databricks, AI, and ML initiatives.
• Availability to work during U.S. business hours and collaborate effectively with U.S.-based business and technical teams.

Skills

Python

SQL

PySpark

Spark performance tuning

Communication

Stakeholder management

Problem solving

Tools

Databricks

Delta Lake

Azure Databricks

MLflow

MLOps

LLM APIs

Vector search

CI/CD

Terraform

Git

Job description

Job Description

Key Responsibilities- Design, build, and support scalable data and analytics solutions on Databricks.- Develop data pipelines, Delta Lake tables, notebooks, workflows, and Databricks SQL assets.- Create data models, forecasting capabilities, scenario analysis, and reporting solutions.- Integrate Databricks with enterprise applications, databases, APIs, cloud storage, and BI tools.- Develop and operationalize machine learning models for predictive analytics and business use cases.- Evaluate and implement generative AI solutions using Claude, OpenAI, Azure OpenAI, or similar tools.- Build AI-enabled applications using LLM APIs, retrieval-augmented generation (RAG), embeddings, and vector search.- Partner with business and technical stakeholders to translate requirements into production-ready solutions.- Establish standards for security, governance, testing, documentation, deployment, and performance optimization.- Provide technical guidance and support for Databricks, AI, and ML initiatives.- Availability to work during U.S. business hours and collaborate effectively with U.S.-based business and technical teams.

Skills and Requirements

• 5+ years of experience in data engineering, analytics, software development, or data architecture.
• 3+ years of hands‑on Databricks experience.
• Strong Python, SQL, PySpark, and Spark performance‑tuning skills.
• Experience with Delta Lake, Databricks Workflows, data modeling, and cloud‑based data platforms.
• Hands‑on experience building and deploying machine learning models, preferably using MLflow and MLOps practices.
• Experience with generative AI, LLM APIs, prompt engineering, RAG, embeddings, and vector search.
• Familiarity with AI tools such as Claude, OpenAI, Azure OpenAI, or similar platforms.
• Strong communication, problem‑solving, and stakeholder‑management skills.
• Experience with Azure Databricks, AWS, or GCP.
• Experience integrating data platforms with enterprise applications and BI tools.
• Knowledge of Unity Catalog, CI/CD, Git, Terraform, and data governance.
• Databricks, cloud, or AI/ML certifications.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to [e-mail].