Lead Data Engineer/Senior Data Engineer

Indsafri; South Africa; Vor Ort; ZAR 800.000 - 1.200.000; Vollzeit

Zusammenfassung

Indsafri India Private Limited is looking for a highly experienced Senior Data Engineer to design and operate robust Databricks and Lakehouse data platforms. This role emphasizes analytics and AI, driving innovative Generative AI applications. Key responsibilities include developing data pipelines, ensuring compliance with governance standards, and collaborating with cross-functional teams. Ideal candidates possess a strong background in data engineering with over 10 years of experience, particularly with Databricks and enterprise data architectures, contributing to cutting-edge projects.

Qualifikationen

• 10-15 years of industry experience.
• Proven experience as a Senior/Lead Data Engineer (5+ years).
• Hands-on experience with Databricks environments (2+ years).
• Strong understanding of enterprise data lake and lake house architecture (5+ years).
• Proficiency in Python, SQL, and Apache Spark (5+ years).
• Experience building and operating production-grade data platforms (3+ years).
• Experience working in enterprise or regulated environments (5+ years).

Aufgaben

• Design, build, and operate data solutions using Databricks components.
• Develop production-grade data pipelines using Python, SQL, and Apache Spark.
• Implement automated testing and ensure data solutions are observable and performant.
• Enable data consumption for Generative AI use cases.
• Collaborate closely with Product Owners and AI/ML Engineers.
• Ensure data solutions comply with security and governance standards.

Kenntnisse

Databricks

Python

SQL

Apache Spark

CI/CD

Generative AI

Data Engineering

Machine Learning

Tools

AWS

Azure

Databricks Delta Lake

Jobbeschreibung

We are seeking a highly experienced Senior Data Engineer to design, build, and operate robust Databricks and Lakehouse data platforms. This role will focus on enabling analytics, AI, and Generative AI applications by delivering high-quality, governed, and scalable data assets. Working within product-aligned squads, you will collaborate with AI Engineers, Product Owners, and analytics teams to support cutting‑edge GenAI use cases, including Retrieval Augmented Generation (RAG), and contribute to the development of AI Engineers.

Key Responsibilities

• Design, build, and operate data solutions using Databricks components such as Delta Lake, Databricks Jobs and Workflows, and Unity Catalog.
• Develop production‑grade data pipelines using Python, SQL, and Apache Spark.
• Implement automated testing, CI/CD practices, and ensure data solutions are observable, resilient, performant, and cost‑efficient.

Data Enablement for AI/GenAI

• Enable data consumption for Generative AI use cases (RAG, AI services, agent workflows), analytics, reporting tools, and downstream systems.
• Support feature‑style and curated data access patterns required for AI and GenAI workloads.
• Build data pipelines to feed Generative AI applications, including curated knowledge datasets, structured/semi‑structured data, and metadata/lineage for AI consumption.
• Implement data patterns essential for GenAI, such as RAG, context/prompt data preparation, and model input/output/feedback data flows.

Collaboration and Governance

• Work as a senior individual contributor within a cross‑functional product squad.
• Collaborate closely with Product Owners, AI/ML Engineers, Analytics teams, and platform/security teams.
• Provide engineering input into design discussions and delivery decisions, supporting peer reviews and shared engineering standards.
• Ensure data solutions comply with enterprise security, risk, and governance standards.
• Support operational stability, participate in incident resolution, root cause analysis, and maintain documentation.

Required Skills and Qualifications

• 10-15 years of industry experience.
• Proven experience as a Senior/Lead Data Engineer (5+ years).
• Hands‑on experience with Databricks environments (2+ years).
• Strong understanding of enterprise data lake and lake house architecture (5+ years).
• Proficiency in Python, SQL, and Apache Spark (5+ years).
• Experience building and operating production‑grade data platforms (3+ years).
• Experience working in enterprise or regulated environments (5+ years).

Preferred Qualifications

• Experience enabling AI, ML, or Generative AI use cases from a data engineering perspective.
• Familiarity with RAG data patterns, feature‑style or AI‑serving datasets, and vector or embedding‑ready data workflows.
• Experience working in Agile, product‑aligned squads.
• Exposure to cloud‑native data platforms (AWS or Azure).

Skills

Databricks Delta Lake Unity Catalog Python SQL apache spark CI/CD Generative AI RAG Al Services Agent Workflows Data Warehousing Machine Learning Cloud‑native data platforms AWS Azure