Data Engineer- Databricks + AI

InfoCepts; Pune District; On-site; INR 2,400,000 - 4,000,000; Full time

Job summary

InfoCepts is seeking an experienced Data Engineer- Databricks + AI to build and modernize AI agents and data pipelines on the Databricks platform. You will work with architects to translate requirements into scalable implementations using PySpark and Python within the Databricks ecosystem.

The role requires 6+ years in data engineering, strong SQL and data warehousing knowledge, and hands-on Databricks experience including Unity Catalog and Autoloader.

Qualifications

• 6+ years of experience in data engineering and 4+ years on Databricks.
• Proficient in PySpark and Python.
• Experience building AI agents and Genies on Databricks.
• Hands-on with SQL and data warehousing concepts.

Responsibilities

• Translate requirements into technical implementations per AI First strategy.
• Develop, implement, and maintain Agents and pipelines in Databricks using Python/pyspark.
• Code, test, and document data systems for robust analytics applications.

Skills

Databricks

PySpark

Python

RAG pipelines

Langchain

Genie Agent

SQL

Data warehousing

Unity Catalog

Autoloader

Education

Bachelors in CS/Engineering

Technical certifications

Tools

Databricks

Unity Catalog

Notebook

Autoloader

Genie

Job description

Position: Data Engineer- Databricks + AI

Purpose of the Position

Build and enhance AI Agents and modern pipelines leveraging Databricks platform features and services.

Location

Nagpur/ Pune/ Bangalore/ Chennai

Key Responsibilities

• Work with account/project architects to understand and translate requirements into technical implementations following the AI First strategy.
• Develop, implement, and maintain Agents and pipelines using python/pyspark within Databricks eco system by using the native support.
• Code, test, and document new or modified data systems to create robust and scalable applications for data analytics.

Roles and Responsibilities

Work and Technical Experience

Mandatory Skills

• 6+ years of overall experience with 4+ years of hands-on Databricks experience, PySpark and Python
• Understanding of Databricks Agent framework Agent Bricks. Built RAG pipelines using langchain
• Capability to build Genie Agent Chatbot and fine tuning
• Worked on complex business processes and implementations of Big Data Projects
• Understand end to end lakehouse build up on Databricks and usage of Genie code.
• Thorough understanding of Databricks Unity Catalog, Notebook, Autoloader, Data apps, Genie and other key services/features
• Proficiency in SQL and data manipulation skills
• Understanding of data warehousing concepts and technologies.

Desirable Skills

• Understanding of AWS platform and its key services like S3, Lambda etc.
• Familiarity with ETL processes and tools for data extraction, transformation, and loading.
• Familiarity with AI/BI dashboards in Databricks, Agentic workflows in Databricks

Qualifications

• Bachelor’s degree in computer science, engineering, or related field
• Demonstrated continued learning through one or more technical certifications or related methods
• 6+ years of relevant experience in Data Engineering
• Relevant experience in Retail domain

Qualities

• Self-motivated and focused on delivering outcomes for a fast-growing team and firm
• Able to communicate persuasively through speaking, writing, and client presentations
• Able to consult, write, and present persuasively
• Able to work in a self-organized and cross-functional team
• Able to iterate based on new information, peer reviews, and feedback
• Able to work with teams and clients in different time zones
• Research focused mindset