Databricks Architect

InfoCepts; Pune District; On-site; INR 1,500,000 - 2,200,000; Full time

Job summary

InfoCepts is seeking a data engineering leader to design, develop, and implement scalable Databricks-based data solutions on cloud platforms. You will architect end-to-end data pipelines, optimize performance, and ensure governance and security across enterprise analytics.

You will drive architecture, pipeline development, and cloud integration while collaborating with data engineers, analysts, and business stakeholders to translate requirements into robust technical solutions.

Qualifications

• Strong expertiese in Apache Spark and Databricks platform.
• Hands-on experience with Python or Scala for data engineering tasks.
• Proficiency in SQL for data manipulation and queries.
• Hands on awareness about latest features in Databricks like AI/BI, Delta Lake, and Lakehouse.
• Experience with Delta Lake, Lakehouse architecture, and data versioning.
• Knowledge of CI/CD pipelines and DevOps practices in data engineering.
• Familiarity with cloud platforms (Azure/AWS) and their data services.
• Understanding of data governance, security, and compliance framework.
• Willingness to contribute to presales activities.

Responsibilities

• Architecture design: Develop scalable, secure, high-performance data architecture using Databricks.
• Data pipeline development: Lead ETL/ELT pipelines for structured and unstructured data.
• Cloud integration: Ensure seamless Databricks integration with Azure/AWS.
• Performance optimization: Monitor and optimize Spark jobs and data workflows.
• Stakeholder collaboration: Work with data engineers, analysts, and business stakeholders to translate requirements into solutions.

Skills

Apache Spark

Databricks

Python

Scala

SQL

Delta Lake

Lakehouse architecture

CI/CD in data engineering

Cloud platforms (Azure/AWS)

Data governance & security

Tools

Airflow

Kafka

Power BI

Tableau

Job description

To lead the design, development, and implementation of scalable data solutions using Databricks on cloud platforms. The role involves architecting data pipelines, optimizing performance, and ensuring data governance and security across enterprise-level analytics initiative

s.Key Result Areas and Activities

• :Architecture Desi gn: Develop and maintain scalable, secure, and high-performance data architecture using Databric
• ks.Data Pipeline Developmen t: Lead the creation and optimization of ETL/ELT pipelines for structured and unstructured da
• ta.Cloud Integratio n: Ensure seamless integration of Databricks with cloud platforms like Azure, A
• WS.Performance Optimiza tion: Monitor and enhance the performance of data workflows and Spark jo
• bs.Stakeholder Collaboration: Work closely with data engineers, analysts, and business stakeholders to translate requirements into technical solutio

ns.Must Have Ski

• llsStrong expertisie in Apache Spark and Databricks plat
• form.Hands-on experience with Pytho n or Scala for data engineering t
• ects.Proficiency in SQL for data manipulation and quer
• ying.Hands on experience and awareness about latest features in databricks like AI/BI G e nie, Agentbricks, Databricks
• Apps.Experience with Delta Lake, Lakehouse architecture, and data versi o
• ning.Knowledge of CI/CD pipelines and DevOps practices in data enginee
• ring.Familiarity with cloud platforms (Azure/AWS) and their data serv
• ices.Understanding of data governance, security, and compliance framework
• orks.Contribute to presales and

RFPs.Good To Have

• SkillExperiences with MLflow and machine learning model dep
• oyment.Knowledge of Apache Airflow or other orchestration
• tools.Exposure to real-time data processing using Kafka or similar
• tools.Familiarity with Power BI, Tableau, or other BI
• tools.Certification in Databricks or relevant cloud techno