Databricks Architect

InfoCepts; Pune District; On-site; INR 2,500,000 - 5,000,000; Full time

Job summary

InfoCepts in Pune is seeking a Databricks Architect to lead design, development and deployment of scalable data solutions on cloud platforms. You will architect data pipelines, optimize Spark jobs, and ensure governance and security across analytics initiatives.

Collaborate with data engineers, analysts and stakeholders to translate requirements into robust technical solutions. Bring deep expertise in Spark, Databricks, Python or Scala, SQL, Delta Lake, and CI/CD practices in data engineering.

Qualifications

• Strong expertise in Apache Spark and Databricks platform.
• Hands-on experience with Python or Scala for data engineering tasks.
• Proficiency in SQL for data manipulation and querying.
• Experience with Delta Lake, Lakehouse architecture, and data versioning.
• Knowledge of CI/CD pipelines and DevOps practices in data engineering.
• Familiarity with cloud platforms (Azure/AWS/GCP) and their data services.
• Understanding of data governance, security, and compliance frameworks.

Responsibilities

• Architecture Design: Develop and maintain scalable, secure, and high-performance data architecture using Databricks.
• Data Pipeline Development: Lead the creation and optimization of ETL/ELT pipelines for structured and unstructured data.
• Cloud Integration: Ensure seamless integration of Databricks with cloud platforms like Azure, AWS, or GCP.
• Performance Optimization: Monitor and enhance the performance of data workflows and Spark jobs.
• Stakeholder Collaboration: Work closely with data engineers, analysts, and business stakeholders to translate requirements into technical solutions.

Skills

Apache Spark

Databricks platform

Python

Scala

SQL

Delta Lake

Lakehouse architecture

CI/CD in data engineering

Cloud platforms

Data governance & security

DevOps practices

Education

Bachelor’s or Master’s degree in Computer Science/IT or related field

Tools

Databricks platform

Apache Spark

Python

Scala

SQL

Delta Lake

CI/CD tooling

Cloud services (Azure/AWS/GCP)

Job description

Position: Databricks Architect

Purpose of the Position: To lead the design, development, and implementation of scalable data solutions using Databricks on cloud platforms. The role involves architecting data pipelines, optimizing performance, and ensuring data governance and security across enterprise-level analytics initiatives.

Location: Pune/Nagpur/Bangalore/Chennai

Type of Employment: Full time

Key Result Areas and Activities

• Architecture Design: Develop and maintain scalable, secure, and high-performance data architecture using Databricks
• Data Pipeline Development: Lead the creation and optimization of ETL/ELT pipelines for structured and unstructured data.
• Cloud Integration: Ensure seamless integration of Databricks with cloud platforms like Azure, AWS, or GCP.
• Performance Optimization: Monitor and enhance the performance of data workflows and Spark jobs.
• Stakeholder Collaboration: Work closely with data engineers, analysts, and business stakeholders to translate requirements into technical solutions.

Roles and Responsibilities

Work and Technical Experience

Mandatory Skills

• Strong expertise in Apache Spark and Databricks platform.
• Hands‑on experience with Python or Scala for data engineering tasks.
• Proficiency in SQL for data manipulation and querying.
• Experience with Delta Lake, Lakehouse architecture, and data versioning.
• Knowledge of CI/CD pipelines and DevOps practices in data engineering.
• Familiarity with cloud platforms (Azure/AWS/GCP) and their data services.
• Understanding of data governance, security, and compliance frameworks.

Good To Have Skills

• Experience with MLflow and machine learning model deployment.
• Knowledge of Apache Airflow or other orchestration tools.
• Exposure to real‑time data processing using Kafka or similar tools.
• Familiarity with Power BI, Tableau, or other BI tools.
• Certification in Databricks or relevant cloud technologies.

Qualification

• Experience: Minimum 10+ years in data engineering/architecture roles, with at least 3 years on Databricks.
• Education: Bachelor’s or master’s degree in computer science, Information Technology, or related field.

Qualities

• Strong analytical and problem‑solving skills.
• Excellent communication and stakeholder management.
• Ability to mentor and guide junior team members.
• Adaptability to fast‑paced and evolving technology environments.
• Proactive approach to learning and innovation.