Azure Data Engineer (Databricks, PySpark & GenAI) | IBM | Bangalore

International Business Machines Corporation; Bengaluru; On-site; INR 4,200,000 - 6,000,000; Full time

Job summary

IBM in Bangalore is seeking an Azure Data Engineer with 6+ years of experience to own cloud data pipelines and Databricks workloads. You will design, build, and optimize ADF pipelines, PySpark/Delta Live Tables, and streaming architectures, while integrating GenAI tools and GitHub Copilot into enterprise data workflows.

The role requires strong Python/SQL, Databricks expertise, and hands-on experience with cost optimization and governance in an on-site setting.

Qualifications

• Bachelor’s or Master’s degree in a technical field.
• 6+ years in cloud data engineering, data warehousing, and ETL.
• Strong Python and SQL proficiency with cloud data platforms.
• Hands-on with ADF and Databricks stack including DLT/Delta Live Tables.

Responsibilities

• Design, develop, test, and deploy data pipelines using ADF and SQL.
• Architect and maintain Databricks environments with PySpark and DLT.
• Tune DLT performance, reduce latency, and optimize queries.
• Apply Photon-based cost optimization and serverless compute strategies.
• Develop ELT frameworks with ADLS and DAB integrations.
• Integrate GenAI tools (Genie, Mosaic AI, LLMs) and GitHub Copilot.
• Collaborate with data scientists, engineers, and client stakeholders.
• Maintain data governance, uptime SLAs, and runbooks.

Skills

Python scripting

SQL querying

PySpark

Databricks

Delta Live Tables

Structured Streaming

Unity Catalog

Photon

Serverless compute

GenAI tools

GitHub Copilot

Education

Bachelor’s or Master’s degree in Computer Science, Data Science, IT, Software Engineering

Tools

Azure Data Factory (ADF)

Databricks

Delta Live Tables (DLT)

Unity Catalog

Photon

GitHub Copilot

Job description

Position Summary

Driving advanced cloud data engineering, Databricks analytics, PySpark stream processing, and generative AI integrations in Bangalore, India, IBM is actively seeking an experienced Azure Data Engineer with 6+ years of relevant professional experience for a permanent full-time engagement. In this vital data engineering and digital transformation role, you will take full operational, tactical, and technical ownership of Azure Data Factory pipelines, advanced SQL querying, Python scripting, Databricks Delta Live Tables, structured streaming, DLT pipeline performance optimization, and Databricks cost optimization utilizing Photon and serverless frameworks. Bringing deep technical proficiency in modern data platforms and emerging AI/GenAI tools such as Genie, Mosaic AI, Large Language Models (LLMs), AI agents, and GitHub Copilot, you will empower an elite international technology consulting and enterprise software corporation to achieve uncompromised data integrity, ultra-high throughput processing, and superior business intelligence delivery. This prestigious career opportunity offers competitive remuneration, enterprise client exposure, and professional growth within a leading global technology and IT consulting corporation.

Detailed Job Description

As an Azure Data Engineer supporting IBM on a professional on-site basis in Bangalore, India, you will take full operational, tactical, and technical ownership of cloud data architecture, ETL pipeline engineering, Databricks optimization, and cross-functional engineering collaboration. Your day-to-day responsibilities include designing, developing, and deploying robust data pipelines utilizing Python coding, advanced SQL, Azure Data Factory (ADF), and sophisticated data modeling techniques, architecting and optimizing Databricks solutions including PySpark, Delta Live Tables (DLT), Structured Streaming, Unity Catalog, and Databricks Asset Bundles (DAB), executing DLT pipeline performance tuning and Databricks cost optimization strategies via Photon and serverless/non-serverless compute configurations, integrating cutting-edge AI and generative AI tools such as Genie, Mosaic AI, LLMs, AI agents, and GitHub Copilot into enterprise data workflows, collaborating proactively with data scientists, software engineering squads, and technical recruitment leadership like Nidhi Kumari at IBM, adhering to strict corporate data security and governance standards, and managing application requirements by submitting your updated CV along with your total experience, relevant Azure data engineering experience, current location, notice period, and real-time project skill breakdown via direct email to [e-mail].

Key Responsibilities

• Design, develop, test, and deploy enterprise-grade cloud data engineering pipelines utilizing Azure Data Factory (ADF) and advanced SQL.
• Architect and maintain scalable Databricks environments utilizing PySpark, Delta Live Tables (DLT), Structured Streaming, and Unity Catalog.
• Execute rigorous DLT pipeline performance optimization, latency reduction, and query tuning across distributed clusters.
• Implement comprehensive Databricks cost optimization strategies leveraging Photon acceleration and serverless/non-serverless compute models.
• Develop robust Extract, Load, Transform (ELT) frameworks, Azure Data Lake Storage (ADLS) integrations, and Databricks Asset Bundles (DAB).
• Integrate advanced AI and Generative AI tools including Genie, Mosaic AI, Large Language Models (LLMs), AI agents, and GitHub Copilot.
• Write high-performance Python scripts and advanced SQL queries for complex data manipulation and schema management.
• Collaborate proactively with cross-functional data science, software engineering, and client stakeholder squads.
• Ensure strict adherence to enterprise data governance standards, compliance frameworks, and uptime service level agreements.
• Maintain meticulous data architecture blueprints, pipeline documentation, version control records, and technical runbooks.

Required Qualifications & Skills

• Bachelor’s or Master’s degree in Computer Science, Data Science, Information Technology, Software Engineering, or a related technical discipline.
• Minimum of 6+ years of progressive professional hands‑on experience in cloud data engineering, data warehousing, and ETL development.
• Strong, demonstrable operational proficiency in Python coding/scripting and advanced SQL.
• Comprehensive working knowledge and practical hands‑on experience with Azure Data Factory (ADF) and advanced data modeling.
• Specialized expertise in Databricks technologies including PySpark, Delta Live Tables (DLT), Structured Streaming, and Unity Catalog.
• Proven experience in DLT pipeline performance optimization and Databricks cost optimization (Photon, Serverless/Non-Serverless).
• Solid practical grasp of ELT architectures, Azure Data Lake Storage (ADLS), and Databricks Asset Bundles (DAB).
• Strong analytical problem‑solving skills and effective stakeholder communication capabilities.
• Mandatory legal work authorization and physical availability for professional on‑site operations in Bangalore, India.

Nice-to-Have Skills

• Professional vendor certifications such as Databricks Certified Data Engineer Professional, Azure Data Engineer Associate (DP-203), or AWS Certified Data Analytics.
• Prior professional working experience supporting Tier-1 global technology enterprises, financial institutions, or SaaS corporations.
• Hands‑on familiarity with Generative AI tools (Genie, Mosaic AI, LLMs, AI agents, GitHub Copilot) and AI-assisted development.
• Experience utilizing enterprise agile tracking and IT service desk platforms (Jira, Confluence, ServiceNow).
• Active participation in data engineering user groups, Apache Spark forums, or open‑source community projects.

Application Information

• Recruiter: IBM Recruitment Division
• Contact Name: Nidhi Kumari (Technical Recruitment Specialist at IBM)
• Email: [e-mail]
• Phone:
• Company Website: https://www.ibm.com
• Salary/Rate: Market Competitive / Enterprise Data Engineering Package
• Work Schedule: On-Site Professional Operations (Bangalore, India)
• Deadline: Open until filled (Urgent Bangalore Azure Data Engineer Opening)
• Notice Period: Immediate to 30 Days Preferred
• Contract Duration: Permanent / Full-Time Employment

Recruitment Pro Tip

When applying for this Azure Data Engineer position with IBM in Bangalore, ensure your resume and application email explicitly highlight your Databricks DLT optimization tenure, PySpark streaming expertise, and real-time project experience with GenAI tools.