Senior Data Engineer (Azure Databricks)

Talpro; Bengaluru Urban; On-site; INR 1,004,000 - 1,674,000; Full time

Job summary

Talpro seeks a Senior Data Engineer (Azure Databricks) for a 6-month contract-to-hire role in Bangalore. You will design and implement high-performance data pipelines, collaborate with analysts and developers, and align to UK business hours with a hybrid work setup.

Required 6–8 years in data engineering with 5–6 years in Databricks and experience delivering 2+ end-to-end projects. Immediate joiners preferred; on Talpro payroll.

Qualifications

• 6-8 years of overall Data Engineering experience, preferably on Microsoft Azure.
• 5-6 years hands-on Databricks experience with 2+ end-to-end Databricks projects.
• Azure Databricks expertise with Spark internals and performance tuning.
• Experience building data pipelines using Azure Data Factory and Azure Databricks; Python/PySpark proficiency.

Responsibilities

• Design and implement data ingestion and transformation pipelines using Azure Databricks and Spark.
• Develop scalable frameworks for data ingestion and transformation of large datasets.
• Perform Spark performance tuning and ensure cost-effective data pipelines.
• Collaborate with data analysts/scientists and support reporting tools and APIs.
• Implement CI/CD practices for data pipelines in an Agile/DevOps setup.

Skills

Azure Databricks

Apache Spark

Python

ETL/ELT

Azure Data Factory

Data Warehousing

DevOps & CI/CD

Geospatial data

NoSQL & Big Data

Tools

Informatica

Talend

MongoDB

Cassandra

Neo4j

Kafka

NiFi

Job description

Bangalore South, India | Posted on 08/18/2025

• Assigned Recruiter(s) Recruitment Manager
• Date Opened 08/18/2025
• Job Type Contract
• Country India

About Us

Talpro is leading the way in transforming the talent acquisition landscape. We deliver innovative, sustainable, and cost-effective recruitment solutions tailored to today’s business needs. Our mission is to offer comprehensive hiring strategies that address immediate recruitment demands while laying the groundwork for long-term success.

Job Description

Role Type: Fixed Term Direct Contract with Talpro

Contract Duration:6 months (Fixed Term Contract)

Years of Experience: 6+ Yrs.

Max CTC Offered: INR 1.50 L Per Months

Notice Period: Immediate Joiners Only

Work Mode: Hybrid (3 Days from Office Weekly)

Domain: Any IT Domain

About the Role

Talpro is seeking a Senior Data Engineer (Azure Databricks) for an internal position to be deployed at a client location. This is a 6-month contract-to-hire role – if you excel during the contract, there is potential to transition into a permanent position. You will be on Talpro’s payroll and work with the client’s team in Bangalore. The work model is hybrid (mix of on-site and remote as needed) and aligned to UK business hours (approximately 2 PM to 7 PM IST).

Key Responsibilities

• Data Pipeline Development: Design and implement highly performant, metadata-driven data ingestion and transformation pipelines from multiple sources using Azure Databricks and Apache Spark. This includes building both streaming and batch data processes to handle large-scale data in real time and in bulk.
• Framework & Optimization: Develop scalable and reusable frameworks for data ingestion and transformation of large datasets. Perform Spark performance tuning and optimization to ensure data pipelines are efficient and cost-effective. Provide technical guidance on complex data problems (including handling geospatial data) using Spark DataFrames and other big data tools.
• Data Quality & Integration: Design and implement data quality checks and processes to ensure data integrity is maintained through the pipeline. Integrate end-to-end data pipelines from source systems to target data repositories (data lakes/data warehouses), ensuring consistency, quality, and reliability of data at all times.
• Collaboration & Tooling: Work closely with cross-functional teams (data analysts, data scientists, application developers) to support additional project components such as reporting tools, APIs, and search applications. Evaluate various tools and technologies (e.g., Azure Data Factory, Azure Databricks features, etc.) against project requirements to recommend the best fit.
• Agile Delivery: Operate within an Agile/DevOps delivery environment. Participate in iterative sprint planning, deliver proof-of-concepts (POCs), and guide the project from concept to production. Implement continuous integration/continuous deployment (CI/CD) practices for data pipelines and ensure solutions are delivered in a timely, iterative manner.

Requirements

• Required Qualifications
• Experience: 6-8 years of overall Data Engineering experience, preferably in building data solutions on Microsoft Azure. Within this, at least 5 -6 years of hands‑on Databricks experience, including delivering 2+ end-to-end Databricks projects from concept to production.
• 3 - 4 years of experience in designing Databricks Data Platform and familiar with Non functional requirements and best practices for designing data platform.
• Azure Databricks Expertise: Proven experience in designing Databricks-based data platforms, with a strong understanding of Spark internals, performance tuning, and best practices for building data lakes/warehouses on Databricks. Familiarity with addressing non-functional requirements (scalability, reliability, security) in a data platform design.
• Data Pipeline Skills: Hands‑on experience building data pipelines using Azure Data Factory and Azure Databricks. Proficient in developing data integration workflows using Python (including PySpark) for ETL/ELT processes.
• Certifications: Databricks Certified Data Engineer Associate (at least one Databricks certification is required). Microsoft Azure Data Engineer Associate certification is highly desired.
• Data Warehousing & ETL: Experience designing and delivering solutions on the Azure Data Analytics stack (e.g., Azure Data Lake, Azure Synapse Analytics). Background in building data warehouse solutions using Informatica or Talend.
• Data Management: Strong understanding of data management best practices – data profiling, data cleansing, data transformation, and data quality assurance. Experience implementing routines for data standardization, deduplication, and matching to ensure high data quality.
• Preferred Qualifications (Nice to Have)
• DevOps & CI/CD: Experience working in a DevOps-oriented environment using tools like Azure DevOps (formerly VSTS), Chef, Puppet, or Terraform for infrastructure automation and deployment. Knowledge of setting up CI/CD pipelines for data projects is a plus.
• Diverse Data Experience: Exposure to working with structured and unstructured data, including experience handling geospatial data or large imaging datasets. Ability to incorporate such data types into pipelines and ensure performance.
• NoSQL & Big Data Tech: Familiarity with open-source NoSQL databases and graph databases (e.g., MongoDB, Cassandra, Neo4j) and understanding of how to integrate or migrate data to/from these systems.
• Streaming & IoT: Experience with streaming data platforms and event ingestion services such as Azure Event Hub, Azure IoT Hub, Apache Kafka, or Apache NiFi. Ability to design pipelines for real-time data processing and event-driven architectures.

Why Join Talpro?

• Innovative Projects: Work on cutting-edge data engineering projects using Azure Databricks and the modern Azure data stack, tackling challenges in big data and analytics.
• Contract-to-Hire Opportunity: The 6-month C2H model allows you to prove your skills with the possibility of a long-term role. You’ll get a foot in the door with a leading client and the stability of Talpro’s backing.
• Flexible Work Environment: Enjoy a hybrid work setup with the flexibility of remote work combined with collaborative on-site days. Align with UK timing for better work-life balance if you prefer later start times.
• Professional Growth: Leverage your expertise and certifications while also having opportunities to learn new technologies (cloud services, DevOps tools, streaming platforms). Talpro supports continuous learning and certification for our employees.
• Note: This position is for Bangalore-based candidates (or those willing to relocate) as on-site presence is required on a case-to-case basis. Immediate joiners are preferred due to project urgency.