InfoBeans Technologies - Data Engineer - Azure Databricks

InfoBeans; Indore District; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

InfoBeans is seeking an experienced Data Engineer to design and implement scalable data processing solutions using Python, PySpark, SQL, and Azure Databricks/ADF. The role emphasizes building reusable pipelines, optimizing transformations, and delivering reliable data solutions while collaborating with analysts and stakeholders.

You will contribute to scalable data frameworks, ensure data quality, and optimize performance across large datasets in a cloud-based environment.

Qualifications

• 10–13 years of experience in data engineering or related software engineering roles.
• Strong hands-on proficiency in Python and PySpark for scalable data processing.
• Advanced SQL skills for complex data extraction, transformation, and analysis.

Responsibilities

• Design, develop, and maintain scalable data processing solutions using Python and PySpark.
• Build and orchestrate data pipelines with Azure Databricks and Azure Data Factory (ADF).
• Develop reusable, modular code frameworks and ensure clean, well-documented data processing.

Skills

Python

PySpark

SQL

Azure Databricks

Azure Data Factory

Git

Tools

Git

Job description

About The Role

We are looking for an experienced Data Engineer with strong hands-on expertise in Python, PySpark, SQL, and cloud-based data engineering. The role involves designing and developing scalable data processing solutions, building reusable frameworks, optimizing data transformation pipelines, and delivering reliable data solutions using Azure Databricks and Azure Data Factory (ADF).

Key Responsibilities

• Design, develop, and maintain scalable data processing solutions using Python and PySpark.


• Write complex SQL queries for data extraction, transformation, validation, reconciliation, and analysis.


• Build and maintain robust data pipelines and workflows using Azure Databricks and Azure Data Factory (ADF).


• Develop reusable, modular, and scalable code frameworks to support efficient data engineering.


• Design and implement data transformations and processing logic for large datasets.


• Develop solutions from scratch, following clean coding practices, reusable design patterns, and comprehensive documentation standards.


• Optimize existing code, Spark jobs, SQL queries, and data pipelines to improve performance, scalability, and resource utilization.


• Troubleshoot data pipeline failures, processing bottlenecks, and data quality issues.


• Collaborate with data engineers, analysts, architects, and business stakeholders.


• Ensure data accuracy, consistency, integrity, and reliability throughout the data processing lifecycle.


• Participate in technical design discussions, code reviews, and continuous improvement initiatives.

Required Skills & Experience

• 10 - 13 years of experience in data engineering, data development, or related software engineering roles.


• Strong hands-on proficiency in Python, including core programming concepts, data structures, modular design, and reusable code development.


• Strong experience with PySpark for distributed data processing and large-scale data transformations.


• Advanced SQL skills, including complex queries, joins, aggregations, analytical functions, and query optimization.


• Hands-on experience with Azure Databricks for data engineering and transformation workloads.


• Experience building and orchestrating data pipelines using Azure Data Factory (ADF).


• Strong understanding of data engineering principles, ETL/ELT processes, data transformation, and pipeline orchestration.


• Experience designing scalable and reusable data processing frameworks.


• Experience working with Git, code reviews, testing, and deployment practices.


• Excellent analytical, communication, and cross-functional collaboration skills.

(ref:hirist.tech)