PySpark + Databricks Engineer-Female Hiring

Infosys; Chennai District, Thiruvananthapuram; Sur place; INR 2 500 000 - 4 000 000; Plein temps

Résumé du poste

Infosys is seeking a PySpark and Databricks Engineer to design, develop, and optimize data pipelines in an Azure-based environment. The role focuses on building scalable data processing solutions using PySpark, Databricks, and Azure Data Factory.

Ideal candidates will have strong experience with ADLS, Azure SQL, and end-to-end data integration, along with CI/CD practices and collaboration with business stakeholders to deliver reliable analytics solutions.

Qualifications

• Proficient in designing data pipelines using Spark-based tech.
• Experience with PySpark and Databricks notebooks for analytics.
• Strong knowledge of Azure data services (ADF/ADLS/SQL).

Responsabilités

• Design, develop and maintain data pipelines and ETL/ELT workflows.
• Build scalable data processing solutions with PySpark and Databricks.
• Optimize Spark jobs for performance and scalability.
• Collaborate with data teams and stakeholders on data requirements.

Connaissances

PySpark

Databricks

Azure Data Factory

Azure Databricks

Python

ADLS

Azure SQL

ETL/ELT

CI/CD

Outils

Azure

Description du poste

Job Description:

Position: PySpark , Databricks Engineer

Experience: 6 to 8 years

Location: Chennai, Trivandrum

Roles & Responsibilities

• Design, develop, and maintain data pipelines using Azure Data Factory (ADF).
• Build scalable data processing solutions using Azure Databricks, Python, and PySpark.
• Develop ETL/ELT workflows for data ingestion, transformation, and loading.
• Work with Azure Data Lake Storage (ADLS), Azure SQL, and other Azure data services.
• Optimize Spark jobs and Databricks notebooks for performance and scalability.
• Implement data quality checks, validation rules, and monitoring processes.
• Create and manage end-to-end data integration solutions across multiple data sources.
• Collaborate with business stakeholders and data teams to understand data requirements.
• Troubleshoot and resolve data pipeline and performance issues.
• Follow coding standards, version control, and CI/CD best practices.
• Support data warehousing and analytics initiatives.
• Prepare technical documentation and maintain deployment artifacts.