Pyspark Data Engineer

Virtusa; Dubai; In loco; AED 320.000 - 520.000; Tempo pieno

Descrizione del lavoro

Virtusa is seeking a highly skilled Data Engineer with 6+ years of experience to design, build, and optimize data pipelines using PySpark and Informatica BDM. Banking domain experience is essential, with a focus on data integration, governance, and secure processing.

The role involves collaborating with data architects, analysts, and stakeholders to deliver scalable ETL/ELT solutions, data lake/warehouse support, and ensuring regulatory compliance across banking data assets.

Competenze

• 6+ years of experience in Data Engineering or related roles.
• Strong hands-on experience with PySpark for large-scale data processing.
• Solid experience with Informatica BDM (Big Data Management).
• Experience with Big Data technologies and distributed data processing frameworks.
• Strong knowledge of SQL and data modeling concepts.
• Experience with data integration, ETL development, and workflow orchestration.
• Familiarity with cloud platforms or big data ecosystems (Hadoop/Spark environments).
• Good understanding of data governance, security, and compliance requirements in banking.

Mansioni

• Design, develop, and maintain scalable data pipelines using PySpark for processing large volumes of structured and unstructured data.
• Develop and manage ETL/ELT workflows using Informatica BDM.
• Work closely with data architects, analysts, and business stakeholders to understand data requirements and translate them into technical solutions.
• Optimize data processing jobs to ensure high performance and reliability.
• Implement data quality, data governance, and data validation processes.
• Integrate data from multiple sources including databases, APIs, and enterprise systems.
• Support data warehouse and data lake initiatives within the organization.
• Ensure compliance with banking regulatory and security standards.
• Troubleshoot and resolve data pipeline issues and performance bottlenecks.
• Participate in code reviews, testing, and deployment processes to maintain high development standards.

Conoscenze

PySpark

Informatica BDM

SQL

Data modeling

ETL development

Workflow orchestration

Big Data technologies

Cloud platforms

Data governance

Strumenti

Informatica BDM

Descrizione del lavoro

Job Requirements

"We are seeking a highly skilled Data Engineer with 6+ years of experience in building and optimizing data pipelines and data integration solutions. The ideal candidate should have strong hands-on experience with PySpark as a primary skillset, along with Informatica BDM (Big Data Management) expertise. Prior experience working within the banking or financial services domain is essential."

Key Responsibilities

Design, develop, and maintain scalable data pipelines using PySpark for processing large volumes of structured and unstructured data. Develop and manage ETL/ELT workflows using Informatica BDM. Work closely with data architects, analysts, and business stakeholders to understand data requirements and translate them into technical solutions. Optimize data processing jobs to ensure high performance and reliability.Implement data quality, data governance, and data validation processes. Integrate data from multiple sources including databases, APIs, and enterprise systems. Support data warehouse and data lake initiatives within the organization. Ensure compliance with banking regulatory and security standards. Troubleshoot and resolve data pipeline issues and performance bottlenecks. Participate in code reviews, testing, and deployment processes to maintain high development standards.

Required Skills & Qualifications

6+ years of experience in Data Engineering or related roles.Strong hands-on experience with PySpark for large-scale data processing.Solid experience with Informatica BDM (Big Data Management).Experience working with Big Data technologies and distributed data processing frameworks. Strong knowledge of SQL and data modeling concepts. Experience with data integration, ETL development, and workflow orchestration. Familiarity with cloud platforms or big data ecosystems (Hadoop/Spark environments) is a plus. Good understanding of data governance, security, and compliance requirements in banking.

Domain Requirement: Mandatory experience in the Banking or Financial Services domain, preferably working with enterprise data platforms, regulatory reporting systems, or large-scale financial datasets.

Preferred Qualifications: Experience with data lake architectures and data warehouse platforms. Exposure to Agile development methodologies. Knowledge of Python-based data engineering frameworks and automation.

Work Experience

5-7Years