Data Engineer | PySpark | Databricks | Azure Data Factory

Infosys; Hyderabad, Pune District, Bengaluru; Híbrido; INR 1.200.000 - 2.400.000; Jornada completa

Descripción de la vacante

Infosys Hyderabad is seeking a data engineer to design, develop, and maintain scalable data pipelines using PySpark, Spark, Databricks, and cloud services across AWS, Azure, and GCP.

You will optimize ETL/ELT workflows, work with Delta Lake, ADLS, S3, BigQuery, Redshift, and Snowflake, ensure data quality and governance, and collaborate with analysts and stakeholders to deliver cost-efficient, secure data solutions.

Responsabilidades

• Design, develop, and maintain scalable data pipelines using PySpark, Spark, Databricks, Azure Data Factory (ADF), and AWS Glue.
• Build and optimize ETL/ELT workflows for processing large-scale structured and unstructured data.
• Develop and manage cloud-based data solutions on AWS, Azure, and GCP platforms.
• Work with Databricks, Delta Lake, ADLS, S3, BigQuery, Redshift, and Snowflake for data ingestion, transformation, and storage.
• Implement data quality, monitoring, and performance optimization best practices.
• Collaborate with data analysts, data scientists, and business stakeholders to understand requirements and deliver scalable solutions.
• Ensure data security, governance, and compliance standards are followed across cloud platforms.
• Optimize Spark jobs and SQL queries for improved performance and cost efficiency.
• Support real-time and batch data processing frameworks.
• Participate in code reviews, troubleshooting, and production support activities.
• Create technical documentation and maintain data engineering standards and best practices.

Descripción del empleo

Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark, Spark, Databricks, Azure Data Factory (ADF), and AWS Glue.
• Build and optimize ETL/ELT workflows for processing large-scale structured and unstructured data.
• Develop and manage cloud-based data solutions on AWS, Azure, and GCP platforms.
• Work with Databricks, Delta Lake, ADLS, S3, BigQuery, Redshift, and Snowflake for data ingestion, transformation, and storage.
• Implement data quality, monitoring, and performance optimization best practices.
• Collaborate with data analysts, data scientists, and business stakeholders to understand requirements and deliver scalable solutions.
• Ensure data security, governance, and compliance standards are followed across cloud platforms.
• Optimize Spark jobs and SQL queries for improved performance and cost efficiency.
• Support real-time and batch data processing frameworks.
• Participate in code reviews, troubleshooting, and production support activities.
• Create technical documentation and maintain data engineering standards and best practices.