PySpark Databricks Engineer

Infosys; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Infosys in Bengaluru is seeking a Data Analytics Engineer to design, implement, and optimize ETL/ELT pipelines using PySpark and Databricks. You will build scalable data processing solutions for large datasets and ensure performance and security standards.

The role requires strong Python, SQL, and cloud data tooling experience, with hands-on work on Delta Lake and data lake architectures. Collaboration with stakeholders and agile teams is expected in a dynamic data environment.

Qualifications

• Strong experience with PySpark and Apache Spark.
• Hands-on experience with Databricks.
• Proficiency in Python programming.
• Experience in building ETL pipelines and data transformations.
• Strong SQL knowledge and query optimization skills.
• Experience with Azure Data Factory (ADF) / AWS Glue or similar tools.
• Knowledge of Delta Lake, Medallion Architecture, and Data Lakes.
• Experience with Git, CI/CD pipelines, and Agile methodologies.

Responsibilities

• Design, develop, and maintain ETL/ELT pipelines using PySpark and Databricks.
• Build scalable data processing solutions for large datasets.
• Develop and optimize Spark jobs for performance and efficiency.
• Implement data ingestion pipelines from various structured and unstructured data sources.
• Work with Delta Lake, Data Lake, and cloud storage solutions.
• Collaborate with business stakeholders, data analysts, and architects to understand data requirements.
• Ensure data quality, governance, and security standards are followed.
• Troubleshoot performance issues and optimize data workflows.
• Participate in code reviews and follow DevOps best practices.

Skills

PySpark

Apache Spark

Python programming

ETL pipelines

SQL

Agile methodologies

CI/CD pipelines

Git

Education

BSc

BTech

BCA

MTech

MCA

MSc

Tools

Databricks

Azure Data Factory (ADF)

AWS Glue

Delta Lake

Data Lakes

Job description

Educational Requirements

• BSc


• BTech


• BCA


• MTech


• MCA


• MSc

Service Line

Data Analytics Unit

Responsibilities

• Design, develop, and maintain ETL/ELT pipelines using PySpark and Databricks.


• Build scalable data processing solutions for large datasets.


• Develop and optimize Spark jobs for performance and efficiency.


• Implement data ingestion pipelines from various structured and unstructured data sources.


• Work with Delta Lake, Data Lake, and cloud storage solutions.


• Collaborate with business stakeholders, data analysts, and architects to understand data requirements.


• Ensure data quality, governance, and security standards are followed.


• Troubleshoot performance issues and optimize data workflows.


• Participate in code reviews and follow DevOps best practices.

Required Skills

• Strong experience with PySpark and Apache Spark.


• Hands-on experience with Databricks.


• Proficiency in Python programming.


• Experience in building ETL pipelines and data transformations.


• Strong SQL knowledge and query optimization skills.


• Experience with Azure Data Factory (ADF) / AWS Glue or similar tools.


• Knowledge of Delta Lake, Medallion Architecture, and Data Lakes.


• Experience with Git, CI/CD pipelines, and Agile methodologies.

Preferred Skills

• Technology->Data Engineering->Databricks


• Technology->Big Data - Data Processing->PySpark