Data Engineer-Databricks & Pyspark

Pineswift Technologies; Gurugram District; Vor Ort; INR 900.000 - 1.500.000; Vollzeit

Zusammenfassung

Pineswift Technologies is seeking a Data Engineer to design and implement data pipelines using Databricks, PySpark, Python and SQL. You will integrate data from multiple sources and build analytical data models for reporting and ML use cases.

The role requires strong ETL/ELT experience, data quality awareness, and optimization of Spark workloads. You will ensure data security and collaborate on production deployment and maintenance in a banking context.

Qualifikationen

• Hands-on experience with Databricks and PySpark.
• Proficiency in Python and SQL.
• Experience building ETL/ELT pipelines and Data Lake.
• Understanding of distributed processing and performance tuning.
• Knowledge of banking data and metrics.
• Awareness of data quality requirements.

Aufgaben

• Develop data ingestion and transformation pipelines using Databricks, PySpark, Python and SQL.
• Integrate data from multiple sources and prepare datasets for reporting, analytics and machine-learning use cases.
• Build reusable transformation components and analytical data models.
• Implement data-quality checks, reconciliation, exception handling and pipeline monitoring.
• Optimize Spark workloads for performance, reliability and cost.
• Apply data security, access controls, lineage and documentation standards.
• Support automated deployment, production troubleshooting and ongoing pipeline maintenance.
• Translate banking business requirements into sustainable data solutions.

Kenntnisse

Databricks

PySpark

Python

SQL

ETL/ELT pipelines

Data Lake

Distributed processing

Performance tuning

Banking domain knowledge

Data quality

Jobbeschreibung

Greetings from Pineswift!!

We have a few open requirements for Data Englineers. Below are the details:

Key Responsibilities

• Develop data ingestion and transformation pipelines using Databricks, PySpark, Python and SQL.
• Integrate data from multiple sources and prepare datasets for reporting, analytics and machine-learning use cases.
• Build reusable transformation components and analytical data models.
• Implement data-quality checks, reconciliation, exception handling and pipeline monitoring.
• Optimize Spark workloads for performance, reliability and cost.
• Apply data security, access controls, lineage and documentation standards.
• Support automated deployment, production troubleshooting and ongoing pipeline maintenance.
• Translate banking business requirements into sustainable data solutions.

Required Skills & Experience

• Strong hands-on experience with Databricks and PySpark
• Proficiency in Python and SQL
• Experience building ETL/ELT pipelines and working with Data Lake
• Understanding of distributed processing, workflow orchestration and performance tuning
• Working knowledge of the banking domain, including banking data and metrics
• Understanding of data-quality requirements, including validation, completeness, accuracy and consistency