MS Fabric + Pyspark Tuning - S

Tata Consultancy Services; Hyderabad, Chennai District, Bengaluru; Vor Ort; INR 1.800.000 - 3.000.000; Vollzeit

Zusammenfassung

Tata Consultancy Services in Hyderabad is seeking a data engineering expert to design, develop, and optimize data solutions using Microsoft Fabric and PySpark. You will analyze Spark workloads, build Lakehouse architectures, and improve ingestion, transformation, and processing pipelines.

Collaborate with data architects, data scientists, and business stakeholders to implement best practices for distributed computing, data quality, governance, and security.

Qualifikationen

• Experience building data solutions on Microsoft Fabric platform.
• Strong knowledge of PySpark and Spark optimization.
• Experience with data lake architectures (Delta Lake, Medallion).
• Proficient in Python and SQL; Scala preferred.
• Experience with Azure data services and cloud analytics.
• Strong collaboration with cross-functional teams.

Aufgaben

• Design, develop, and optimize data solutions using Microsoft Fabric and PySpark.
• Tune Spark workloads for performance and cost efficiency.
• Develop Lakehouse architectures for analytics platforms.
• Optimize data ingestion and processing pipelines.
• Review Spark code and implement performance improvements.
• Collaborate with data architects, scientists, and stakeholders.
• Document optimization efforts and create dashboards.

Kenntnisse

Fabric Data Engineering

Fabric Lakehouse

Fabric Notebooks

Fabric Data Factory

Fabric OneLake

Fabric Warehouse

PySpark

Spark SQL

Spark DataFrames

Spark Structured Streaming

AQE

Partitioning

Caching

Broadcast Joins

ETL/ELT

Data Modeling

Data Lake

Medallion Architecture

Delta Lake

Data Quality

Python

SQL

Scala

SQL Server

Azure SQL

PostgreSQL

Snowflake

Azure Data Services

ADLS

Power BI

Azure Monitor

Log Analytics

Git

Azure DevOps

CI/CD

Infrastructure as Code

Jobbeschreibung

Key Responsibilities

• Design, develop, and optimize data solutions using Microsoft Fabric and PySpark.
• Analyze and tune Spark workloads for performance, scalability, and cost efficiency.
• Develop and maintain Lakehouse architectures for enterprise analytics platforms.
• Optimize data ingestion, transformation, and processing pipelines.
• Perform Spark code reviews and recommend performance improvements.
• Implement partitioning, caching, indexing, and file optimization techniques.
• Monitor and troubleshoot Fabric workloads, Spark jobs, and resource utilization.
• Collaborate with data architects, data scientists, and business stakeholders.
• Design robust ETL/ELT frameworks using Fabric Data Factory and Notebooks.
• Ensure data quality, governance, security, and compliance standards.
• Implement best practices for distributed computing and big data processing.
• Create technical documentation, optimization reports, and performance metrics dashboards.

Required Skills

Microsoft Fabric

• Microsoft Fabric
• Fabric Data Engineering
• Fabric Lakehouse
• Fabric Data Factory
• Fabric Notebooks
• Fabric Real-Time Analytics
• Fabric OneLake
• Fabric Warehouse

PySpark & Spark Optimization

• PySpark Development
• Apache Spark Architecture
• Spark SQL
• Spark DataFrames
• Spark Structured Streaming
• DAG Optimization
• Catalyst Optimizer
• Adaptive Query Execution (AQE)
• Broadcast Joins
• Partitioning & Bucketing
• Shuffle Optimization
• Caching & Persistence Techniques

Data Engineering

• ETL/ELT Development
• Data Modeling
• Data Lake Architecture
• Medallion Architecture
• Delta Lake
• Data Quality Frameworks

Programming

• Python
• PySpark
• SQL
• Scala (Preferred)

Databases

• SQL Server
• Azure SQL
• PostgreSQL
• Snowflake (Preferred)

Cloud & Analytics

• Azure Data Services
• Azure Data Lake Storage (ADLS)
• Power BI
• Azure Monitor
• Log Analytics

DevOps & Automation

• Git
• Azure DevOps
• CI/CD Pipelines
• Infrastructure as Code (Preferred)