Microsoft Fabric + Pyspark Tuning

Tata Consultancy Services; Hyderabad, Chennai District, Bengaluru; Remote; INR 2,400,000 - 4,000,000; Full time

Job summary

Tata Consultancy Services is seeking a data engineering specialist to design, develop, and optimize data solutions using Microsoft Fabric and PySpark. You will tune Spark workloads, implement Lakehouse architectures, and ensure data quality and governance across enterprise analytics platforms.

Responsibilities include developing ETL/ELT pipelines, optimizing ingestion and processing, reviewing Spark code, and collaborating with data scientists and business stakeholders to deliver scalable

Qualifications

• Proficiency in Microsoft Fabric components and PySpark development.
• Experience tuning Spark workloads for performance and cost efficiency.
• Strong data modeling and ETL/ELT development skills.
• Familiarity with Lakehouse architectures and Delta Lake is a plus.

Responsibilities

• Design, develop, and optimize data solutions using Microsoft Fabric and PySpark.
• Analyze and tune Spark workloads for performance, scalability, and cost efficiency.
• Develop and maintain Lakehouse architectures for enterprise analytics platforms.
• Optimize data ingestion, transformation, and processing pipelines.
• Perform Spark code reviews and recommend performance improvements.
• Implement partitioning, caching, indexing, and file optimization techniques.
• Monitor and troubleshoot Fabric workloads, Spark jobs, and resource utilization.
• Collaborate with data architects, data scientists, and business stakeholders.
• Design robust ETL/ELT frameworks using Fabric Data Factory and Notebooks.
• Ensure data quality, governance, security, and compliance standards.
• Implement best practices for distributed computing and big data processing.
• Create technical documentation, optimization reports, and performance metrics dashboards.

Skills

Microsoft Fabric

Fabric Data Engineering

Fabric Lakehouse

Fabric Data Factory

Fabric Notebooks

Fabric Real-Time Analytics

Fabric OneLake

Fabric Warehouse

PySpark Development

Apache Spark Architecture

Spark SQL

Spark DataFrames

Spark Structured Streaming

DAG Optimization

Catalyst Optimizer

AQE

Broadcast Joins

Partitioning

Bucketing

Shuffle Optimization

Caching

Persistence Techniques

ETL/ELT Development

Data Modeling

Data Lake Architecture

Medallion Architecture

Delta Lake

Data Quality Frameworks

Python

SQL

Scala

SQL Server

Azure SQL

PostgreSQL

Snowflake

Azure Data Services

Azure Data Lake Storage

Power BI

Azure Monitor

Log Analytics

Git

Azure DevOps

CI/CD Pipelines

Infrastructure as Code

Job description

Role-MS FAbric + Pyspark Tuning

Desired Experience Range- 6-8 Years

Location- Pan India

Roles & Responsibilities-

• Design, develop, and optimize data solutions using Microsoft Fabric and PySpark.
• Analyze and tune Spark workloads for performance, scalability, and cost efficiency.
• Develop and maintain Lakehouse architectures for enterprise analytics platforms.
• Optimize data ingestion, transformation, and processing pipelines.
• Perform Spark code reviews and recommend performance improvements.
• Implement partitioning, caching, indexing, and file optimization techniques.
• Monitor and troubleshoot Fabric workloads, Spark jobs, and resource utilization.
• Collaborate with data architects, data scientists, and business stakeholders.
• Design robust ETL/ELT frameworks using Fabric Data Factory and Notebooks.
• Ensure data quality, governance, security, and compliance standards.
• Implement best practices for distributed computing and big data processing.
• Create technical documentation, optimization reports, and performance metrics dashboards.

Required Skills-

Microsoft Fabric

• Microsoft Fabric
• Fabric Data Engineering
• Fabric Lakehouse
• Fabric Data Factory
• Fabric Notebooks
• Fabric Real-Time Analytics
• Fabric OneLake
• Fabric Warehouse

PySpark & Spark Optimization

• PySpark Development
• Apache Spark Architecture
• Spark SQL
• Spark DataFrames
• Spark Structured Streaming
• DAG Optimization
• Catalyst Optimizer
• Adaptive Query Execution (AQE)
• Broadcast Joins
• Partitioning & Bucketing
• Shuffle Optimization
• Caching & Persistence Techniques

Data Engineering

• ETL/ELT Development
• Data Modeling
• Data Lake Architecture
• Medallion Architecture
• Delta Lake
• Data Quality Frameworks

Programming

• Python
• PySpark
• SQL
• Scala (Preferred)

Databases

• SQL Server
• Azure SQL
• PostgreSQL
• Snowflake (Preferred)

Cloud & Analytics

• Azure Data Services
• Azure Data Lake Storage (ADLS)
• Power BI
• Azure Monitor
• Log Analytics

DevOps & Automation

• Git
• Azure DevOps
• CI/CD Pipelines
• Infrastructure as Code (Preferred)