Data Engineer (PySpark+SQL)

Fragma Data Systems; Bengaluru; On-site; INR 1,000,000 - 1,500,000; Full time

Job summary

A data solutions company in Bengaluru is seeking a Data Engineer proficient in PySpark and SQL. Responsibilities include data transformation and engineering, with a focus on data lakes and Azure services. Ideal candidates will have a strong background in Big Data programming and excellent communication skills.

Qualifications

• 5+ years of experience in Data Engineering.
• Strong knowledge of PySpark and SQL for data transformation.
• Experience with Azure services and experience in data warehouse migrations.

Skills

Good experience in PySpark

Good experience in SQL DBs

Excellent experience in Big Data programming

Good at ELT architecture

Good customer communication

Good Analytical skill

Tools

Azure Synapse

Azure SQL DWH

Azure Data Lake

Databricks

Power BI

Job description

Overview

Join to apply for the Data Engineer (PySpark+SQL) role at Fragma Data Systems.

This range is provided by Fragma Data Systems. Your actual pay will be based on your skills and experience — talk with your recruiter to learn more.

Base pay range

Must-Have Skills

• Good experience in PySpark - Including DataFrame core functions and Spark SQL
• Good experience in SQL DBs - Be able to write queries including fair complexity.
• Should have excellent experience in Big Data programming for data transformation and aggregations
• Good at ELT architecture. Business rules processing and data extraction from Data Lake into data streams for business consumption.
• Good customer communication.
• Good Analytical skill

Technology Skills (Good To Have)

• Building and operationalizing large scale enterprise data solutions and applications using one or more of AZURE data and analytics services in combination with custom solutions - Azure Synapse/Azure SQL DWH, Azure Data Lake, Azure Blob Storage, Spark, HDInsights, Databricks, CosmosDB, EventHub/IOTHub.
• Experience in migrating on-premise data warehouses to data platforms on AZURE cloud.
• Designing and implementing data engineering, ingestion, and transformation functions
• Azure Synapse or Azure SQL data warehouse
• Spark on Azure is available in HD insights and data bricks

Good To Have

• Experience with Azure Analysis Services
• Experience in Power BI
• Experience with third-party solutions like Attunity/Stream sets, Informatica
• Experience with PreSales activities (Responding to RFPs, Executing Quick POCs)
• Capacity Planning and Performance Tuning on Azure Stack and Spark.

Skills

PySpark, SQL, Data engineering, Data Warehouse (DWH) and Spark

Seniority level

• Mid-Senior level

Employment type

• Full-time

Job function

• Information Technology