Azure Data Engineer

TechDigital Group; Jackson (MI); On-site; USD 90,000 - 140,000; Full time

Job summary

TechDigital Group in Michigan is seeking a hands-on Data Engineer proficient with Azure Synapse Analytics, Data Factory, and Databricks to design and operate robust data pipelines. You will work with PySpark, Pandas, NumPy, and SQL to transform, integrate, and load data from APIs and various sources.

The role involves deploying code through CI/CD, collaborating with BI tools like Power BI, and supporting data quality across the landscape. On-site position in Michigan with growth opportunities.

Qualifications

• Hands-on experience with Azure Synapse Analytics, Azure Data Factory and Databricks.
• Proficient in Python libraries: PySpark, Pandas, NumPy.
• Experience with APIs and data retrieval/processing.
• Hands-on deployment via GitHub and CI/CD pipelines for data projects.
• Strong skills in SQL, stored procedures and data orchestration.

Skills

Data integration

SQL

Python

CI/CD

API integration

Power BI

Tools

Azure Synapse Analytics

Azure Data Factory

Azure Databricks

Azure Storage

Azure Key Vault

SQL Pool CI/CD

Functions

Logic Apps

Linked services

Runtimes

Datasets

Pipelines

Activities

Notebooks

PySpark

Pandas

NumPy

MySQL

Oracle

Job description

• Hands on Experience in Azure Synapse Analytics, Azure Data Factory and Data Bricks, Azure Storage, Azure Key Vault, SQL Pools CI/CD Pipeline Designing and other Azure services like functions, logic apps - Linked services, Various Runtimes, Datasets, Pipelines, Activities
• Strong Hands-on Experience in Various Activities like Control flow logic and conditions (For Each, if, switch, until), Lookup, Stored procedure, scripts, validations, Copy Data, Data flow, Azure functions, Notebooks, SQL Pool Stored procedures etc.
• Strong hands on exp in deployment of code throughout landscape (Dev -> QA -> Prod), Git Hub, CI/CD pipelines etc.
• Strong hands on creating the SQL stored procedures - Functions, Stored Procedures, how to call one SP into another, how to process record-by-record - Dynamic SQL - Data Bricks - Must have
• Strong background about the Python libraries like PySpark, Pandas, NumPy, MySQL, Oracle, PySpark libraries
• Must have strong hands on to get data through APIs
• Must be able to install libraries and help users to troubleshoot issues
• Must have knowledge to get the data through stored procedures via Python
• Should be able to debug the Python code Hands on experience in Spark Pools, Spark
• Should be able to merge data/delta loads through Notebooks
• Must have strong background about the Python libraries and PySpark - Hands on Exp on Power BI