Data Engineer

TechDigital Group; Redmond (WA); On-site; USD 90,000 - 140,000; Full time

Job summary

An established industry player is seeking a seasoned data engineer with extensive experience in Microsoft technologies, particularly Pyspark and KUSTO. This role involves working with large datasets, developing solutions in Azure Synapse, and guiding team members to enhance their technical skills. The ideal candidate will have a deep understanding of data warehousing, data management, and the Microsoft ecosystem, ensuring best practices are followed. Join a dynamic team where your expertise will significantly impact project success and efficiency.

Qualifications

• 10+ years of Microsoft experience with 3-4 years in Pyspark.
• Experience handling large datasets and debugging notebooks.

Responsibilities

• Provide technical guidance and advocate best practices for the team.
• Develop team members' technical capabilities and ensure compliance.

Skills

KUSTO

Pyspark

Python

Azure Synapse

SQL

Data Warehousing

Data Management

Data Modeling

Debugging

Data Engineering

Tools

Azure Data Lake

Azure DevOps

Azure WebApps

Azure Functions

Azure Cosmos DB

Event Hub

Job description

Location: someone from Redmond, WA is an ideal candidate. Next preference is within Pacific timezone.

Mandatory skills: Working experience on KUSTO.

Job Description:

10 years of Microsoft experience is a MUST with the last 3-4 years in Pyspark. Working experience on KUSTO for telemetry data is required. Candidates should be able to write Python code in Azure Synapse Notebook and possess a good understanding of Azure Synapse end-to-end. Experience with Spark structured streaming data, possibly from Event Hubs, is essential. Candidates should have experience handling large volumes of data, possibly in the 100s of GB. They must be able to debug and fine-tune notebooks for optimum memory consumption and processing. Knowledge of data warehousing and data management for efficient data handling is necessary. A good understanding of data modeling and data engineering work is important. Experience working with large datasets using SQL, Azure Data Lake, PySpark, ADF, Synapse, etc., to derive actionable insights is highly desirable. Candidates should be productive as soon as possible after joining.

Nice to Have:
Local Redmond resource, PST working hour resource, Microsoft ecosystem knowledge, Azure DevOps, Azure WebApps, Azure Functions, Azure Cosmos DB, Event Hub.

1. To be responsible for providing technical guidance or solutions; define, advocate, and implement best practices and coding standards for the team.
2. To develop and guide the team members in enhancing their technical capabilities and increasing productivity.
3. To ensure process compliance in the assigned module, and participate in technical discussions or reviews as a technical consultant for feasibility study (technical alternatives, best packages, supporting architecture best practices, technical risks, breakdown into components, estimations).
4. To prepare and submit status reports for minimizing exposure and risks on the project or closure of escalations.