Data Engineer: Azure Spark & Streaming Pipelines

Software Technology Inc; Brentsville (KY); On-site; USD 80,000 - 120,000; Full time

Job summary

Software Technology Inc is seeking a Data Engineer to work with data pipelines in Azure Databricks and Apache Spark. The ideal candidate will have programming skills in Scala, Python, or Java and experience in developing streaming and batch data solutions.

The position includes remote work until COVID-19 restrictions are lifted. Familiarity with tools such as Power BI, Azure Data Factory, and NoSQL databases is a plus. Strong communication skills are essential.

Qualifications

• Experience working in Azure Databricks, Apache Spark.
• Proficient programming in Scala, Python, or Java.
• Experience developing and deploying data pipelines for streaming and batch data sources from multiple sources.
• Experience creating data models and implementing business logic using listed tools and languages.
• Working knowledge in Kafka, Structured Streaming, DataFrame API, SQL, NoSQL Database.
• Comfortable with API, Azure Data Lake, Git, Notebooks, Spark Cluster, Spark Jobs, Performance tuning.
• Telecom domain experience not required but helpful.

Skills

Azure Databricks

Apache Spark

Scala

Python

Java

Data pipelines

Data models

SQL

NoSQL Database

API

Azure Data Lake

Git

Power BI

Delta Lake

Lambda Architecture

Azure Data Factory

Azure Synapse

Telecom domain

Excellent communication skills

Job description

Software Technology Inc is seeking a Data Engineer to work with data pipelines in Azure Databricks and Apache Spark. The ideal candidate will have programming skills in Scala, Python, or Java and experience in developing streaming and batch data solutions.

The position includes remote work until COVID-19 restrictions are lifted. Familiarity with tools such as Power BI, Azure Data Factory, and NoSQL databases is a plus. Strong communication skills are essential.