Azure Data Engineer at Programmers.io Cary, NC

Programmers.io; Cary (NC); On-site; USD 110,000 - 150,000; Full time

Job summary

Programmers.io in Cary, NC is seeking an Azure Data Engineer to collect, store, process and analyze large datasets, building ETL pipelines with Spark, Hive, and cloud services. You will design scalable data ingestion and curation processes to support analytics across business units, ensuring reliability and performance.

Collaborating with global teams, you will contribute to data warehousing, real-time and batch use cases, and optimize data workflows using Azure Databricks, Data Factory, Data

Qualifications

• Experience designing and building ETL pipelines for large datasets.
• Strong knowledge of Spark, Hive, and SQL for analytics.
• Hands-on experience with Azure data services and cloud-based data platforms.

Responsibilities

• Design and implement ETL processes and data warehouses for analytics.
• Develop scalable data ingestion and curation pipelines using Spark, Hive and cloud tools.
• Collaborate with global teams to optimize performance of data workflows and analytics projects.

Skills

Strong SQL

Data modeling

Analytical thinking

Tools

BigQuery

Cloud Composer

Dataproc

Cloud Storage

Azure Databricks

Azure Data Factory

Azure Data Lake

Synapse

Spark

PySpark

Kafka

Hive

HDFS

Sqoop

Kerberos

Impala

Airflow

Job description

Azure Data Engineer job at Programmers.io. Cary, NC.

Role Description

Collect, store, process and analyze large datasets to build and implement extract, transfer, load (ETL) processes. Develop reusable frameworks to reduce the development effort involved thereby ensuring cost savings for the projects. Develop quality code with thought through performance optimizations in place right at the development stage. Appetite to learn new technologies and be ready to work on new cutting‑edge cloud technologies. Work with team spread across the globe in driving the delivery of projects and recommend development and performance improvements. Building and Implementing data ingestion and curation process developed using Big data tools such as Spark (Scala, python, Java), Hive, HDFS, Sqoop, Kafka, Kerberos, Impala etc. and CDP 7.x. Ingesting huge volumes data from various platforms for Analytics needs and writing high‑performance, reliable, and maintainable ETL code. StrongStrong analytic skills related to working with unstructured datasets. Strong experience in building designing Data warehouses, data stores for analytics consumption On prem and Cloud (real time as well as batch use cases). Proficiency and extensive Experience with Spark Scala, Python and performance tuning is a MUST. Hive database management and Performance tuning is a MUST. (Partitioning Bucketing). Strong SQL knowledge and data analysis skills for data anomaly detection and data quality assurance.

Essential Skills

Digital Microsoft Azure

Top 3 Required Skills

1. Should have strong experience in BigQuery, Big SQL, Cloud Composer, Dataproc, Cloud Storage
2. Should have strong hands on experience in Azure platform using Azure Databricks, Azure Data Factory (ADF), Azure Data Lake and Synapse
3. Should be familiar about relational data modeling, and implementing Star/Snowflake schemas using build analytical data warehouses concepts

Top 3 Preferred Skills

1. BigQuery, Databricks, Azure Synapse and Snowflake.
2. Pyspark / Spark, kafka
3. Python, Scala, SQL, MySQL.