Azure Data Engineer: Pipelines & ML Insights

Kforce Inc; New York (NY); On-site; USD 120,000 - 180,000; Full time

Job summary

Kforce Inc is hiring for a data engineering role in New York. The candidate will work with architects and data teams to identify data sources, design scalable data pipelines, and develop ML-powered insights across cloud and on-premise environments.

Required experience includes 2-5 years in data app development, 2+ years in ETL/ELT on columnar MPP databases, and strong Python/PySpark skills. Familiarity with streaming and distributed data ecosystems is essential.

Qualifications

• 2-5 years of progressive data application development experience.
• 2+ years of experience modeling and implementing ETL/ELT on columnar MPP database technologies
• Strong Experience with Python and/or PySpark needed
• Experience working in large scale/distributed SQL, NoSQL, and/or Hadoop environments
• Experience with streaming architectures e.g. Kafka, Stream, PubSub

Responsibilities

• Identify, engage, and integrate data sources for discovery and profiling and, where necessary and define data services that empower business processes
• Design, build, and scale data pipelines across a variety of source systems and streams (internal, third-party, as well as cloud-based), distributed/elastic environments, and downstream applications and/or self-service solutions
• Develop solutions combining data blending, profiling, mining, statistical analysis, and machine learning, to better define and curate models, test hypothesis, and deliver key insights
• Collaborate in establishing and evolving development, testing, and documentation standards, as well as related code reviews
• Partner with business analysts, application engineers, data scientists, leveraging the appropriate tools, solutions, and/or processes as part of their data mining, profiling, blending, and analytical activities
• Collaborate in establishing and evolving development, testing, and documentation standards, as well as related code reviews

Skills

Python

PySpark

Kafka

NoSQL

ETL/ELT

Distributed SQL

Hadoop

Streaming

Tools

Hadoop ecosystem

Spark

Job description

Kforce Inc is hiring for a data engineering role in New York. The candidate will work with architects and data teams to identify data sources, design scalable data pipelines, and develop ML-powered insights across cloud and on-premise environments.

Required experience includes 2-5 years in data app development, 2+ years in ETL/ELT on columnar MPP databases, and strong Python/PySpark skills. Familiarity with streaming and distributed data ecosystems is essential.