Sr. Data Engineer

TechDigital Group; New York (NY); On-site; USD 80,000 - 120,000; Full time

Job summary

An established industry player is seeking a skilled data engineer to design and maintain scalable data pipelines using Azure technologies. This role involves integrating data from various sources into the Azure Databricks platform, implementing ETL processes, and ensuring data quality and integrity. The successful candidate will collaborate with data scientists and analysts to facilitate data-driven decision-making and assist with data visualization using Power BI. Join a dynamic team and contribute to impactful projects in a fast-paced environment where your expertise will drive innovation and success.

Qualifications

• Experience in developing scalable data pipelines using Azure technologies.
• Strong skills in data integration and ETL processes.

Responsibilities

• Design and maintain scalable data pipelines using Azure Databricks and Data Lake.
• Collaborate with data scientists to support data-driven decision-making.

Skills

Azure Data Bricks

Azure Data Lake

Python

Pyspark

Bloomberg data acquisition

Tools

Azure Data Factory

Power BI

Job description

Mandatory Skills

• Azure Data Bricks
• Azure Data Lake
• Python
• Pyspark

Desired Skill: Bloomberg data acquisition and transformation

• Design, develop, and maintain scalable data pipelines using Azure Databricks and Azure Data Lake.
• Integrate data from various sources into the Databricks platform.
• Implement data integration and ETL processes using Azure Data Factory.
• Develop and optimize data processing workflows and pipelines using PySpark.
• Support solving business use cases involving Bloomberg data acquisition and transformation.
• Collaborate with data scientists and analysts to support data-driven decision-making.
• Ensure data quality and integrity across various data sources and storage solutions.
• Monitor and troubleshoot data pipeline performance and reliability.
• Assist with dashboarding and data visualization using Power BI.