ADF with Data Bricks, Snowflake Lead

Infinity Outsourcing; Oregon (WI); Hybrid; USD 108,000 - 132,000; Full time

Job summary

A leading data solutions company seeks an experienced ADF with Data Bricks, Snowflake Lead. In this remote position, you will design and maintain complex data processing pipelines and support enterprise data governance. The ideal candidate has over 10 years of experience, strong SQL skills, and expertise in data ingestion processes. You will collaborate with various teams to ensure data quality and compliance with regulations. Join us to leverage your skills in an innovative environment.

Qualifications

• 10+ years of experience in data engineering roles.
• Strong experience with cloud data processing and architecture.
• Ability to write complex SQL queries and optimize performance.

Responsibilities

• Design and build data processing pipelines using Databricks and Azure.
• Provide operational support for enterprise data management.
• Implement data quality standards and security measures.

Skills

Data Ingestion Pipeline Design

Databricks

Azure Data Factory

SQL Optimization

Data Quality Checks

Data Governance

Tools

Snowflake

Snow-Pipe

Snow-SQL

Job description

About the job ADF with Data Bricks, Snowflake Lead

Position: ADF with Data Bricks, Snowflake Lead

Salary: $120,000 around with QPLC

Location : Remote

10+Years of Exp Need

Responsibilities

• Designing and building the data Ingestion Pipeline
• Designing, building and maintain large, complex data processing pipelines using Databricks and Azure Data Factory in Azure to meet enterprise Data requirements
• Providing operational and functional support on creating, storing, managing, and maintaining enterprise data, including the ability to incorporate policies and procedures for centrally managing and sharing data through the data life cycle
• Identifying, designing, and implement internal process improvements: automating manual processes, optimizing data delivery, and re-designing for flexibility and scalability
• Building the infrastructure required for extraction, transformation, and loading of data from a wide variety of cloud and on-premises data sources.
• Working with stakeholders including to assist with data-related technical issues and support their data infrastructure needs.
• Working closely with data architecture, data governance and data analytics teams to ensure pipelines adhere to enterprise standards, usability, and performance.
• Experience in Snow-Pipe and Snow-SQL
• Write complex SQL queries, stored procedures, and user-defined functions in Snowflake to support data processing and analytics requirements.
• Optimize SQL code and query performance by understanding query execution plans, indexing strategies, and query optimization techniques.
• Implement query performance tuning and indexing strategies to improve overall system performance.
• Data Quality and Governance:
• Implement data quality checks and validation processes to ensure the accuracy, consistency, and integrity of data in Snowflake.
• Collaborate with data governance teams to enforce data quality standards and compliance with data regulations.
• Implement and maintain data security measures in Snowflake to ensure data privacy and compliance.
• Document data models, ETL processes, data transformation rules, and best practices for future reference and knowledge sharing.
• Participate in code reviews, quality assurance, and documentation reviews.