Data Platform Lead: Azure Data Factory & Databricks

HCL Technologies Limited; United States; Remote; USD 120,000 - 160,000; Full time

Job summary

Sapsf is seeking a skilled Data Engineer to design and optimize scalable data pipelines, architectures, and datasets in the United States. You will ensure data availability and enable data-driven decision making while implementing robust ETL processes and governance.

The role emphasizes collaboration with data scientists and analysts, building data warehouses and lakes, and optimizing performance across large datasets with modern tooling.

Qualifications

• Bachelor’s degree in computer science, engineering, information technology, or a related field.
• 5+ years of experience as a Data Engineer or in a similar role.
• Proficiency in Snowflake SQL and experience with relational databases (e.g., MySQL, PostgreSQL).
• Knowledge of data warehousing solutions- BigQuery.
• Good to have hands on programming languages like Python.
• Strong understanding of Matillion ETL tool and processes.
• Experience with data modeling and data architecture design.

Responsibilities

• Data Pipeline Development: Design, construct, test, and maintain highly scalable data management systems.
• Data Integration: Integrate structured and unstructured data from multiple data sources into a unified data system.
• Data Warehousing: Build and maintain data warehouses and data lakes for large data volumes.
• Data Processing: Implement data processing frameworks e.g., Hadoop, Spark for real-time or batch processing.
• Automation and Monitoring: Automate manual processes, optimize data delivery, and monitor data integrity.
• Collaboration: Work with data scientists, analysts, and stakeholders to meet data needs.
• Data Governance: Ensure data security, privacy, and regulatory compliance.
• Performance Tuning: Optimize ETL processes and data pipelines to reduce processing times.

Skills

Data Pipeline Development

Data Integration

Data Warehousing

Data Processing

Automation and Monitoring

Collaboration

Data Governance

Performance Tuning

Education

Bachelor's degree

Tools

Snowflake SQL

MySQL/PostgreSQL

BigQuery

Python

Matillion ETL

Job description

Sapsf is seeking a skilled Data Engineer to design and optimize scalable data pipelines, architectures, and datasets in the United States. You will ensure data availability and enable data-driven decision making while implementing robust ETL processes and governance.

The role emphasizes collaboration with data scientists and analysts, building data warehouses and lakes, and optimizing performance across large datasets with modern tooling.