Senior Data Lead - Azure Data Factory (ADF), Databricks

HCLTech; Chicago (IL); Vor Ort; USD 110.000 - 170.000; Vollzeit

Zusammenfassung

HCL Technologies Limited seeks experienced Data Engineers to lead the migration of large-scale analytical workloads from Amazon Redshift to Databricks Lakehouse Platform. The role focuses on data warehouse modernization, ETL/ELT migration, performance optimization, and cloud-native architectures.

You will design and implement scalable data solutions using Databricks, Delta Lake, and Spark, while converting SQL-based ETL/ELT processes into notebooks and pipelines, and ensuring data quality and

Qualifikationen

• Experience with large-scale data warehousing and data lake architectures.
• Hands-on experience with Redshift and Databricks.
• Expertise in PySpark, Spark SQL, and Python.
• Strong SQL development and query optimization skills.
• Experience with data modeling and ETL/ELT migrations.

Aufgaben

• Lead migration of data pipelines, models, and reporting workloads from Redshift to Databricks.
• Assess existing Redshift environments, schemas, workloads, and dependencies.
• Design and implement scalable data solutions using Databricks, Delta Lake, and Spark.

Kenntnisse

Data warehousing

ETL/ELT migration

SQL optimization

Big data processing

Cloud-native architectures

Spark/PySpark

Tools

Amazon Redshift

Databricks

PySpark

Spark SQL

Python

Delta Lake

Jobbeschreibung

We are seeking experienced Data Engineers to lead and support the migration of large-scale analytical and reporting workloads from Amazon Redshift to Databricks Lakehouse Platform. The ideal candidate will have hands-on experience in data warehouse modernization, ETL/ELT migration, performance optimization, cloud-native architectures, and big data processing.

Key Responsibilities

Lead the migration of data pipelines, data models, and reporting workloads from Redshift to Databricks.Perform assessment and discovery of existing Redshift environments, schemas, workloads, and dependencies.Design and implement scalable data solutions using Databricks, Delta Lake, and Spark.Convert SQL-based ETL/ELT processes into Databricks notebooks, workflows, and pipelines.Optimize data ingestion, transformation, and storage for performance and cost efficiency.Develop batch and streaming data pipelines using PySpark and Spark SQL.Implement data quality, reconciliation, and validation frameworks during migration.Collaborate with business stakeholders, architects, and analytics teams to ensure successful migration and adoption.Troubleshoot performance bottlenecks and tune Spark jobs, Delta tables, and SQL workloads.Establish CI/CD processes and infrastructure-as-code practices for Databricks environments.Document migration strategies, architecture, and operational procedures.

Skill Requirements

Strong experience with large-scale data warehousing and data lake architectures.Hands-on experience with Amazon Redshift and Databricks.Expertise in PySpark, Spark SQL, and Python.Strong SQL development and query optimization skills.Experience with data modeling (Star Schema, Snowflake Schema, Dimensional Modeling).

Other Requirements

Migration ExperiencePrior experience migrating from:Redshift to DatabricksTeradata/Snowflake/Hadoop to Databricks (preferred)Data validation, reconciliation, and cutover planning.Performance tuning and workload modernization.