Solution Data Engineer Sr.

VALCE T@lent Solutions; United States; Remote; USD 120.000 - 180.000; Vollzeit

Zusammenfassung

VALCE T@lent Solutions is seeking an experienced Databricks and data engineering specialist to manage complex analytics environments across cloud platforms. The role emphasizes governance, workspace administration, and efficient data processing with PySpark.

You will implement Unity Catalog with Delta Lake, maintain pipelines with Airflow, and collaborate via Git. The position also involves supporting Snowflake, BigQuery, Athena, and AWS Glue, plus handling Adobe Analytics data ingestion.

Qualifikationen

• Strong experience with Databricks ecosystem administration.
• Experience with Unity Catalog and Delta Lake.
• Proficiency in Python, SQL, and PySpark.
• Git-based collaborative development.
• Snowflake data warehousing experience.
• Adobe Analytics and data feed ingestion experience.
• Airflow orchestration experience.
• Familiarity with BigQuery, Athena, and AWS Glue.

Aufgaben

• Manage the Databricks environment, workspace administration, and configuration.
• Implement governance with Unity Catalog and manage Delta Lake storage.
• Build and optimize data processing logic with Python, SQL, and PySpark.
• Manage code lifecycle and collaboration using Git.
• Support data architecture, performance, and integrations in Snowflake.
• Handle Adobe Data Feeds and Adobe Analytics ingestion and transformation.
• Maintain and monitor pipelines across cloud platforms, including Airflow pipelines.
• Support data availability and basic operations in BigQuery, Athena, and AWS Glue.

Kenntnisse

Advanced English

Python

SQL

PySpark

Git

Databricks administration

Tools

Databricks

Unity Catalog

Delta Lake

Snowflake

Airflow

BigQuery

Athena

AWS Glue

Jobbeschreibung

Responsibilities

• Manage the Databricks environment, workspace administration, and configuration
• Implement governance with Unity Catalog and manage Delta Lake storage
• Build and optimize data processing logic with Python, SQL, and PySpark
• Manage code lifecycle and collaboration using Git
• Support data architecture, performance, and integrations in Snowflake
• Handle Adobe Data Feeds and Adobe Analytics ingestion and transformation
• Maintain and monitor pipelines across cloud platforms, including Airflow workflows
• Support data availability and basic operations in BigQuery, Athena, and AWS Glue

Requirements

• Advanced English
• Strong experience with Databricks ecosystem administration
• Experience with Unity Catalog and Delta Lake
• Proficiency in Python, SQL, and PySpark
• Git-based collaborative development
• Snowflake data warehousing experience
• Adobe Analytics and data feed ingestion experience
• Airflow orchestration experience
• Familiarity with BigQuery, Athena, and AWS Glue