Databricks Data Engineer — Delta Lake & Pipelines Lead

Light-it Inc.; Buenos Aires; Sur place; ARS 4 200 000 - 6 000 000; Plein temps

Résumé du poste

Light-it Inc. is seeking a Data Engineer to design and operate production data pipelines using Databricks, Delta Lake, and dbt. You will build and monitor orchestration, manage Unity Catalog governance, and collaborate with cross-functional teams to ensure data quality and regulatory compliance.

The role requires strong SQL/PySpark skills, experience with cloud dataLake ecosystems, and a proactive approach to performance tuning and data lineage. Remote work is not specified.

Qualifications

• Proven experience building production data pipelines.
• Hands-on Databricks experience in notebooks, repos, clusters and SQL warehouses.
• Strong Python and PySpark for ingestion and transformation.
• Delta Lake with bronze/silver/gold architecture and MERGE/upserts.
• Databricks Workflows/Lakeflow for orchestration, scheduling, monitoring.
• Unity Catalog governance, lineage and access control.
• dbt-based datamodeling on Databricks.
• Strong SQL skills (Spark SQL) and PostgreSQL experience.
• Cloud familiarity (AWS S3/IAM; Azure or GCP Databricks).
• Knowledge of data ingestion patterns and data lineage maintenance.
• Experience in regulated/compliance environments and cross-functional teams.

Responsabilités

• Develop, test, and maintain PySpark, SparkSQL and dbt models for data transformation across bronze/silver/gold layers.
• Build, monitor, and troubleshoot Databricks Workflows and Lakeflow Jobs for scheduled and event-driven ingestion.
• Maintain ingestion pipelines from source systems including file-based and system-to-system patterns using Structured Streaming.
• Optimize Delta Lake tables and gold-layer marts for production applications with partitioning and OPTIMIZE/VACUUM.
• Right-size and tune job clusters, serverless compute, and SQL warehouses balancing performance and cost.
• Manage Unity Catalog catalogs, schemas, permissions, and data lineage with least-privilege access.
• Validate data accuracy, completeness, and traceability across the pipeline.
• Document architecture, lineage, and model logic; ensure traceability.
• Track work using project tools (Clickup/JIRA); follow CI/CD and asset deployment standards.
• Support incident response and root-cause analysis for data pipeline issues.
• Participate in knowledge transfer, pairing, and onboarding for new projects.

Connaissances

Data pipelines

English proficiency

Agile delivery

Stakeholder comms

Governance awareness

Outils

Databricks

Delta Lake

dbt

Unity Catalog

Spark/PySpark

Airflow

Terraform

S3

PostgreSQL

Snowflake

Genie/MLflow

Description du poste

Light-it Inc. is seeking a Data Engineer to design and operate production data pipelines using Databricks, Delta Lake, and dbt. You will build and monitor orchestration, manage Unity Catalog governance, and collaborate with cross-functional teams to ensure data quality and regulatory compliance.

The role requires strong SQL/PySpark skills, experience with cloud dataLake ecosystems, and a proactive approach to performance tuning and data lineage. Remote work is not specified.