Data Engineer

Light-it Inc.; Buenos Aires; On-site; ARS 4,200,000 - 6,000,000; Full time

Job summary

Light-it Inc. is seeking a Data Engineer to design and operate production data pipelines using Databricks, Delta Lake, and dbt. You will build and monitor orchestration, manage Unity Catalog governance, and collaborate with cross-functional teams to ensure data quality and regulatory compliance.

The role requires strong SQL/PySpark skills, experience with cloud dataLake ecosystems, and a proactive approach to performance tuning and data lineage. Remote work is not specified.

Qualifications

• Proven experience building production data pipelines.
• Hands-on Databricks experience in notebooks, repos, clusters and SQL warehouses.
• Strong Python and PySpark for ingestion and transformation.
• Delta Lake with bronze/silver/gold architecture and MERGE/upserts.
• Databricks Workflows/Lakeflow for orchestration, scheduling, monitoring.
• Unity Catalog governance, lineage and access control.
• dbt-based datamodeling on Databricks.
• Strong SQL skills (Spark SQL) and PostgreSQL experience.
• Cloud familiarity (AWS S3/IAM; Azure or GCP Databricks).
• Knowledge of data ingestion patterns and data lineage maintenance.
• Experience in regulated/compliance environments and cross-functional teams.

Responsibilities

• Develop, test, and maintain PySpark, SparkSQL and dbt models for data transformation across bronze/silver/gold layers.
• Build, monitor, and troubleshoot Databricks Workflows and Lakeflow Jobs for scheduled and event-driven ingestion.
• Maintain ingestion pipelines from source systems including file-based and system-to-system patterns using Structured Streaming.
• Optimize Delta Lake tables and gold-layer marts for production applications with partitioning and OPTIMIZE/VACUUM.
• Right-size and tune job clusters, serverless compute, and SQL warehouses balancing performance and cost.
• Manage Unity Catalog catalogs, schemas, permissions, and data lineage with least-privilege access.
• Validate data accuracy, completeness, and traceability across the pipeline.
• Document architecture, lineage, and model logic; ensure traceability.
• Track work using project tools (Clickup/JIRA); follow CI/CD and asset deployment standards.
• Support incident response and root-cause analysis for data pipeline issues.
• Participate in knowledge transfer, pairing, and onboarding for new projects.

Skills

Data pipelines

English proficiency

Agile delivery

Stakeholder comms

Governance awareness

Tools

Databricks

Delta Lake

dbt

Unity Catalog

Spark/PySpark

Airflow

Terraform

S3

PostgreSQL

Snowflake

Genie/MLflow

Job description

• Proven experience as a Data Engineer or similar role, building and operating production data pipelines
• Strong hands-on production experience with Databricks, including notebooks and repos (Git folders), job clusters, and SQL warehouses
• Strong hands-on experience with Python and PySpark for data ingestion and transformation, and with Apache Spark fundamentals (partitioning, shuffles, joins, performance tuning)
• Production experience with Delta Lake andthe medallion (bronze/silver/gold)architecture, including MERGE/upserts, timetravel, and schema enforcement
• Production experience with DatabricksWorkflows / Lakeflow Jobs for pipelineorchestration, including task design,scheduling, monitoring, and troubleshooting
• Experience with Unity Catalog forgovernance, lineage, and access control
• Production experience with dbt for datamodeling and transformation, ideally onDatabricks
• Strong SQL skills (Spark SQL / Databricks SQL)and experience with Postgres or similarrelational databases
• Familiarity with the underlying cloud (AWSpreferred: S3, IAM; Azure or GCP Databricksexperience is also welcome)
• Understanding of data ingestion patterns,including file-based, streaming, andAPI/system-to-system integration
• Experience maintaining data lineage,traceability, and auditability across a pipeline
• Comfort working within a regulated orcompliance-driven environment (e.g., SOC-2,HIPAA, or similar)
• Experience working in a sprint-based agiledelivery model, coordinating with cross-functional engineering teams
• Strong documentation and communicationskills; able to work effectively with bothtechnical and non-technical stakeholders
• Many of our projects involve business-critical data governed by regulatory requirements, so comfort operating in a regulated environment matters as much as pipeline engineering skill
• Advanced English

We are looking for someone able to

• Develop, test, and maintain PySpark, SparkSQL, and dbt (dbt-databricks) models fordata transformation and business logicacross bronze, silver, and gold layers
• Build, monitor, and troubleshoot DatabricksWorkflows / Lakeflow Jobs and LakeflowDeclarative Pipelines supporting scheduledand event-driven data ingestion, includingdata quality expectations
• Build and maintain ingestion pipelinespulling data from source systems, includingfile-based (Auto Loader over Amazon S3) andsystem-to-system integration patterns,using Structured Streaming where near-real-time delivery is needed
• Maintain and optimize Delta Lake tablesand gold-layer marts that feed productionapplications (e.g., partitioning/liquidclustering, OPTIMIZE and VACUUM, schemaevolution)
• Right-size and tune job clusters, serverlesscompute, and SQL warehouses to balanceperformance and cost, monitoring usagethrough system tables
• Manage catalogs, schemas, permissions,and lineage in Unity Catalog, applying least-privilege access and controls for sensitive(PHI/PII) data
• Validate data accuracy, completeness, andtraceability across the ingestion-to-application data flow
• Document pipeline architecture, datalineage, and model logic
• Track and manage pipeline work using theproject’s tooling (e.g., Clickup and JIRA);contribute code following the team’s CI/CDand branching standards (e.g., Github orGitLab), deploying with Databricks AssetBundles
• Support incident response and root-causeanalysis for data pipeline issues affectingproduction
• Participate in knowledge transferactivities, including documentation review,pairing, and onboarding support, as projectsrequire

Other valuable skills

• Databricks certifications: Data EngineerAssociate and/or Data Engineer Professional
• Experience with Databricks Asset Bundlesand CI/CD for Databricks (e.g., GitLab CI/CDor GitHub Actions), and Terraform (Databricksprovider) for workspace and Unity Cataloginfrastructure-as-code
• Experience with Apache Airflow or otherexternal orchestrators triggering Databricksjobs
• Exposure to Snowflake or other cloud datawarehouses (e.g., Amazon Redshift)
• Familiarity with Databricks AI/BI dashboardsand Genie, or MLflow
• Experience with AWS Secrets Manager /Databricks secret scopes and AWSCloudWatch or similar monitoring
• Experience with drug supply chain, pharmacylicensing, or similar healthcare regulatoryframeworks
• Experience with state-level or federalregulatory and compliance systems
• Prior work with not-for-profit orgovernmental organizations
• Familiarity with C#

By applying for this position, you acknowledge that Light-it will process your information in accordance with our Privacy Policy.