Data Engineer w/DevOps

Techwarely LLC; Argentina; Remote; ARS 1,800,000 - 3,000,000; Full time

Job summary

Techwarely LLC is seeking a semi senior Data Engineer w/DevOps for a Databricks-focused team. You will migrate legacy platforms to Delta Lake and run production pipelines using Python, SQL and PySpark on Databricks, while owning CI/CD, IaC and platform operations.

The role is fully remote and requires collaboration with engineers and stakeholders. You will design and optimize pipelines, manage Unity Catalog, and implement data quality checks to ensure reliability and cost efficiency.

Qualifications

• Experience designing and operating production data pipelines on Databricks.
• Proven ability to migrate to Delta Lake medallion models.
• Hands-on CI/CD with Jenkins, GitHub Actions, or Azure DevOps.

Responsibilities

• Design and maintain batch and streaming pipelines on Databricks.
• Migrate legacy data to Delta Lake bronze/silver/gold layers.
• Develop and optimize Spark jobs, PySpark code and SQL.
• Implement data quality checks and validation.
• Build and maintain CI/CD pipelines for Databricks code.
• Manage Terraform-based infrastructure and Unity Catalog access controls.
• Set up monitoring and alerts for pipeline reliability.
• Document pipelines and deployment processes; participate in Scrum with Jira.

Skills

Ownership of services end-to-end

Strong communication in English

Team collaboration

Tools

Python

SQL

PySpark

Databricks

Delta Lake

Jenkins

GitHub Actions

Azure DevOps

Terraform

Unity Catalog

Jira

Airflow

Datadog

Kubernetes

Helm

AWS

Azure

GCP

Job description

Stack

Python SQL PySpark Databricks Delta Lake Jenkins CI/CD GitHub Actions Azure DevOps Terraform Git Unity Catalog Jira Apache Airflow Datadog Kubernetes Helm Amazon Web Services (AWS) Microsoft Azure Google Cloud Platform (GCP)

Summary

We are seeking a semi senior Data Engineer w/DevOps to join a Databricks-focused team migrating legacy data platforms to Delta Lake and operating production pipelines. This role combines hands‑on data engineering—building batch and streaming pipelines with Python, SQL and PySpark on Databricks—with DevOps responsibilities for CI/CD, infrastructure as code and platform operations. The position is fully remote and involves close collaboration with engineers, analysts and stakeholders to deliver reliable, scalable data services.

Responsibilities

• Design, build and maintain batch and streaming pipelines on Databricks that ingest data from multiple sources, including APIs.
• Migrate legacy data and workloads into Delta Lake modeled in medallion layers (bronze, silver, gold).
• Develop and optimize Spark jobs, PySpark code and SQL queries for performance and cost-efficiency.
• Implement data quality checks and validation to detect issues before they reach consumers.
• Build and maintain CI/CD pipelines to test and deploy Databricks code, jobs and workflows.
• Provision and manage cloud and Databricks resources using infrastructure as code, primarily Terraform.
• Manage Unity Catalog, access controls, cluster policies and secrets within the Databricks platform.
• Set up monitoring and alerting for pipeline reliability and platform health.
• Automate environment setup and promotion across development, test and production.
• Document pipelines, infrastructure, runbooks and deployment processes and participate in Scrum ceremonies tracked in Jira.

Requirements

• Semi senior level data engineer with responsibility to own services end to end and support less experienced engineers.
• Proven experience designing, building and operating production data pipelines using Python, SQL and PySpark on Databricks.
• Production experience with Databricks and Delta Lake, including jobs, workflows and notebooks.
• Hands‑on CI/CD experience with tools such as Jenkins, GitHub Actions or Azure DevOps and strong Git practices.
• Experience with infrastructure as code, specifically Terraform.
• Experience on at least one major cloud platform (AWS, Azure or Google Cloud Platform).
• Solid understanding of data modeling, data warehousing and ETL/ELT patterns.
• Experience working in Scrum teams and tracking work in Jira.
• Strong communication skills in English, both written and spoken.

Nice to Have

• Experience with data platform migrations into Databricks.
• Unity Catalog administration and data governance experience.
• Orchestration experience with Databricks Workflows or Apache Airflow.
• Familiarity with observability tooling such as Datadog.
• Kubernetes and Helm experience.
• Databricks Data Engineer, Azure Data Engineer Associate or Terraform Associate certifications.