Databricks Data Developer: Build Scalable Data Pipelines

Ciandt; Colombia; On-site; COP 60,000,000 - 110,000,000; Full time

Benefits offered by this job

Premium Healthcare

Meal voucher

Maternity and Parental leaves

Mobile services subsidy

Sick pay-Life insurance

CI&T University

Colombian Holidays

Paid Vacations

Job summary

CI&T is hiring a Data Developer in Colombia to design, develop, and maintain scalable data infrastructure and pipelines for analytics and business intelligence. You will implement modern data stack patterns, work with Databricks, AWS services, and cloud-native tools, translating business requirements into robust data solutions.

You will collaborate with Data & Analytics teams, optimize pipelines for performance and cost, ensure data quality and governance, and contribute to AI-enabled data

Qualifications

• Hands-on Databricks migration with the databricks/migrate tool (notebooks, jobs, clusters, secrets, ACLs).
• Notebook and job migration: notebooks, DAG pipelines, DLT pipelines.
• Storage path refactoring: /mnt/ DBFS references to S3-backed locations.
• Data migration: Delta Lake to Iceberg on S3 (Delta tables, raw files).
• Streaming: Auto Loader / DLT checkpoint re-bootstrap, event_id dedup.
• Validation: row counts, checksums, sample queries, ETL runtime comparison.
• Secrets: recreate secret scopes in AWS Secrets Manager.
• AWS services: S3, Glue, Secrets Manager, IAM basics, Athena, LakeFormation and EMR.
• Languages: Spark SQL, PySpark, Java, Python and Scala.
• Agile pod delivery: daily ticket updates, standups, code review.

Responsibilities

• Data Pipeline Development: Build, optimize, and maintain scalable data pipelines and ETL/ELT processes to support analytics and business intelligence needs.
• Data Architecture Implementation: implement data architectures, including data lakes, warehouses, and integration layers that support the organization's data strategy and modernization.
• Data Quality & Governance: Implement data quality checks, validation processes, and monitoring systems to ensure data accuracy, consistency, and reliability.
• Integration & API Development: Develop and maintain integrations between various data sources, third-party tools, and analytics platforms.
• Collaboration: Work closely with Data & Analytics Managers and business stakeholders to understand requirements and deliver technical solutions that enable data-driven decision making.
• Performance Optimization: Monitor and optimize data pipeline performance, query efficiency, and storage costs to ensure scalable and cost-effective solutions.
• Documentation: Create and maintain comprehensive technical documentation for data architectures, pipelines, and processes.
• Data Security: Implement security best practices and ensure compliance with data privacy regulations across all data systems.
• Powered by AI: Leverage AI tools and automation to enhance data engineering workflows and productivity.

Skills

Data engineering

Cloud platforms

SQL

PySpark

Java

Python

Scala

ETL/ELT

Agile delivery

Tools

Databricks migration tool

Unity Catalog to Glue Catalog

AWS Secrets Manager

S3

Glue

Athena

Lake Formation

EMR

Job description

CI&T is hiring a Data Developer in Colombia to design, develop, and maintain scalable data infrastructure and pipelines for analytics and business intelligence. You will implement modern data stack patterns, work with Databricks, AWS services, and cloud-native tools, translating business requirements into robust data solutions.

You will collaborate with Data & Analytics teams, optimize pipelines for performance and cost, ensure data quality and governance, and contribute to AI-enabled data