Master Data Developer (Databricks), Colombia

CI&T; Colombia; On-site; COP 60,000,000 - 120,000,000; Full time

Benefits offered by this job

Maternity and Parental leaves

Mobile services subsidy

Sick pay-Life insurance

CI&T University

Colombian Holidays

Paid Vacations

Job summary

CI&T in Colombia seeks a Data Developer to design, build, and maintain scalable data infrastructure and pipelines for analytics and business intelligence.

You will implement data architectures, ensure data quality, and collaborate with analytics teams to deliver high-performance data solutions using modern data stack technologies and AI-enabled workflows.

Qualifications

• Hands-on Databricks migration with the databricks/migrate tool, notebooks and jobs.
• Notebook and job migration: notebooks, DAG pipelines, DLT pipelines.
• Storage path refactoring: /mnt/ DBFS references to S3-backed locations.
• Data migration execution: Delta to Iceberg on S3, incremental sync.
• Streaming: Auto Loader / DLT checkpoint re-bootstrap, event_id dedup.
• Secrets: recreate secret scopes in AWS Secrets Manager.
• AWS services: S3, Glue, Secrets Manager, IAM basics, Athena, LakeFormation and EMR.
• Languages: Spark SQL, PySpark, Java, Python and Scala.
• Agile pod delivery: daily ticket updates, standups, code review

Responsibilities

• Build, optimize, and maintain scalable data pipelines for analytics.
• Implement data architectures including data lakes and warehouses.
• Enforce data quality checks and monitoring.
• Develop integrations between data sources and analytics platforms.
• Collaborate with Data & Analytics Managers to translate requirements into solutions.
• Monitor and optimize pipeline performance and storage costs.
• Create and maintain technical documentation for architectures and pipelines.
• Ensure data security and regulatory compliance across data systems.
• Leverage AI tools to enhance data engineering workflows.

Skills

Databricks migration

Notebook migration

Data pipelines

Data architecture

Data quality

Streaming

AWS basics

Spark SQL

PySpark

Java/Scala

Agile delivery

Tools

Databricks

AWS Glue

S3

Delta Lake

Iceberg

DLT

Job description

At CI&T, we help large enterprises transform the potential of AI into real business impact with AI Deployment, AI-native execution, and tech-integrated business solutions.

With 30 years of experience in technological transformation, we accelerate innovation with expertise in Agentic SDLC, Application modernization, Data & AI, Martech and Business strategy.

We are 8,000 CI&Ters across more than 25 countries, collaborating to build solutions with real impact. AI is already part of how we work, evolve, and innovate every day.

As CI&T grows, we seek a talented, motivated, and experienced Data Developer to join our team and build robust data-driven solutions for our customers. This role is pivotal in designing, developing, and maintaining scalable data infrastructure and pipelines that enable our data initiatives to thrive.

The Data Developer will be responsible for implementing and optimizing data architectures, ensuring data quality and reliability, and collaborating closely with analytics and business teams to deliver high-performance data solutions. This position requires strong technical expertise in data engineering, cloud platforms, and modern data stack technologies, combined with the ability to translate business requirements into effective technical solutions.

Responsibilities:

• Data Pipeline Development: Build, optimize, and maintain scalable data pipelines and ETL/ELT processes to support analytics and business intelligence needs
• Data Architecture Implementation: implement data architectures, including data lakes, warehouses, and integration layers that support the organization's data strategy and modernization
• Data Quality & Governance: Implement data quality checks, validation processes, and monitoring systems to ensure data accuracy, consistency, and reliability
• Integration & API Development: Develop and maintain integrations between various data sources, third-party tools, and analytics platforms
• Collaboration: Work closely with Data & Analytics Managers and business stakeholders to understand requirements and deliver technical solutions that enable data-driven decision making
• Performance Optimization: Monitor and optimize data pipeline performance, query efficiency, and storage costs to ensure scalable and cost-effective solutions
• Documentation: Create and maintain comprehensive technical documentation for data architectures, pipelines, and processes
• Data Security: Implement security best practices and ensure compliance with data privacy regulations across all data systems
• Powered by AI: Leverage AI tools and automation to enhance data engineering workflows and productivity

Requirements:

• Hands-on Databricks migration with the databricks/migrate tool (notebooks,
  jobs, clusters, secrets, ACLs), Unity catalog to Glue Catalog
• Notebook and job migration: notebooks, DAG pipelines, DLT pipelines
• Storage path refactoring: /mnt/ DBFS references to S3-backed locations
• Data migration execution: Deltalake format to Iceberg format on S3 (Delta tables, raw files), incremental sync
• Streaming: Auto Loader / DLT checkpoint re-bootstrap, event_id dedup
• Secrets: recreate secret scopes in AWS Secrets Manager
• AWS services: S3, Glue, Secrets Manager, IAM basics, Athena, LakeFormation and
  EMR
• Languages: Spark SQL, PySpark, Java, Python and Scala
• Agile pod delivery: daily ticket updates, standups, code review

Nice to Have:

Databricks Certified Consultant (Associate/Professional); DBRto EMR on AWS migration experience

Our benefits include:

• - Maternity and Parental leaves
• - Mobile services subsidy
• - Sick pay-Life insurance
• - CI&T University
• - Colombian Holidays
• - Paid Vacations

And many others.

Collaboration is our superpower, diversity unites us, and excellence is our standard.

We value diverse identities and life experiences, fostering a diverse, inclusive, and safe work environment. We encourage applications from diverse and underrepresented groups to our job positions.