Senior Data Engineer (Databricks)

Parser; Colombia; Remote; COP 90,000,000 - 140,000,000; Full time

Benefits offered by this job

Remote-friendly

Flexible schedule

Job summary

Parser in Colombia is seeking a senior data engineer to design, develop, and optimize data pipelines using Databricks and Apache Spark, building scalable ETL/ELT processes from banking systems.

You will implement Delta Lake data lakehouse architecture, collaborate with data scientists and stakeholders, and mentor junior engineers while upholding security and regulatory requirements such as PCI-DSS, SOX, and GDPR.

Qualifications

• 5+ years of experience in data engineering or related roles.
• 3+ years hands-on experience with Databricks platform.
• Strong proficiency in Apache Spark (PySpark/Scala).
• Expert-level SQL skills and experience with large-scale data processing.
• Experience with Delta Lake and data lakehouse architectures.
• Solid understanding of cloud platforms (AWS, Azure, or GCP).
• Experience with banking or financial services data and regulatory requirements.
• Knowledge of data modeling, data warehousing concepts, and dimensional modeling.
• Strong problem-solving skills and attention to detail.
• Advance level english

Responsibilities

• Design, develop, and optimize data pipelines using Databricks and Apache Spark
• Build scalable ETL/ELT processes to ingest, transform, and deliver data from various banking systems
• Implement data lakehouse architecture using Delta Lake for reliable data storage and processing
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements
• Optimize Databricks cluster configurations and query performance for cost efficiency
• Develop and maintain data quality frameworks and monitoring solutions
• Implement security best practices and ensure compliance with banking regulations (PCI-DSS, SOX, GDPR)
• Create and maintain technical documentation for data workflows and processes
• Mentor junior engineers and promote best practices in data engineering
• Participate in code reviews and contribute to continuous improvement initiatives

Skills

Advanced English

Problem solving

Mentoring

Collaboration

Data quality

Tools

Databricks

Apache Spark

Delta Lake

Terraform

CloudFormation

Pandas

NumPy

Job description

This position offers you the opportunity to join a fast-growing technology organization that is redefining productivity paradigms in the software engineering industry. If you are passionate about technology and want to learn about new trends that are shaping human behavior, this opportunity is for you.

The impact you’ll make:

• Design, develop, and optimize data pipelines using Databricks and Apache Spark
• Build scalable ETL/ELT processes to ingest, transform, and deliver data from various banking systems
• Implement data lakehouse architecture using Delta Lake for reliable data storage and processing
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements
• Optimize Databricks cluster configurations and query performance for cost efficiency
• Develop and maintain data quality frameworks and monitoring solutions
• Implement security best practices and ensure compliance with banking regulations (PCI-DSS, SOX, GDPR)
• Create and maintain technical documentation for data workflows and processes
• Mentor junior engineers and promote best practices in data engineering
• Participate in code reviews and contribute to continuous improvement initiatives

Preferred Skills:

• Databricks certification (Data Engineer Associate or Professional)
• Experience with real-time streaming data processing (Kafka, Event Hubs)
• Knowledge of MLOps and ML model deployment on Databricks
• Experience with Infrastructure as Code (Terraform, CloudFormation)
• Familiarity with CI/CD pipelines and DevOps practices
• Experience with data governance tools and frameworks
• Knowledge of Python libraries for data processing (Pandas, NumPy)
• Understanding of banking products, risk management, or fraud detection systems

Required Qualifications

• 5+ years of experience in data engineering or related roles
• 3+ years of hands-on experience with Databricks platform
• Strong proficiency in Apache Spark (PySpark/Scala)
• Expert-level SQL skills and experience with large-scale data processing
• Experience with Delta Lake and data lakehouse architectures
• Solid understanding of cloud platforms (AWS, Azure, or GCP)
• Experience with banking or financial services data and regulatory requirements
• Knowledge of data modeling, data warehousing concepts, and dimensional modeling
• Strong problem-solving skills and attention to detail
• Advance level english

Some of the benefits you’ll enjoy working with us:

• The chance to work in innovative projects with leading brands, that use the latest technologies that fuel transformation.
• The opportunity to be part of an amazing, multicultural community of tech experts.
• A flexible and remote working environment.