Senior Databricks Data Engineer: PySpark & Lakehouse

Perficient; Colombia; Híbrido; COP 280.260.000 - 373.680.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Competitive benefits

Paid time off & holidays

Billable bonus opportunities

Descripción de la vacante

Perficient is seeking a Data Engineer to design, build, and operate end-to-end data pipelines using PySpark on Databricks. You will implement ETL/ELT pipelines, contribute to lakehouse architectures, and ensure data quality and observability across large-scale systems.

You will collaborate with data architects, analysts, and stakeholders to translate requirements into scalable data solutions, while applying CI/CD and version control practices and leveraging cloud platforms such as Azure or AWS.

Formación

• BS in Computer Science, Engineering, or related technical field, or equivalent practical experience.

Responsabilidades

• Design, build, and maintain end-to-end data pipelines for ingestion, transformation, and delivery of large-scale data.
• Develop and optimize data processing logic using PySpark on Databricks (Apache Spark).
• Implement ETL/ELT pipelines integrating data from multiple structured and semi-structured sources.
• Contribute to the design and implementation of lakehouse architectures (Delta Lake, Medallion architecture).
• Ensure data quality, reliability, performance, and observability across pipelines.
• Optimize Spark jobs through partitioning, caching, and performance tuning techniques.
• Collaborate with data architects, analysts, and business stakeholders to translate requirements into scalable data solutions.
• Implement best practices in CI/CD, version control, and pipeline automation.
• Support the evolution of modern data platforms and analytics capabilities.
• Work with orchestration tools (ADF, Airflow, or similar) to manage workflows.
• Communicate technical solutions and trade-offs effectively to both technical and non-technical stakeholders.

Conocimientos

Python

SQL

Data pipelines

ETL/ELT

Lakehouse architecture

CI/CD practices

Git

Cloud platforms

Airflow

Azure Data Factory

Communication

English

Big data

Educación

BS in Computer Science, Engineering, or related technical field

Herramientas

Databricks

PySpark

Python

SQL

Airflow

Azure Data Factory

Fabric

Git

Delta Lake

Descripción del empleo

Perficient is seeking a Data Engineer to design, build, and operate end-to-end data pipelines using PySpark on Databricks. You will implement ETL/ELT pipelines, contribute to lakehouse architectures, and ensure data quality and observability across large-scale systems.

You will collaborate with data architects, analysts, and stakeholders to translate requirements into scalable data solutions, while applying CI/CD and version control practices and leveraging cloud platforms such as Azure or AWS.