Databricks Data Engineer: Lakehouse Pipelines & PySpark

Perficient; Colombia; Presencial; Jornada completa

Descripción del empleo

Job Description

• Design, build, and maintain end-to-end data pipelines for ingestion, transformation, and delivery of large‑scale data.
• Develop and optimize data processing logic using PySpark on Databricks (Apache Spark).
• Implement ETL/ELT pipelines integrating data from multiple structured and semi‑structured sources.
• Contribute to the design and implementation of lakehouse architectures (Delta Lake, Medallion architecture).
• Ensure data quality, reliability, performance, and observability across pipelines.
• Optimize Spark jobs through partitioning, caching, and performance tuning techniques.
• Collaborate with data architects, analysts, and business stakeholders to translate requirements into scalable data solutions.
• Implement best practices in CI/CD, version control, and pipeline automation.
• Support the evolution of modern data platforms and analytics capabilities.
• Work with orchestration tools (ADF, Airflow, or similar) to manage workflows.
• Communicate technical solutions and trade‑offs effectively to both technical and non‑technical stakeholders.

Qualifications

• BS in Computer Science, Engineering, or related technical field, or equivalent practical experience.
• 4+ years of experience in Data Engineering or similar roles.
• Strong experience with Databricks and PySpark for large‑scale data processing.
• Solid knowledge of Spark architecture and performance optimization.
• Strong proficiency in Python and SQL.
• Experience designing and building data pipelines and ETL/ELT processes.
• Understanding of modern data architectures (lakehouse, distributed systems, big data processing).
• Experience with version control (Git) and CI/CD practices.
• Familiarity with cloud platforms (Azure preferred, AWS also acceptable).
• Experience with orchestration tools (Azure Data Factory, Airflow, Fabric).
• Strong communication skills and ability to work with cross‑functional teams.
• Advanced English (written and spoken).
• Proactive, analytical mindset and comfortable working in fast‑paced environments.

Benefits

• Competitive compensation and paid time off, including holidays.
• Benefits package that includes health, dental, and vision coverage.
• Reward and recognition programs, including billable bonus opportunities.

Equal Employment Opportunity

Perficient, Inc. proudly provides equal employment opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, gender, sexual orientation, national origin, age, disability, genetic information, marital status, amnesty, or status as a protected veteran in accordance with applicable federal, state and local laws. Perficient, Inc. complies with applicable state and local laws governing non‑discrimination in employment in every location in which the company has facilities. This policy applies to all terms and conditions of employment, including, but not limited to, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation, and training. Perficient, Inc. expressly prohibits any form of unlawful employee harassment based on race, color, religion, gender, sexual orientation, national origin, age, genetic information, disability, or covered veterans. Improper interference with the ability of Perficient, Inc. employees to perform their expected job duties is absolutely not tolerated.

Disability Accommodations

Perficient is committed to providing a barrier‑free employment process with reasonable accommodations for qualified individuals with disabilities and disabled veterans in our job application procedures. If you need assistance or accommodation due to a disability, please contact us.

Disclaimer

Disclaimer: The above statements are not intended to be a complete statement of job content, rather to act as a guide to the essential functions performed by the employee assigned to this classification. Management retains the discretion to add or change the duties of the position at any time.