Remote Databricks Lakehouse Engineer - Senior

Cognizant; Navalcarnero; Presencial; EUR 60.000 - 90.000; Jornada completa

Descripción de la vacante

Cognizant in Spain is seeking a Databricks Developer with 5+ years to design, build and optimize enterprise data pipelines on the Databricks Lakehouse platform. You will transform raw data into high-quality datasets and support analytics, BI dashboards and GenAI initiatives in a fully remote setup.

You will implement Medallion architecture, optimize Spark workloads, govern data with Unity Catalog, and collaborate with data scientists.

Formación

• 5+ years of dedicated data engineering experience in an enterprise environment.
• Expert-level proficiency in Python and SQL.
• Extensive hands-on experience with Databricks, Apache Spark and Delta Lake.
• Strong understanding of distributed systems, big data architecture and data modeling techniques (e.g., Kimball, Data Vault).
• Deep familiarity with cloud-native data services (AWS, Azure or GCP), specifically cloud storage (S3/ADLS) and compute provisioning.
• Proven experience with version control (Git), CI/CD methodologies and agile software development life cycles.

Responsabilidades

• Design, build and maintain scalable, robust ETL/ELT pipelines using Python, SQL and Apache Spark within the Databricks environment.
• Implement and manage a robust Medallion architecture (Bronze, Silver, Gold layers) to process and refine data from diverse sources.
• Develop and maintain the Gold semantic layer specifically optimized for high-performance consumption by BI tools (e.g., Power BI).
• Optimize Databricks workloads, cluster configurations and Spark queries to ensure high performance and cost efficiency.
• Work extensively with open table formats, specifically Delta Lake and Apache Iceberg, to ensure ACID compliance, time travel and efficient data storage.
• Execute complex data migrations, including transitioning legacy workloads from traditional cloud data warehouses (e.g., AWS Redshift) into the Databricks Lakehouse.
• Implement data governance and access control policies at the table, row and column levels using Databricks Unity Catalog.
• Automate deployment processes and pipeline orchestration using Databricks Workflows, CI/CD pipelines (e.g., GitHub Actions, Azure DevOps) and tools like Terraform.
• Embed data quality checks and monitoring directly into pipelines to ensure strict Master Data Management (MDM) standards are upheld.
• Collaborate closely with Data Scientists and AI Engineers to provision clean, structured data for machine learning model training and inference.
• Support the data foundations required for GenAI frameworks, autonomous agents and AI observability platforms.

Conocimientos

Python

SQL

Databricks

Apache Spark

Delta Lake

Cloud platforms

Git

Terraform

Data modeling

Educación

Bachelor's or Master's in Computer Science / IT / Data Engineering

Herramientas

Databricks

Apache Spark

Delta Lake

AWS/Azure/GCP

Git

Terraform

Descripción del empleo

Cognizant in Spain is seeking a Databricks Developer with 5+ years to design, build and optimize enterprise data pipelines on the Databricks Lakehouse platform. You will transform raw data into high-quality datasets and support analytics, BI dashboards and GenAI initiatives in a fully remote setup.

You will implement Medallion architecture, optimize Spark workloads, govern data with Unity Catalog, and collaborate with data scientists.