Databricks Lakehouse Architect & Real-Time Data Engineer

Atos; Phoenix (AZ); On-site; USD 120,000 - 180,000; Full time

Job summary

Atos is seeking an experienced data engineering architect to lead enterprise Lakehouse initiatives on Databricks. You will design end-to-end pipelines, implement real-time streaming, and drive governance, security, and CI/CD practices across cross-functional teams.

Deep expertise in Spark, Delta Lake, and cloud-native storage, plus strong Python, PySpark, Scala, and SQL skills, are essential to translate business needs into scalable technical solutions for clients.

Qualifications

• Extensive experience architecting lakehouse solutions on Databricks.
• Strong skills in building data pipelines using PySpark, Spark, and SQL.
• Proven ability to design scalable data models and data governance.

Responsibilities

• Architect and implement enterprise-grade Lakehouse solutions using Databricks.
• Design and deliver end-to-end data engineering pipelines, including batch and real-time streaming solutions.
• Lead implementation of real-time data processing pipelines for operational and analytical use cases.
• Develop scalable ETL/ELT pipelines using PySpark, Scala, and SQL.
• Implement advanced data modeling solutions including 3NF, dimensional modeling, and enterprise data warehousing strategies.
• Design and build incremental data loading frameworks and metadata-driven ingestion pipelines.
• Establish data quality frameworks and governance standards.
• Implement and manage Unity Catalog, including fine-grained security and access controls.
• Leverage Databricks components such as Autoloader, Structured Streaming, Databricks Workflows.
• Integration with orchestration tools (e.g., Apache Airflow)
• Drive CI/CD automation, deployment strategies, and DevOps best practices.
• Optimize performance of pipelines, Spark jobs, and compute resources.
• Provide architectural guidance and technical leadership across cross-functional teams.
• Engage with stakeholders and clients to translate business requirements into scalable technical solutions.

Skills

Databricks

PySpark

Scala

SQL

Python

Structured Streaming

Tools

Databricks

Apache Airflow

Delta Lake

Job description

Atos is seeking an experienced data engineering architect to lead enterprise Lakehouse initiatives on Databricks. You will design end-to-end pipelines, implement real-time streaming, and drive governance, security, and CI/CD practices across cross-functional teams.

Deep expertise in Spark, Delta Lake, and cloud-native storage, plus strong Python, PySpark, Scala, and SQL skills, are essential to translate business needs into scalable technical solutions for clients.