Databricks developer

Atos; Phoenix (AZ); On-site; USD 120,000 - 180,000; Full time

Job summary

Atos is seeking an experienced data engineering architect to lead enterprise Lakehouse initiatives on Databricks. You will design end-to-end pipelines, implement real-time streaming, and drive governance, security, and CI/CD practices across cross-functional teams.

Deep expertise in Spark, Delta Lake, and cloud-native storage, plus strong Python, PySpark, Scala, and SQL skills, are essential to translate business needs into scalable technical solutions for clients.

Qualifications

• Extensive experience architecting lakehouse solutions on Databricks.
• Strong skills in building data pipelines using PySpark, Spark, and SQL.
• Proven ability to design scalable data models and data governance.

Responsibilities

• Architect and implement enterprise-grade Lakehouse solutions using Databricks.
• Design and deliver end-to-end data engineering pipelines, including batch and real-time streaming solutions.
• Lead implementation of real-time data processing pipelines for operational and analytical use cases.
• Develop scalable ETL/ELT pipelines using PySpark, Scala, and SQL.
• Implement advanced data modeling solutions including 3NF, dimensional modeling, and enterprise data warehousing strategies.
• Design and build incremental data loading frameworks and metadata-driven ingestion pipelines.
• Establish data quality frameworks and governance standards.
• Implement and manage Unity Catalog, including fine-grained security and access controls.
• Leverage Databricks components such as Autoloader, Structured Streaming, Databricks Workflows.
• Integration with orchestration tools (e.g., Apache Airflow)
• Drive CI/CD automation, deployment strategies, and DevOps best practices.
• Optimize performance of pipelines, Spark jobs, and compute resources.
• Provide architectural guidance and technical leadership across cross-functional teams.
• Engage with stakeholders and clients to translate business requirements into scalable technical solutions.

Skills

Databricks

PySpark

Scala

SQL

Python

Structured Streaming

Tools

Databricks

Apache Airflow

Delta Lake

Job description

• Architect and implement enterprise-grade Lakehouse solutions using Databricks.
• Design and deliver end-to-end data engineering pipelines, including batch and real-time streaming solutions.
• Lead implementation of:
• Real-time data processing pipelines for operational and analytical use cases.
• Develop scalable ETL/ELT pipelines using PySpark, Scala, and SQL.
• Implement advanced data modeling solutions including 3NF, dimensional modeling, and enterprise data warehousing strategies.
• Design and build incremental data loading frameworks and metadata-driven ingestion pipelines.
• Establish data quality frameworks and governance standards.
• Implement and manage Unity Catalog, including fine-grained security and access controls.
• Leverage Databricks components such as:
• Autoloader
• Structured Streaming
• Databricks Workflows
• Integration with orchestration tools (e.g., Apache Airflow)
• Drive CI/CD automation, deployment strategies, and DevOps best practices.
• Optimize performance of pipelines, Spark jobs, and compute resources.
• Provide architectural guidance and technical leadership across cross-functional teams.
• Engage with stakeholders and clients to translate business requirements into scalable technical solutions.

Deep expertise in:

• Databricks and cloud-native storage/compute platforms
• Apache Spark (batch & streaming)
• Delta Lake & Lakehouse architecture
• Distributed data processing systems

Strong hands-on programming skills in Python, PySpark, Scala, and SQL.