Senior Data Engineer: Spark, Databricks & Cloud Pipelines

Boardroom Appointments; Sandton; On-site; ZAR 900,000 - 1,500,000; Full time

Job summary

Boardroom Appointments is seeking a Senior Data Engineer to join our datacenter team and build/maintain an operational data pipeline. You will create, test, debug and document infrastructure components while monitoring performance and using Python to interact with MySQL or Postgres databases.

You will develop information extraction, analysis and management solutions, support urgent analytical needs and collaborate with development teams on architecture, design, and deployment of applications in

Qualifications

• Bachelor's degree in Computer Science or related field.
• 8 years of experience in a technical role with emphasis on data and algorithm development and implementation, server-side programming, database administration, web development and design.
• Knowledge of Databricks, SQL and Python preferred.
• 8 years of software development experience with a strong background in databases and data modeling.
• 5 years of experience designing and implementing high performance applications for cloud platforms such as AWS preferred.
• Core capability is on Databricks and Synapse.

Responsibilities

• Develops, tests and deploys information extraction, analysis and management solutions.
• Provides direct and responsive support for urgent analytical needs.
• Collaborates with the development team to address issues within the framework of the project requirements.
• Participates in architecture, design, testing and deployment of the applications.
• Uses coding languages, scripting methodologies and sophisticated tools to solve a problem with a custom workflow.
• Performs incremental testing actions on code, processes and deployments to identify ways to streamline execution and minimize errors.
• Installs and maintains Spark, Apache Spark, Erlang, Spark cluster.
• Manages the day to day operations of a Spark environment from provisioning, maintenance and monitoring.
• Use Python for debugging and custom data mining.

Skills

Python programming

SQL

Databricks

Apache Spark

Cloud AWS

Data modeling

Server-side programming

Education

Bachelor's degree in Computer Science

Tools

Databricks

Apache Spark

Synapse

AWS

MySQL

PostgreSQL

Job description

Boardroom Appointments is seeking a Senior Data Engineer to join our datacenter team and build/maintain an operational data pipeline. You will create, test, debug and document infrastructure components while monitoring performance and using Python to interact with MySQL or Postgres databases.

You will develop information extraction, analysis and management solutions, support urgent analytical needs and collaborate with development teams on architecture, design, and deployment of applications in