Senior Data Engineer (Databricks)

Opinov8; United States; Remote; USD 100,000 - 150,000; Full time

Job summary

A leading company specializing in risk assessment seeks a Senior Data Engineer to design and implement data pipelines. The role offers the opportunity to work remotely on impactful projects in a collaborative environment, utilizing modern data technologies.

Benefits

Digital-First Approach

Flexible Working Conditions

Opportunities for Continuous Learning

Collaborate with a Diverse Team

Qualifications

• Extensive experience in designing and implementing data pipelines.
• Familiarity with modern data stack and analytics.
• Proficiency in tools such as PySpark, Databricks, and Azure technologies.

Responsibilities

• Model datasets for internal and external consumption.
• Establish and maintain data documentation.
• Set software engineering best practices for analytics.

Skills

Data Processing with Spark

Structured Streaming

Python

SQL

Databricks

Kafka

Azure Cloud

Tools

PowerBI

Terraform

Azure DevOps

Job description

PROJECT DESCRIPTION:

Our client is a company that provides risk assessment and valuable statistics on bank transactions to their clients.
We are seeking a Senior Data Engineer with extensive hands-on experience in designing and implementing data pipelines.
Ideally, the candidate is familiar with the modern data stack from both engineering and analytics perspectives.

Besides building data pipelines, the Senior Data Engineer is expected to assume the following responsibilities:

• Model datasets for internal and external consumption;
• Establish and maintain data documentation;
• Set software engineering best practices for analytics (DataOps);
• Collaborate closely with team members and business stakeholders.

IF YOU HAVE:

Good knowledge and experience in the modern data stack:

• Ingestion (using Autoloader, potential future use of Kafka);
• Storage (Azure Data Lake Storage);
• Data transformation (PySpark, structured streaming);
• Workflow orchestration (Databricks workflows);
• Data import/export using queues (Azure EventHub/Kafka);
• Data governance, catalog, quality (Unity Catalog with RBAC, Great Expectations);
• BI tools (PowerBI).

Expected skills:

- Strong understanding of Distributed Data Processing with Spark;

- Experience with structured streaming, Autoloader, UDFs, pandas, etc.;

- Proficiency with Databricks and Asset Bundles;

- Experience with Kafka and Azure EventHub;

- Python and SQL expertise;

- PowerBI experience (a plus);

- Understanding of Terraform (a plus);

- Experience with Azure Cloud & Azure DevOps (a plus).

AS AN OPINOV8R, YOU WILL HAVE:

• Digital-First Approach: Work from anywhere, collaborating globally.
• Remote Work Model: Flexible working conditions to balance professional and personal life.
• Exciting Projects: Impactful projects across various industries that challenge and inspire.
• Boost Your Expertise: Opportunities for continuous learning and growth.
• Join the Best Team Ever: Collaborate with a diverse, cross-cultural team of passionate technologists and thinkers.

HOW’S THE HIRING PROCESS GOING:

We aim for a smooth, transparent process to find the right fit:

• Initial Interview: With a Talent Acquisition Specialist if your background matches.
• Technical Assessment: Complete a test or task to demonstrate your skills.
• Final Decision: After interviews and assessments, we will inform you of the outcome.