Databricks Data Engineer

IT CONNECT Sp. z o.o. Sp. k.; Kraków, Wrocław; On-site; PLN 220,000 - 340,000; Full time

Job summary

IT CONNECT Sp. z o.o. Sp. k. is looking for a data engineer with Databricks expertise to design and implement scalable data processing on cloud platforms. You will work on large, diverse datasets and collaborate with architects to follow best practices.

You will use Python for automation, PySpark, and real-time streaming with Kafka or Spark Streaming, ensuring data quality and security in cloud environments.

Qualifications

• Hands-on Databricks expertise for data engineering.
• Strong Python for automation and data transformation.
• Experience with major cloud platforms (AWS/Azure/GCP).
• Solid SQL knowledge with data modeling skills.
• Experience with real-time streaming (Kafka or Spark Streaming).
• Familiarity with Terraform or Bicep (IaC) and CI/CD pipelines.
• Proficient in PySpark for distributed processing.

Responsibilities

• Design and implement data processing solutions using Databricks for large-scale and diverse datasets.
• Design, build, and enhance data pipelines with Python and cloud-native tools.
• Work closely with solution architects to define and uphold best practices in data engineering.
• Ensure data consistency, security, and scalability within cloud-based environments.

Skills

Databricks

Python

Cloud platforms

SQL

CI/CD

Kafka

PySpark

Spark Streaming

Data modeling

Data Lake

Snowflake

Synapse

English

Databricks Cert

Tools

Terraform

Bicep

Job description

Nasze wymagania:

• Experience in data engineering, with hands-on Databricks expertise.
• Strong experience in Python for automation and data transformation.
• Experience working with at least one major cloud platform (AWS, Azure, or GCP).
• Strong communication skills and strong English language skills.
• Solid understanding of SQL with experience in query optimization and data modeling.
• Familiarity with DevOps methodologies, CI/CD pipelines, and Infrastructure as Code (Terraform, Bicep).
• Experience with real-time data streaming technologies such as Kafka or Spark Streaming.
• Knowledge of cloud storage solutions like Data Lake, Snowflake, or Synapse.
• Hands-on experience with PySpark for distributed data processing.
• Relevant certifications such as Databricks Certified Data Engineer Associate or cloud-based data certifications.

O projekcie:

Projekt obejmuje projektowanie i wdrażanie rozwiązań do przetwarzania danych z wykorzystaniem Databricks, dla dużych i zróżnicowanych zbiorów danych w środowiskach chmurowych.

Zakres obowiązków:

• Design and implement data processing solutions using Databricks for large-scale and diverse datasets.
• Design, build, and enhance data pipelines with Python and cloud-native tools.
• Work closely with solution architects to define and uphold best practices in data engineering.
• Ensure data consistency, security, and scalability within cloud-based environments.