Databricks Data Engineer — Real-Time Cloud Pipelines

IT CONNECT Sp. z o.o. Sp. k.; Kraków, Wrocław; On-site; PLN 220,000 - 340,000; Full time

Job summary

IT CONNECT Sp. z o.o. Sp. k. is looking for a data engineer with Databricks expertise to design and implement scalable data processing on cloud platforms. You will work on large, diverse datasets and collaborate with architects to follow best practices.

You will use Python for automation, PySpark, and real-time streaming with Kafka or Spark Streaming, ensuring data quality and security in cloud environments.

Qualifications

• Hands-on Databricks expertise for data engineering.
• Strong Python for automation and data transformation.
• Experience with major cloud platforms (AWS/Azure/GCP).
• Solid SQL knowledge with data modeling skills.
• Experience with real-time streaming (Kafka or Spark Streaming).
• Familiarity with Terraform or Bicep (IaC) and CI/CD pipelines.
• Proficient in PySpark for distributed processing.

Responsibilities

• Design and implement data processing solutions using Databricks for large-scale and diverse datasets.
• Design, build, and enhance data pipelines with Python and cloud-native tools.
• Work closely with solution architects to define and uphold best practices in data engineering.
• Ensure data consistency, security, and scalability within cloud-based environments.

Skills

Databricks

Python

Cloud platforms

SQL

CI/CD

Kafka

PySpark

Spark Streaming

Data modeling

Data Lake

Snowflake

Synapse

English

Databricks Cert

Tools

Terraform

Bicep

Job description

IT CONNECT Sp. z o.o. Sp. k. is looking for a data engineer with Databricks expertise to design and implement scalable data processing on cloud platforms. You will work on large, diverse datasets and collaborate with architects to follow best practices.

You will use Python for automation, PySpark, and real-time streaming with Kafka or Spark Streaming, ensuring data quality and security in cloud environments.