Databricks Data Engineer

KPI Partners; United States; Remote; USD 90,000 - 130,000; Full time

Job summary

KPI Partners is seeking a Data Engineer in the United States to develop and maintain data pipelines using Databricks and Spark, collaborate with data scientists and analysts, and optimize ETL processes for analytics and reporting. The role emphasizes data quality, accessibility, and leveraging notebooks for exploration and visualization.

The candidate should be proficient in Python or Scala, with experience in cloud platforms (AWS/Azure/GCP), SQL, and data modeling.

Qualifications

• Proficiency in Databricks and Apache Spark.
• Strong programming skills in Python or Scala for data processing.
• Experience with ETL processes and tools.
• Knowledge of data modeling and data warehousing concepts.
• Familiarity with cloud platforms such as AWS, Azure, or Google Cloud.
• Experience with SQL and database technologies (e.g., MySQL, PostgreSQL, or others).
• Familiarity with data visualization tools (e.g., Tableau, Power BI) is a plus.
• Strong analytical and problem-solving skills.
• Excellent communication and collaboration skills to work effectively within a team.

Responsibilities

• Develop and maintain data pipelines using Databricks, Spark, and other data processing technologies.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and ensure data quality and accessibility.
• Implement and optimize ETL processes to support analytics and reporting initiatives.
• Monitor and troubleshoot data pipeline performance, ensuring timely delivery and reliability.
• Contribute to the design and implementation of data architecture and data models.
• Utilize Databricks notebooks for data exploration, visualization, and sharing insights with team members.
• Stay updated with the latest trends and advancements in data engineering and the Databricks platform.

Skills

Databricks

Apache Spark

Python

Scala

ETL

Data modeling

Data warehousing

AWS

Azure

GCP

SQL

MySQL

PostgreSQL

Tableau

Power BI

Analytical

Communication

Collaboration

Job description

Role & responsibilities

Key Responsibilities:

• Develop and maintain data pipelines using Databricks, Spark, and other data processing technologies.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and ensure data quality and accessibility.
• Implement and optimize ETL processes to support analytics and reporting initiatives.
• Monitor and troubleshoot data pipeline performance, ensuring timely delivery and reliability.
• Contribute to the design and implementation of data architecture and data models.
• Utilize Databricks notebooks for data exploration, visualization, and sharing insights with team members.
• Stay updated with the latest trends and advancements in data engineering and the Databricks platform.

Skills and Tools Required:

• Proficiency in Databricks and Apache Spark.
• Strong programming skills in Python or Scala for data processing.
• Experience with ETL processes and tools.
• Knowledge of data modeling and data warehousing concepts.
• Familiarity with cloud platforms such as AWS, Azure, or Google Cloud.
• Experience with SQL and database technologies (e.g., MySQL, PostgreSQL, or others).
• Familiarity with data visualization tools (e.g., Tableau, Power BI) is a plus.
• Strong analytical and problem-solving skills.
• Excellent communication and collaboration skills to work effectively within a team.