Senior Data Engineer

Bitful; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

Bitful is seeking an experienced Senior Data Engineer to design and maintain scalable data pipelines in a Databricks lakehouse. You will transform and integrate data from multiple sources using Python, PySpark, and SQL, and build analytics-ready datasets for BI tools like Tableau and Spotfire.

You will collaborate with analysts, architects, and stakeholders to improve data quality, ensure CI/CD/monitoring, and mentor junior engineers in a fast-paced cloud environment.

Qualifications

• Strong SQL skills and knowledge of data modeling, database design, and data warehousing.
• Hands-on experience with Databricks, Apache Spark, and cloud data platforms.
• Strong proficiency in Python and PySpark.
• Experience developing ETL/ELT pipelines and integrating data from multiple source systems.
• Experience preparing reliable datasets for Tableau or comparable business intelligence tools.
• Familiarity with source control, automated testing, and CI/CD practices.
• Ability to troubleshoot complex data issues and optimize pipeline performance.
• Strong communication skills and experience collaborating with technical teams and business stakeholders.

Responsibilities

• Develop and maintain scalable data pipelines and reusable datasets in a Databricks lakehouse environment.
• Ingest, validate, clean, and transform data from multiple sources using Python, PySpark, and SQL.
• Build data models and reporting datasets for Tableau, Spotfire, and analytics tools.
• Integrate data warehouse and reporting systems with cloud analytics platforms.
• Improve pipeline reliability, data quality, and processing performance.
• Implement automated testing, monitoring, documentation, and CI/CD practices.
• Partner with analysts, architects, developers, and stakeholders to deliver data solutions.
• Provide technical guidance, conduct code reviews, and mentor other data engineers.

Skills

SQL

Python

PySpark

Data modeling

CI/CD practices

Data warehousing

Communication skills

Tools

Databricks

Apache Spark

Tableau

Spotfire

Oracle

Informatica

R

Data virtualization

Job description

Description:

• Develop and maintain scalable data pipelines and reusable datasets in a Databricks lakehouse environment.
• Ingest, validate, clean, and transform structured and unstructured data from multiple sources using Python, PySpark, and SQL.
• Build data models and reporting datasets for Tableau, Spotfire, and other analytics tools.
• Integrate existing data warehouse and reporting systems with modern cloud analytics platforms.
• Improve pipeline reliability, data quality, and processing performance.
• Implement automated testing, monitoring, documentation, and CI/CD practices.
• Partner with analysts, architects, developers, and business stakeholders to deliver data solutions.
• Provide technical guidance, conduct code reviews, and mentor other data engineers.

Requirements:

• 7+ years of data engineering experience, including building and supporting production data pipelines.
• Strong SQL skills and knowledge of data modeling, database design, and data warehousing.
• Hands-on experience with Databricks, Apache Spark, and cloud data platforms.
• Strong proficiency in Python and PySpark.
• Experience developing ETL/ELT pipelines and integrating data from multiple source systems.
• Experience preparing reliable datasets for Tableau or comparable business intelligence tools.
• Familiarity with source control, automated testing, and CI/CD practices.
• Ability to troubleshoot complex data issues and optimize pipeline performance.
• Strong communication skills and experience collaborating with technical teams and business stakeholders.
• Experience with Spotfire, Oracle, Informatica, R, or data virtualization; Databricks certification preferred.