Senior Data Engineer: Databricks & Scalable Pipelines Lead

Bitful Consulting; United States; Remote; USD 120,000 - 200,000; Full time

Job summary

Bitful Consulting seeks an experienced Data Engineer to design and maintain scalable data pipelines and datasets that support reporting, visualization, ML, and data-driven decision making across finance, sales, supply chain, and marketing. You will ingest, validate, and transform data from multiple sources using Python, PySpark and SQL, and build data models for Tableau and other analytics tools.

You will collaborate with analysts, architects, developers and business stakeholders, improve data

Qualifications

• 7+ years of experience building and supporting production data pipelines
• Strong SQL skills and understanding of data modeling, database design and data warehousing
• Hands-on experience with Databricks, Apache Spark and cloud data platforms
• Strong proficiency in Python and PySpark
• Experience developing ETL/ELT pipelines and integrating data from multiple sources
• Experience preparing reliable datasets for Tableau or comparable BI tools
• Familiarity with source control, automated testing and CI/CD practices
• Ability to troubleshoot complex data issues and optimize pipeline performance
• Excellent communication and collaboration skills with technical teams and stakeholders
• Experience with Spotfire, Oracle, Informatica, R or data virtualization preferred
• Databricks certification preferred

Responsibilities

• Build and maintain scalable data pipelines and datasets for reporting, visualization, ML and data-driven decisions
• Ingest, validate, clean and transform data from multiple sources using Python, PySpark and SQL
• Develop data models and reporting datasets for Tableau and analytics tools
• Integrate data warehouse and reporting systems with modern cloud analytics platforms
• Improve pipeline reliability, data quality and processing performance
• Implement automated testing, monitoring, documentation and CI/CD practices
• Collaborate with analysts, architects, developers and stakeholders to deliver data solutions
• Provide technical guidance, review code and mentor other data engineers

Skills

SQL

Python

PySpark

Data modeling

Data warehousing

Data pipeline design

Troubleshooting

Communication

Collaboration

Mentoring

Tools

Databricks

Apache Spark

Tableau

Spotfire

Oracle

Informatica

R

Job description

Bitful Consulting seeks an experienced Data Engineer to design and maintain scalable data pipelines and datasets that support reporting, visualization, ML, and data-driven decision making across finance, sales, supply chain, and marketing. You will ingest, validate, and transform data from multiple sources using Python, PySpark and SQL, and build data models for Tableau and other analytics tools.

You will collaborate with analysts, architects, developers and business stakeholders, improve data