MS Fabric Data Engineer

KPI Partners; San Francisco (CA); Hybrid; USD 150,000 - 190,000; Full time

Job summary

KPI Partners in San Francisco is seeking a Senior/Lead Data Engineer for a 12-month contract, hybrid work with three onsite days. You will design and build data pipelines and warehousing on Azure native services, PySpark, and Databricks as part of multi-year transformation projects.

Join a high-impact team focused on scalable data platforms, mentoring peers, and delivering robust solutions that improve data reliability, performance, and business insights.

Qualifications

• 10+ years overall IT experience.
• 8+ years data engineering focusing on large-scale solutions.
• 4+ years Azure stack experience (Azure, ADF, Databricks, Functions).
• 1-2 years Microsoft Fabric.
• Proven SQL expertise for large datasets.
• Strong ETL and data warehousing fundamentals.
• Self-motivated and able to thrive in a fast-paced environment.

Responsibilities

• Data Engineering : Design, development, and implementation of data pipelines and solutions using PySpark, SQL, and related technologies.
• Collaboration: Work closely with cross-functional teams to understand business requirements and translate them into robust data solutions.
• Data Warehousing: Design and implement data warehousing solutions, ensuring scalability, performance, and reliability.
• Continuous Learning: Stay up to date with modern technologies and trends in data engineering and apply them to improve our data platform.
• Mentorship: Provide guidance and mentorship to junior data engineers, ensuring best practices in coding, design, and development.

Skills

SQL

PySpark

Azure

Databricks

Microsoft Fabric

ADF

Job description

KPI Partners is a 5 times Gartner recognized data, analytics, and AI consulting company. We are leaders in data engineering on Azure, AWS, Google, Snowflake and Databricks. Founded in 2006, KPI has over 400 consultants and has successfully delivered over 1,000 projects to our clients. We are looking for skilled data engineers who want to work with the best team in data engineering.

Job type

Contract - 12 Months

Location

San Francisco Bay Area (Hybrid) - 3 Days onsite in a week

Key Skills

Sql, PySpark, Azure, Microsoft Fabric, ADF, and Databricks.

About the Role

We are seeking a highly skilled and experienced Senior/Lead Data Engineer to join our dynamic team at KPI, working on challenging and multi-year data transformation projects. This is an excellent opportunity for a talented data engineer to play a key role in building innovative data solutions using Azure Native Services and related technologies. If you are passionate about working with large-scale data systems and enjoy solving complex engineering problems, this role is for you.

Key Responsibilities

• Data Engineering : Design, development, and implementation of data pipelines and solutions using PySpark, SQL, and related technologies.
• Collaboration: Work closely with cross-functional teams to understand business requirements and translate them into robust data solutions.
• Data Warehousing: Design and implement data warehousing solutions, ensuring scalability, performance, and reliability.
• Continuous Learning: Stay up to date with modern technologies and trends in data engineering and apply them to improve our data platform.
• Mentorship: Provide guidance and mentorship to junior data engineers, ensuring best practices in coding, design, and development.

Must-Have Skills & Qualifications

• Minimum 10+ years of overall experience in IT Industry.
• 8+ years of experience in data engineering, with a strong background in building large-scale data solutions.
• 4+ years of hands-on experience developing and implementing data pipelines using Azure stack experience (Azure, ADF, Databricks, Functions)
• 1-2 years of experience in Microsoft Fabric
• Proven expertise in SQL for querying, manipulating, and analyzing large datasets.
• Strong knowledge of ETL processes and data warehousing fundamentals.
• Self-motivated and independent, with a "let's get this done" mindset and the ability to thrive in a fast-paced and dynamic environment.

Good-to-Have Skills

• Databricks & MS Fabric Certification is a plus.