Data Engineer Lead

Bitwise; Richmond (VA); Hybrid; USD 140,000 - 180,000; Full time

Job summary

Bitwise is seeking a Data Engineer Lead to drive a Richmond, VA data platform initiative. You will design scalable data pipelines, mentor engineers, and collaborate with customers to deliver robust, cloud-based data solutions using Databricks and Azure Data Factory.

The role emphasizes hands-on Databricks development, Delta Lake governance, and Azure data integration, while promoting best practices, performance, and cost optimization in an Agile environment.

Qualifications

• 8-10 years of experience as a Lead Data Engineer with a strong focus on Databricks & Azure Data Factory.

Responsibilities

• Design, develop, and deploy data pipelines using Databricks, including data ingestion, transformation, and loading (ETL) processes.
• Lead team and drive through technical challenges and solutions. Drive technical calls with customer
• Troubleshoot and resolve data pipeline issues effectively.

Skills

Lead team leadership

Communication

Agile development

Independent worker

Data modeling

Data quality

SQL proficiency

Innovation mindset

Tools

Databricks

Azure Data Factory

Azure Synapse

Delta Lake

Unity Catalog

Parquet

Delta files

Azure Blob Storage

ADLS

Azure Storage Explorer

Azure DevOps

Job description

Role - Data Engineer Lead

Location - Richmond, VA

Fulltime / Hybrid
Job Description Qualifications:

• 8-10 years of experience as a Lead Data Engineer with a strong focus on Databricks & Azure Data Factory.
• Strong experience leading the Data Engineering teams working on ADF & Databricks. Work with the teams providing technical approaches and solutions.
• Proven experience with data warehousing concepts and best practices.
• Strong SQL skills, ability to perform effective querying involving multiple tables and subqueries.
• Strong understanding of data modeling and data quality principles.
• Strong communication & collaboration skills, should be able to work closely with customer driving technical conversations.
• Experience with Agile development methodologies.
• Ability to work independently and as part of a team.
• Lead team and drive through technical challenges and solutions. Drive technical calls with customer
• Focus on continuous value additions and innovations
• Experience with any other ETL tool (Talend, Informatica or similar)
• Good to have Insurance domain knowledge
• Good to have exposure to Gen AI concepts

Databricks

• Hands-on experience with Databricks features such as delta lake, clusters, notebooks, jobs, and workspaces.
• Design, develop, and deploy data pipelines using Databricks, including data ingestion, transformation, and loading (ETL) processes.
• Develop and maintain high-quality, scalable, and maintainable Databricks notebooks using Python.
• Work with Delta Lake and other advanced features.
• Leverage Unity Catalog for data governance, access control, and data discovery.
• Experience working with Parquet files & Delta files for data storage and processing.
• Integrate with various data sources, including but not limited to databases and cloud storage (Azure Blob Storage, ADLS, Synapse), and APIs.
• Perform data quality checks and validation to ensure data accuracy and integrity.

ADF

• Strong working Knowledge and experience in Microsoft Azure, Azure Data Factory, Synapse Analytics, Azure SQL, Data Pipelining, Data Transformation, ETL, copy activity, custom Azure development.
• Develop and optimize data pipelines for performance and cost-effectiveness.
• Sound skills and hands on experience with Azure Data Lake, SQL Data Warehouse Azure Blob, Azure Storage Explorer • Sound knowledge on Azure DevOps, CICD Processes, Release Processes
• Sound Knowledge of Azure Data Catalog, Event Grid, Service Bus, SQL and Synapse
• Troubleshoot and resolve data pipeline issues effectively.