DataBricks Engineer (Insurance)

TECHEAD; Richmond (VA); On-site; USD 150,000 - 160,000; Full time

Job summary

TECHEAD is seeking an IT data engineering professional to join the data solutions team on a direct hire, 100% onsite basis in Virginia. You will enhance DataBricks capabilities, dashboards, and Delta tables to improve data usability for underwriting and policy cost insights in the Finance/Insurance domain.

You will own the full SDLC from requirements through testing and production support, collaborate with business users and a design-led team, and help implement robust data governance and

Qualifications

• Minimum 3-5 years’ experience using Databricks in Azure or AWS.
• Experience with data modeling and medallion architecture.
• Experience building data pipelines with SQL and PySpark.
• Knowledge of data governance and data observability.

Responsibilities

• Own end-to-end SDLC from requirements to production support.
• Collaborate with business users on data requirements.
• Work with architects and tech lead on design for project scope.
• Create documentation and training for reusable frameworks.
• Implement and maintain data security policies and compliance.

Skills

Databricks

SQL

PySpark

CI/CD

GitLab

Agile methodologies

Unit testing

SCD2

Data governance

Data quality

Cloud platforms (Azure/AWS)

Automation

Tools

Lakeflow

Unity Catalog

Spark UI

Genie AI

Intelligent Document Processing

Job description

TECHEAD is celebrating over thirty-five years of incredible heritage, talent, and accomplishments! To learn more about TECHEAD, visit us at TECHEAD.com or onGlassdoor.

Direct Hire

100% Onsite

Salary Range: $150k - $160k

As an IT team member on the data solutions and data engineering team, you will be joining a team that is transforming data solutions and capabilities in the Finance / Insurance Field. You will be focused on improving out of the box DataBricks capabilities and executing project related scope in DataBricks.

You will be improving out of the box DataBricks capabilities with DataBricks dashboards, frameworks, and infrastructure Delta tables. An example is building a framework so other teams can collect the needed data for a DataBricks dashboard heat map to show what pipelines are current to an SLA in a data streaming or micro batch environment.

You will be using the medallion architecture with many data ingestion (bronze), standardization (silver) and curation (gold) functions to improve the data usability with ER and dimensional modeling to create better data structures, values, and derived attributes over legacy data solutions. A specific scope is leveraging underwriting data to improve understanding of future policy costs. This includes using AI to extract data from images and pdfs which was the source of many medical documents that are received about a persons health before approving a policy.

What you will be doing

• You will be expected to perform the entire SDLC process from high level requirements definition, build, testing, and production support.
• Working with business users on data requirements.
• Teaming with architects and a tech lead on the design for specific project scope.

Estimating work effort

• Performing all roles in the SDLC process build and testing are the core roles for this position.
• Creating documentation, training, and consulting with other teams so they know how to leverage frameworks that you have built or enhanced as mentioned above in the example of a SLA data freshness framework for all pipelines.
• Implement and maintain data security policies and procedures, ensuring compliance with industry and company standards and regulations.
• Work with downstream and upstream teams including onshore/offshore vendor teams.
• Ability to quickly upskill in any additional cloud technologies as required.
• Ability to assume new challenges with respect to different technologies or features in DataBricks such as Intelligent Document Processing, Genie coding, and decide on best practices on how to use these features so others can leverage your work and experience.

What are your technical capabilities

• Minimum 3-5 years’ experience full time using Databricks in Azure or AWS using the medallion architecture and data models.
• Using or creating coding frameworks or functions for common data pipeline standardization processes such as standardizing missing values, dates, reference data management, and data repair or imputation for missing values.
• Creating derived business values with simple to complex logic.
• Creating standard dimensions and fact tables with SCD2 time tracking.
• DataBricks orchestration skills with batch and streaming pipelines
• Creating unit tests, performing SIT, and collaborating with users on UAT test results debugging
• DataBricks compute and storage optimization skills
• Comprehensive understanding of automation and deployment tools such as CI/CD and Gitlab.
• Strong understanding of Agile and development methodologies including story creation and defining technical tasks with effort estimates.

Preferred Qualifications

• Five years of experience with DataBricks in a transformation environment where you owned and built significant capabilities from scratch
• Experience with insurance business processes and knowledge of insurance data
• Experience and usage of data governance platforms including data observability build experience.
• Capability to understand data models and how to make minor changes or attribute additions.

Hands on Skills

• Expert in SQL and Intermediate or better in PySpark programming
• Usage of Lakeflow and data quality expectations coding
• Intermediate or better on DataBricks features have been GA for two years or longer including Unity Catalog, Spark UI, Job Scheduling, etc.
• Capability to learn newer features such as Genie AI coding, Intelligent Document Processing, Self-service workspaces, etc.

General Skills

• Excellent communication skills to create training videos and communication with business users and other IT teams
• Problem solving
• Design thinking

TECHEAD's mission is to make our associates successful by placing them in the right environment so they can grow and prosper. How we treat and respond to our clients and employees is a reflection of who we are and makes us stand out from the rest. Keeping our business focused on building and maintaining relationships with our employees and clients is the key to our success. We won't strive for anything less.

TECHEAD provides equal employment opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, gender, sexual orientation, gender identity or expression, national origin, age, disability, genetic information, marital status, amnesty, or status as a covered veteran in accordance with applicable federal, state and local laws governing non-discrimination in employment in every location in which the company has facilities. This policy applies to all terms and conditions of employment, including, but not limited to, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation, and training.

For more information on TECHEAD please visit www.techead.com.