Data Engineer - Pipelines, ML & Spark on Databricks

Avnet

Lexington (KY)

On-site

USD 90,000 - 130,000

Full time

4 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Paid Time Off
401K & Pension
Paid Holidays
Family Support
Medical Insurance
Life Insurance
Disability Insurance
HSA / FSA
Education Assistance
Employee Development
Wellness & Mentorship

Job summary

Avnet in Lexington, KY seeks a Data Engineer to develop and maintain data pipelines and scalable data-processing solutions that transform diverse source data into trusted information for business applications, analytics, and AI.

You will apply data engineering and machine learning techniques to improve data quality, automate classification and extraction, and support data products; collaborate with teams to translate requirements into practical data solutions.

Qualifications

  • Bachelor's degree or equivalent experience.
  • Hands-on experience with Python and SQL.
  • Experience with data transformation, relational databases, and automated data processing.
  • Exposure to Databricks, Apache Spark, cloud data platforms, machine learning libraries, APIs, and version control.

Responsibilities

  • Develops, maintains, and troubleshoots automated data pipelines to ingest, transform, and integrate data from enterprise systems, APIs, files, and external sources.
  • Uses Python, SQL, and distributed processing frameworks such as Apache Spark to prepare and process large datasets in cloud platforms such as Databricks.
  • Implements data cleansing, normalization, deduplication, reconciliation, and validation logic to improve the consistency, completeness, and reliability of consolidated datasets.
  • Supports the development, testing, and application of machine learning models for product classification, attribute mapping, data enrichment, and other automation use cases.
  • Processes structured and unstructured content, including product records, technical documents, and associated digital assets, using programmatic extraction and AI-assisted methods.
  • Builds and maintains data models and curated datasets that support downstream applications, search, analytics, and AI solutions.
  • Monitors pipeline execution and data quality; investigates failures, resolves discrepancies, and contributes to improvements in processing performance and reliability.
  • Applies established data governance, access-control, metadata, and documentation standards to support traceable and maintainable data solutions.
  • Collaborates with technical teams and business stakeholders to translate requirements into practical data solutions and communicate results, limitations, and business value.

Skills

Python
SQL
Distributed processing (Apache Spark)
Machine learning concepts
Data governance

Education

Bachelor's degree or equivalent experience

Tools

Databricks
APIs
Version control
Cloud data platforms

Job description

Avnet in Lexington, KY seeks a Data Engineer to develop and maintain data pipelines and scalable data-processing solutions that transform diverse source data into trusted information for business applications, analytics, and AI.

You will apply data engineering and machine learning techniques to improve data quality, automate classification and extraction, and support data products; collaborate with teams to translate requirements into practical data solutions.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Databricks Engineer: Spark, Delta Lake & Pipelines
Senior Databricks Engineer: Spark, Delta Lake & Pipelines

Ethereum Technologies LLC • Louisville (KY)

On-site
USD 140,000 - 190,000
Senior Databricks Engineer – Remote Data Pipelines
Senior Databricks Engineer – Remote Data Pipelines

Princeton IT Services, Inc • United States

Remote
USD 130,000 - 180,000
Databricks Data Engineer — Pipelines & Big Data
Databricks Data Engineer — Pipelines & Big Data

Cyber Space Technologies LLC • Columbus (OH)

On-site
USD 90,000 - 115,000
Data Engineer - Databricks, PySpark & ETL Pipelines
Data Engineer - Databricks, PySpark & ETL Pipelines

RADcube • Indianapolis (IN)

On-site
USD 110,000 - 160,000
Data Engineer I: AI-Driven Pipelines on Databricks & AWS
Data Engineer I: AI-Driven Pipelines on Databricks & AWS

Travelers Companies, Inc. • Hartford (CT)

On-site
USD 109,000 - 180,000
Health Insurance
401(k) Matching
Paid Time Off
+2
Databricks Data Engineer: Spark & Delta Lake Pipelines
Databricks Data Engineer: Spark & Delta Lake Pipelines

Ethereum Technologies • Madison (WI)

On-site
USD 110,000 - 160,000
Senior Databricks Data Engineer for AI-Powered Pipelines
Senior Databricks Data Engineer for AI-Powered Pipelines

Ciandt • United States

Remote
USD 120,000 - 160,000
Premium Healthcare
Meal voucher
Maternity and Parental leaves
+5
Databricks Data Engineer — Pipelines & Client-Facing Hybrid
Databricks Data Engineer — Pipelines & Client-Facing Hybrid

Infinitive • McLean (VA)

Hybrid
USD 75,000 - 120,000
Data Engineer – Databricks Pipelines & GenAI (Remote)
Data Engineer – Databricks Pipelines & GenAI (Remote)

100 Primoris Services Corporation • Dallas (TX)

Remote
USD 115,000 - 120,000
401k with employer match
Health/Dental/Vision insurance
Paid time off
+2
Data Engineer - Databricks + ADF Data Pipelines
Data Engineer - Databricks + ADF Data Pipelines

Ethereum Technologies LLC • Plano (TX)

On-site
USD 118,000 - 136,000