Senior Data Pipeline Engineer — Python, Spark & Kafka

Jobtailor; Kentucky; On-site; USD 110,000 - 140,000; Full time

Job summary

Jobtailor in Kentucky is seeking a Data Engineer to design, build, and maintain scalable data pipelines and ETL/ELT processes, leveraging Python, PySpark, and Java to transform large datasets.

You will ingest structured and unstructured data via REST APIs, streaming platforms, and batch jobs, work with Kafka and Hadoop, and ensure data quality and governance in an Agile team environment.

Qualifications

• Bachelor's degree required and 4+ years in Data Engineering or Big Data development.
• Hands-on experience with Python, PySpark, Java, Kafka, Hadoop, REST APIs, and SQL.
• Strong analytical and problem-solving abilities.
• Ability to work effectively in both team-oriented and independent environments.

Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL/ELT processes.
• Build and optimize large-scale data processing solutions using Python, PySpark, and Java.
• Develop data ingestion frameworks for structured and unstructured data sources.
• Integrate data from various systems through REST APIs, streaming platforms, and batch processing.
• Work with large datasets in distributed computing environments.
• Develop and support real-time and batch data processing solutions using Kafka and Hadoop ecosystem technologies.
• Implement scalable streaming data pipelines and event-driven architectures.
• Monitor and optimize data workflows for performance and reliability.
• Write complex SQL queries for data extraction, transformation, and analysis.
• Design and optimize database schemas and data models.
• Ensure data quality, consistency, and governance standards are maintained.
• Participate in Agile ceremonies including sprint planning, daily stand-ups, backlog grooming, and retrospectives.
• Use Jira for project tracking, issue management, and sprint execution.
• Collaborate with Data Architects, Data Scientists, Business Analysts, and Application Development teams.
• Work independently as well as within a collaborative team environment.
• Troubleshoot and resolve data-related issues across environments.
• Perform root cause analysis and implement long-term solutions.
• Support production deployments and ongoing maintenance activities.

Skills

Python

PySpark

Java

SQL

ETL/ELT

Data modeling

Distributed computing

Education

Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field

Tools

Kafka

Hadoop

REST APIs

Jira

Job description

Jobtailor in Kentucky is seeking a Data Engineer to design, build, and maintain scalable data pipelines and ETL/ELT processes, leveraging Python, PySpark, and Java to transform large datasets.

You will ingest structured and unstructured data via REST APIs, streaming platforms, and batch jobs, work with Kafka and Hadoop, and ensure data quality and governance in an Agile team environment.