Cloud Data Engineer – Big Data, PySpark & Python
Iprime Info
Get a reply from this employer
Your resume and cover letter, tailored to exactly what they’re hiring for.
Job summary
Iprime Info in Overland Park, KS seeks a data engineering professional to design and manage scalable Dataflow jobs for historical and real-time data processing. You will develop Hadoop/Hive/PySpark solutions using Python, and build robust data pipelines with Pub/Sub, BigQuery and Cloud Functions.
The role emphasizes Agile collaboration, DevOps practices, and monitoring to ensure reliable data processing and analytics. Familiarity with logging and diagnostics is essential.
Qualifications
- Master’s degree in Computer Science, Engineering or related field.
- Experience designing and managing scalable batch and streaming Dataflow jobs.
- Strong knowledge of Hadoop, Hive, PySpark and Python.
Responsibilities
- Design and manage scalable batch and streaming Dataflow jobs for processing historical and real-time data.
- Develop Big Data applications using Hadoop, Hive, PySpark, and Python.
- Build centralized logging systems by integrating Dataflow logs with Cloud Logging for monitoring and troubleshooting.
- Use SQL and BigQuery for data transformation and analytics, and leverage Pub/Sub for reliable data ingestion.
- Automate data workflows with Cloud Functions and perform large-scale data analysis using Python and Pandas.
- Collaborate in Agile teams, following DevOps best practices for deployment and monitoring.
Skills
Education
Tools
Job description
Iprime Info in Overland Park, KS seeks a data engineering professional to design and manage scalable Dataflow jobs for historical and real-time data processing. You will develop Hadoop/Hive/PySpark solutions using Python, and build robust data pipelines with Pub/Sub, BigQuery and Cloud Functions.
The role emphasizes Agile collaboration, DevOps practices, and monitoring to ensure reliable data processing and analytics. Familiarity with logging and diagnostics is essential.
'/%3e%3cpath%20d='M101.744%2012.4726L102.894%2016.1947L99.0021%2013.2201L101.744%2012.4726ZM105.3%208.40937H100.488L99.0026%203.59473L97.5116%208.41047L92.7%208.40385L96.5965%2011.3834L95.1055%2016.1942L99.0021%2013.2196L101.408%2011.3834L105.3%208.40992V8.40937Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_5593_1381'%20x1='90'%20y1='9.44471'%20x2='99'%20y2='9.44471'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2300AD70'/%3e%3cstop%20offset='0.998594'%20stop-color='%2300B67A'/%3e%3cstop%20offset='1'%20stop-color='%23DCDCE6'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e){kind=small}
