Engineer, Big Data - Databricks/Python/SQL/Power BI - Remote

Lensa; Madison (WI); Remote; USD 80,000 - 110,000; Full time

Job summary

An innovative company is seeking a skilled Data Engineer to lead the charge in building scalable data solutions. This role involves designing and implementing data architectures that support both batch and real-time processing, ensuring high performance and reliability. You will work with cutting-edge technologies like Hadoop and Kafka, collaborating with cross-functional teams to deliver optimal data solutions. If you're passionate about data and eager to make a significant impact in a dynamic environment, this opportunity is perfect for you.

Qualifications

• Bachelor's degree required; master's preferred.
• Experience in building stream-processing systems and Big Data tools.

Responsibilities

• Collect, store, and analyze large data sets for optimal solutions.
• Implement and monitor data ingestion/extraction processes.

Skills

Data Architecture

ETL Processes

Hadoop

Data Modeling

Java

Python

NoSQL Databases

API Development

Education

Bachelor's Degree

Master's Degree

Tools

Kafka

Spark

HDFS

Talend

Job description

Responsible for collecting, storing, processing, and analyzing large sets of data. The primary focus will be on choosing optimal solutions to use for these purposes, then maintaining, implementing, and monitoring them. He/she will also be responsible to follow architecture and best practices used across the enterprise.

Knowledge/Skills/Abilities

• Define ideal Architecture, Evaluating tools and Frameworks, Standards & Best Practices for implementing scalable business solutions
• Implement Batch and Real-time data ingestion/extraction processes through ETL, Streaming, API, etc., between diverse source and target systems with structured and unstructured datasets
• Design and build data solutions with an emphasis on performance, scalability, and high-reliability
• Code, test, and document new or modified data systems to create robust and scalable applications for data analytics
• Build data model for analytics and application layers
• Work closely with multiple teams and Business partners, for collecting requirements and providing optimal solutions
• Proven experience on Hadoop cluster components and services (like HDFS, YARN, ZOOKEEPER, AMBARI/CLOUDERA MANAGER, SENTRY/RANGER, KERBEROS, etc.)
• Ability to troubleshoot technical issues while engaged with infrastructure and vendor support teams

Job Qualifications

Required Education

Bachelor's Degree

Required Experience

• Experience in building stream-processing systems, using solutions such as Kafka, Storm or Spark-Streaming
• Proven experience on Big Data tools such as Spark, Hive, Impala, Polybase, Phoenix, Presto, Kylin, etc.
• Experience with integration of data from multiple data sources (using ETL tools such as Talend, etc.)
• Experience building solutions with NoSQL databases, such as HBase, Memsql
• Strong experience with Database technologies, Data Warehouse, Data Validation & Certification, Data Quality, Metadata Management and Data Governance
• Experience with programming languages such as Java/Scala/Python, etc.
• Experience implementing Web application and Web Services APIs (REST/SOAP)

Preferred Education

Master's Degree

Preferred Experience

Experience in the healthcare industry is preferred

Note: The detailed job description and company information are relevant for understanding the role but should be included in the final posting as appropriate.