Lead Data Engineer

JPS Tech Solutions; Atlanta (GA); On-site; USD 140,000 - 170,000; Full time

Job summary

JPS Tech Solutions, based in Atlanta, GA, seeks a Sr Data Engineer to build scalable ETL pipelines with Spark, ensuring data availability and quality for security analytics across client networks. Leverage AWS for scalable solutions and apply machine learning to automate data-driven operations.

Responsibilities include deploying CI/CD pipelines for data products, monitoring data quality, and collaborating across teams to socialize results and drive action within the organization.

Qualifications

• Ability to read, write, speak and understand English.
• Strong communication and presentation skills.
• Expertise in data engineering languages and Spark for BigData.
• Proficiency in building and managing ETL pipelines.
• Quantitative analysis including model interpretation and causality awareness.
• Experience with relational databases and NoSQL platforms.

Responsibilities

• Design, develop, and maintain scalable ETL pipelines to ensure data quality and availability.
• Implement monitoring and alerting solutions to ensure pipeline reliability and performance.
• Develop and manage deployment pipelines to enable CI/CD of data solutions.
• Implement data integration to support analytics and reporting needs.
• Execute the analytics lifecycle: data exploration, grooming, model prototyping, validation.

Skills

ETL pipelines

Spark

Python

SQL

Education

Bachelor's degree in CS/Math/Stats/Physics/OR related

Masters in CS or related field

Tools

Scala

Java

Job description

Job Description:

As part of the Network Analytics team, the Sr Data Engineer role is crucial in executing advanced analytic techniques to enhance security and operational efficiency of client enterprise networks. This position focuses on constructing robust ETL pipelines using Spark to ensure data availability and quality for analytics tasks. The role requires expertise in data engineering, leveraging AWS for scalable solutions and enhancing operational efficiency. The role also requires expertise in statistical analysis and machine learning, utilizing generative AI to build data-driven, automated systems that enhance operational efficiency.

Roles & Responsibilities:

Design, develop, and maintain scalable ETL pipelines to ensure data quality and availability

Implement monitoring and alerting solutions to ensure data pipeline reliability and performance

Develop and manage deployment pipelines to facilitate continuous integration and delivery of data engineering solutions

Implement data integration solutions to support analytics and reporting needs

Execute the complete analytics lifecycle for problem solving, including:

• Algorithm traditionalization


• Model validation


• Model prototyping


• Data exploration


• Data grooming

Survey varied data sources for analytic relevance, including:

• External sources accessed via API


• Flat files


• Relational databases


• Distributed file systems

Interpret, synthesize and communicate results of analyses to effect action and changes within the organization

Collaborate across teams to integrate analytic products with existing production architecture, develop, execute, and evaluate courses of action, and socialize results

Help teach and explain techniques and tools used to a broad set of business

Required Qualifications:

Skills/Abilities and Knowledge

• Ability to read, write, speak and understand English


• Strong communication and presentation skills


• Expertise in data engineering languages such as Scala (preferred) or Java, with proficiency in Python


• Experience with BigData tools, particularly Spark


• Proficiency in building and managing ETL pipelines


• Expert-level quantitative analysis skills including interpretation of model results, consideration of causality, treatment of multicollinearity


• The ability to work in compiled, high-performance languages (e.g., Scala, Java, C++)


• Experience with relational databases


• Strong understanding of relational databases and SQL, and familiarity with NoSQL databases


• Broad experience and solid theoretical foundation on the modeling process using a


• variety of algorithmic techniques, including Machine Learning, and Graph/Network Analytics


• Data pre-processing, exploratory data analysis using a variety of techniques


• Basic understanding of data architecture, data warehouse, and data marts


• Demonstrated ability and desire to continually expand skill set, and learn from and teach others

Preferred Qualifications:

• Experience with Linux-based operating systems


• Knowledge of other relevant techniques such as text analysis and text mining


• Bachelors degree (or relevant experience) in computer science, mathematics, statistics, physics, operations research, or other quantitatively-focused fields


• Masters: 6+ Years Work Experience


• Bachelors: 8+ Years Work Experience