Sr. Data Engineer

INSPYR Solutions; California (MO); Hybrid; USD 124,000 - 138,000; Full time

Benefits offered by this job

Comprehensive medical benefits

Competitive pay

401(k) retirement plan

Job summary

INSPYR Solutions seeks a Senior Data Engineer for a long-term remote project in the United States. You will design, develop, and support data pipelines, ETL/ELT processes, and data products using Spark, Python, and cloud services.

You will collaborate with product and engineering teams to gather requirements, ensure cost efficiency, scalability, and high availability, delivering robust data solutions for Live Analytics offerings.

Qualifications

• Bachelor's degree in Computer Science or related field (not required to have a degree).
• Experience building Data Lakes and Data Warehouses on cloud stack.
• 5+ years of software development experience using Agile and DevOps for large-scale data engineering.
• Expert proficiency in Spark streaming and Spark concepts.
• Expert proficiency in Python and PySpark with 4+ years of experience.
• Experience working with Kafka streams and building ingestion ETLs.
• Expert proficiency in ANSI SQL / Spark SQL with 5+ years.
• Experience with diverse data sources and formats (json, parquet, csv, psv, delta).
• Experience with AWS and S3 cloud services.
• Experience building ETL with batch and real-time streaming.
• Experience using Git and CI/CD.

Responsibilities

• Write new ETLs for real-time and feature data and reporting products.
• Gather requirements with product teams and design cost-aware solutions.
• Design and develop monitoring, ETL processes, and automated workflows.
• Ensure best practices for test coverage, architecture, high availability, and performance.
• Document systems and runbooks for maintenance and troubleshooting.
• Manage day-to-day delivery with agile stand-ups and stakeholder communication.

Skills

Spark Streaming

Python

PySpark

Kafka

SQL / Spark SQL

AWS

Data Lake / Data Warehouse

Agile / DevOps

Education

Bachelor's degree in Computer Science or related field

Tools

Databricks

Git

CI/CD

Job description

Sr. Data Engineer - Long Term Project - Remote (US)

Title: Sr. Data Engineer

Location: Remote (US)

Duration: Long Term Project

Compensation: $90-100/hr W2

Work Requirements: US Citizen, GC Holders or Authorized to Work in the U.S.

Our Client is looking for an experienced, motivated, and self-driven candidate who can contribute towards various aspects of taking the Live Analytics offering to the next level and build new data products for our clients. This involves design, development, and production support of the systems as well as working with product teams and engineering teams in shaping the future of the application.

What you will be doing

• Write new ETLs to support both real time and new features data and reporting products.
• Work with product teams to gather and understand requirements, discuss approaches, design solutions based on the requirements and create design documents for implementation keeping in mind costs, capacity, shelf-life and choosing the right tools and technologies.
• Design and develop peripheral systems and processes - monitoring and alerting services, ETL processes, automated offline workflows, automated build and deployment pipelines, systems for gathering application performance metrics, ensuring efficiency in cost.
• Ensure best practices in system development ranging from test coverage, use of enterprise standards, adherence to target architecture, high-availability and scalability, low latency, and fault-tolerance, thus ensuring delivery of high-quality systems.
• Create detailed documentation on systems as well as processes ranging from comprehensive overview of systems, runbooks for developing, monitoring and troubleshooting system issues and recommendations on future improvements.
• Managing project delivery from a day-to-day standpoint - participating in daily agile stand-ups, escalating blockers to stakeholders, ensuring timely completion of tasks & overall project & spring goals. Communicating updates to stakeholders and working with them on timelines.

What you need to know (or technical skills/competencies)

• Bachelor's degree in Computer Science or related technical field. (NOT REQUIRED TO HAVE A DEGREE)
• Experience building Data Lakes, Data Warehouse using big data and cloud stack.
• 5 years of experience on software development using Agile, dev ops that include designing, developing, and implementing large-scale data engineering solutions.
• Expert Level Proficiency in Spark streaming and spark concepts.
• Expert level proficiency in Python Programming language and PySpark with at least 4 years of experience.
• Experience working with Kafka streams and creating ingestion ETL's.
• Expert level proficiency in ANSI SQL /Spark SQL with at least 5 year's experience.
• Experience working with diverse data sources and data formats ( json, parquet, csv, psv, delta)
• Experience working with cloud services such as AWS along with cloud-based data sources(S3)
• Experience building ETL with batch and real time stream processing.
• Experience using version control system such as git and CI/CD

Nice to know / good to have

• Experience with building workflows and ETL's in the databricks.
• Experience ingesting data produced by real time systems like Kafka
• Guiding and mentoring skills - guiding and mentoring junior members - helping them understand technologies, the project, best practices and in general setting them up for success.
• Experience building and deploying ML models using ML Flow.

You (behavioural skills/competencies)

• Strong verbal communication, ability to clearly communicate ideas, work status, advocate for (or against) a particular approach or design.
• Documentation skills - ability to make the path easier for new team members / new hires / interns by organizing knowledge in Confluence, wiki, or similar.
• Ability to manage self and own time, deliver work on time, evaluate and estimate work.
• Ability to work with various teams, including tech and product, incorporate their inputs into the work being done, and manage expectations on issues & enhancements.

Our benefits package includes (EXCLUDE on perm placements)

• Comprehensive medical benefits
• Competitive pay
• 401(k) retirement plan
• ...and much more!

About INSPYR Solutions

Technology is our focus and quality is our commitment. As a national expert in delivering flexible technology and talent solutions, we strategically align industry and technical expertise with our clients' business objectives and cultural needs. Our solutions are tailored to each client and include a wide variety of professional services, project, and talent solutions. By always striving for excellence and focusing on the human aspect of our business, we work seamlessly with our talent and clients to match the right solutions to the right opportunities. Learn more about us at inspyrsolutions.com.

INSPYR Solutions provides Equal Employment Opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, sex, national origin, age, disability, or genetics. In addition to federal law requirements, INSPYR Solutions complies with applicable state and local laws governing nondiscrimination in employment in every location in which the company has facilities.

INSPYR Solutions provides Equal Employment Opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, sex, national origin, age, disability, genetics, or any other protected status. INSPYR Solutions complies with all applicable laws governing nondiscrimination in employment in every location in which the company has facilities.

Applicants requiring reasonable accommodation during the application or interview process should contact [e-mail] for assistance.

Information collected and processed through your application with INSPYR Solutions (including any job applications you choose to submit) is subject to INSPYR Solutions' Privacy Policy and INSPYR Solutions' AI and Automated Employment Decision Tool Policy: https://www.inspyrsolutions.com/policies/. By submitting an application, you are consenting to being contacted by INSPYR Solutions through phone, email, or text.

26-158636