Senior Data Platform Engineer

INSPYR Solutions

San Jose (CA)

On-site

USD 110,000 - 130,000

Full time

14 days+

Get more replies from employers

Send a job-specific resume in minutes.

Job summary

A leading technology solutions provider is seeking a Senior Data Platform Engineer for a remote position. The role demands expertise in Python/PySpark and the ability to enhance Data Platform Services. Responsibilities include designing onboarding capabilities and ensuring data integrity and compliance. Ideal candidates will have a strong understanding of Data Lakes and cloud services, particularly AWS.

Qualifications

Strong Python/PySpark coding skills.
Good understanding of Data Lakes and Warehouses.
Familiar with cloud services, preferably AWS.

Responsibilities

Enhance performance and reliability of Data Platform Services.
Design self-servicing onboarding and automation capabilities.
Develop scalable data platform solutions.

Skills

Python/PySpark expertise

Version control systems

Data Lakes and Warehouses

Data ingestion services

ANSI SQL and Spark SQL

Workflow automation tools

Databricks or Hadoop

Cloud services (AWS, GCP, Azure)

Streaming and batch data sources

Visualization experience

Agile methodologies

Job description

Overview

Senior Data Platform Engineer - Direct-Hire/FTE - Remote (US)

This is a hands-on Senior Data Platform Engineer role that will require strong Python/PySpark coding skills. This is NOT an ETL/ELT development role.

Responsibilities

Contribute to the enhancements and continuous improvement of performance and reliability of the existing Data Platform Services to meet the requirements from Data Engineering/Product team
Design and build self servicing onboarding and automation capabilities in Core Data Platform services in a scalable, reliable and secure way.
Design and build platform capabilities to support onboarding and operation effectiveness
Design and develop scalable data platform and integration solutions for various data sources leveraging Databricks Unified Platform
Ensure data quality and integrity across all data systems.
Ensure data security and compliance with relevant regulations.
Implement best practices for data management and governance.
Develop and maintain documentation for data systems and processes.
Monitor and troubleshoot data platform issues.
Participate in on-call rotations/Pagerduty for data platform support.

Technical skills / competencies

Good understanding of Data Lakes, Data Warehouses is MUST to have
Software development, coding expertise in the Data engineering field using Python/PySpark is MUST to have
Hands on experience using version control systems such as Git and CI/CD workflows
Design and Develop data ingestion services that are highly performant, reliable and scalable
Decent expertise using ANSI SQL and Spark SQL is key to have
Workflow automation, orchestration using Airflow or equivalent tools stack
Hands on working knowledge on at least one of these: Databricks, Hadoop and related stacks
Working experience in at least one cloud service (AWS, GCP, or Azure), preferably AWS
Streaming (Kafka) and batch based data sources
Diverse data sources and data formats (xml, json, yaml, parquet, avro, delta) and respective use cases
Any visualization experience is an advantage
An understanding of time zones, geo, data formats, and data types across storage systems is beneficial
Agile development methodologies using the Atlassian suite: Jira, Confluence
Collaborate with cross-functional teams to deliver data solutions and provide technical support and guidance
Stay up-to-date with the latest data engineering technologies and trends

About INSPYR Solutions

Technology is our focus and quality is our commitment. As a national expert in delivering flexible technology and talent solutions, we strategically align industry and technical expertise with our clients\' business objectives and cultural needs. Our solutions are tailored to each client and include a wide variety of professional services, project, and talent solutions. By always striving for excellence and focusing on the human aspect of our business, we work seamlessly with our talent and clients to match the right solutions to the right opportunities. Learn more about us at inspyrsolutions.com.

INSPYR Solutions provides Equal Employment Opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, sex, national origin, age, disability, or genetics. In addition to federal law requirements, INSPYR Solutions complies with applicable state and local laws governing nondiscrimination in employment in every location in which the company has facilities.

Information collected and processed through your application with INSPYR Solutions (including any job applications you choose to submit) is subject to INSPYR Solutions’ Privacy Policy and INSPYR Solutions’ AI and Automated Employment Decision Tool Policy: https://www.inspyrsolutions.com/policies/. By submitting an application, you are consenting to being contacted by INSPYR Solutions through phone, email, or text.

Seniority level