Senior Data Platform Engineer

Luxoft; United States; Remote; USD 90,000 - 140,000; Full time

Job summary

A leading company in data engineering is seeking a Senior Full-Stack Software Developer to join their dynamic team. You will play a pivotal role in designing and optimizing ETL processes and developing robust data pipelines using PySpark. The ideal candidate will have extensive experience in software development, particularly in data management and optimization, with a strong ability in SQL and data warehousing concepts.

Qualifications

• Minimum 7 years of IT/Data experience.
• Proficiency in PySpark and SQL for ETL development.
• Experience with Data Warehousing and Monitoring tools.

Responsibilities

• Design technical solutions and implement robust data pipelines.
• Optimize ETL processes for performance and reliability.
• Provide support in coding and QA acceptance processes.

Skills

Problem-solving

Logical reasoning

Proficiency in PySpark

SQL expertise

Data warehousing concepts

ETL processes

Version control

Data quality tools

Education

Bachelor in IT or related field

Tools

Git

DataBricks

PostgreSQL

MySQL

Oracle

Snowflake

Prometheus

Grafana

Job description

Project description

We are hiring a Senior Full-Stack Software Developer. Our client team consists of frontend and backend developers, data engineers, data scientists, QA engineers, cloud engineers, and project managers.

Responsibilities

• Participate in requirements clarification and sprint planning sessions.
• Design technical solutions and implement them, inc ETL Pipelines - Build robust data pipelines in PySpark to extract, transform, using PySpark
• Optimize ETL Processes - Enhance and tune existing ETL processes for better performance, scalability, and reliability
• Writing unit and integration tests.
• Support QA teammates in the acceptance process.
• Resolving PROD incidents as a 3rd line engineer.

SKILLS

Must have

• * Min 7 Years of experience in IT/Data* Bachelor in IT or related field.* Exceptional logical reasoning and problem-solving skills* Programming: Proficiency in PySpark for distributed computing and Python for ETL development.* SQL: Strong expertise in writing and optimizing complex SQL queries, preferably with experience in databases such as PostgreSQL, MySQL, Oracle, or Snowflake.* Data Warehousing: Experience working with data warehousing concepts and platforms, ideally DataBricks* DPE (Data Platform Engineer) practical experience in development and support* ETL Tools: Familiarity with ETL tools & processes* Data Modelling: Experience with dimensional modelling, normalization/denormalization, and schema design.* Version Control: Proficiency with version control tools like Git to manage codebases and collaborate on development.* Data Pipeline Monitoring: Familiarity with monitoring tools (e.g., Prometheus, Grafana, or custom monitoring scripts) to track pipeline performance.* Data Quality Tools: Experience implementing data validation, cleansing, and quality framework

Nice to have

Understanding of Investment Data domain.