Senior Data Engineer

Luxoft India; India; On-site; INR 1,500,000 - 2,000,000; Full time

Job summary

An international data engineering firm is seeking a Senior Engineer to develop advanced data and analytics solutions in the banking and finance sector. The role demands over 8 years of experience and proficiency in developing scalable data architectures using Python and Hadoop tools. You will drive the evolution of products while collaborating with cross-functional teams to ensure data governance and architecture efficiency. The ideal candidate will possess strong communication skills and an ability to solve complex problems, contributing significantly to business insights.

Qualifications

• 8+ years of experience as a Data Engineer.
• Experience in building Big Data products and platforms.
• Hands-on experience with Hadoop big data tools.
• Good communication skills, verbal and written.
• Ability to innovate and solve complex problems.

Responsibilities

• Drive the evolution of Data & Services products/platforms.
• Design and implement scalable data architecture and pipelines.
• Provide support for deployed data applications and models.
• Evaluate trade-offs between analytics solutions.
• Work with cross-functional teams to define the vision.

Skills

Python

PySpark

SQL

Hadoop platforms

Data governance

Cloud APIs (Azure, AWS)

NIFI

Airflow

Debugging skills

Version control

Testing and deployment using Jenkins

Education

Degree in Computer Science or equivalent

Tools

Postgres

Oracle

Hive

Impala

Spark

Job description

As a Senior Engineer in the Data Engineering & Analytics team, you will develop data & analytics solutions that sit atop vast datasets gathered on the banking & finance sector. The challenge will be to create high-performance algorithms, cutting‑edge analytical techniques and intuitive workflows that allow our users to derive insights from big data that in turn drive their businesses. You will have the opportunity to create high‑performance analytic solutions based on data sets measured in the billions of transactions and front‑end visualizations to unleash the value of big data. You will have the opportunity to develop data‑driven innovative analytical solutions and identify opportunities to support business and client needs in a quantitative manner and facilitate informed recommendations/decisions through activities like building automated data pipelines, designing data architecture/schema, performing jobs in big data cluster by using different execution engines and program languages such as Hive/Impala, Python, Kafka, PySpark etc.

Responsibilities:

• • Drive the evolution of Data & Services products/platforms with an impact‑focused on data engineering.
• • Design and implement scalable data architecture and data pipelines.
• • Solving complex problems with multi‑layered data sets, as well as optimizing existing machine learning libraries and frameworks.
• • Provide support for deployed data applications and analytical models by being a trusted advisor to Data Scientists/AI Engineers.
• • Ensure proper data governance policies are followed by implementing or validating Data Lineage, Quality checks, classification, etc.
• • Ingest, and incorporate new sources of real‑time, streaming, batch into our platform to enhance the insights we get from running tests and expand the ways and properties on which we can test and experiment with new tools to streamline the development, testing, deployment, and running of our data pipelines.
• • Evaluate trade‑offs between many analytics solutions to a problem, considering usability, technical feasibility, timelines, and differing stakeholder opinions to make a decision.
• • Break large solutions into smaller, releasable milestones to collect data and feedback from product managers, clients, and other stakeholders.
• • Evangelize releases to users, incorporating feedback, and tracking usage to inform future development.
• • Ensure proper data governance policies are followed by implementing or validating Data Lineage, Quality checks, classification, etc.
• • Work with small, cross‑functional teams to define the vision, establish team culture and processes.
• • Escalate technical errors or bugs detected in project work.

Mandatory Skills Description:

Must‑Have

• Years of Experience : 8+ years
• • Working proficiency in using Python, PySpark, SQL, Hadoop platforms to build Big Data products & platforms.
• • Experience with performance Tuning of Database Schemas, Databases, SQL, ETL Jobs, and related scripts
• • At least 6 years of relevant hands‑on experience as a Data Engineer in an individual contributor capacity.
• • Experience in working with Cloud APIs (e.g., Azure, AWS)
• • Experience in working with SQL database like Postgres, Oracle
• • Preferably with hands‑on experience with Hadoop big data tools (Hive, Impala, Spark)
• • Experience with data pipeline and workflow management tools: NIFI, Airflow.
• • Good troubleshooting and debugging skills.
• • Proficient in standard software development, such as version control, testing, and deployment using Jenkins & DAB (Databricks Asset Bundle).
• • Ability to quickly learn and implement new technologies.
• • Ability to Solve complex problems with multi‑layered data sets.
• • Ability to innovate and determine new approaches & technologies to solve business problems and generate business insights & recommendations.
• • Ability to multi‑task and strong attention to detail
• • Flexibility to work as a member of a matrix based diverse and geographically distributed project teams
• • Good communication skills - both verbal and written - and strong relationship, collaboration skills, and organizational skills.

Nice‑to‑Have Skills Description:

• • Experience in working with CI/CD.
• • Comfortable in developing shell scripts for automation.
• • Degree in Computer Science, Electrical Engineering or equivalent experience