Mandatory skills - MS SQL, Machine Learning and Statistical modeling, PyTorch, Python, TensorFlow
Job Details:
Primary Responsibilities:
- Design and develop ETL/ELT solutions on Azure Databricks, LakeBase and Spark
- Develop, implement, and deploy large scale data pipelines empowering machine learning algorithms, insights generation, business intelligence dashboards, reporting and new data products
- Design, build, optimize, and manage modern large-scale data pipelines ETL/ELT processing to support data integration for analytics, machine learning features and predictive modelling
- Consume data from a variety of sources (RDBMS, APIs, FTPs and other cloud storage) & formats (Excel, CSV, XML, JSON, Parquet, Unstructured)
- Write advanced / complex SQL with performance tuning and optimization
- Identify ways to improve data reliability, data integrity, system efficiency and quality
- Participate in architectural evolution of data engineering patterns, frameworks, systems, and platforms including defining best practices and standards for managing data collections and integration
- Mentor other data engineers and provide technical direction by teaching other data engineers how to leverage cloud data platforms
Required Qualifications:
- 7 + years of experience in data engineering, data integration, data modeling, data architecture, and ETL/ELT processes to provide quality data and analytics solutions
- 5 + years of experience in Python
- 2 + years of experience in Apache Spark (PySpark/Spark SQL)
- 2+ years building and deploying Cloud based solutions using - Azure Databricks with UC, Snowflake, Functions, Service Bus
- 2+ years of experience in SQL with designing complex data schemas and query performance optimization
- Experience with DevOps automation with Terraform.
- Experience with CI/CD process and tools – GitHub Actions, GIT, Artifactory, Sonar
Preferred Qualifications:
- Bachelor's degree in Computer Science, Engineering, Mathematics or related discipline
- Extensive knowledge of data architecture principles (e.g., Data Lake, Databricks Delta Lake, Data Warehousing, etc.)
- Extensive knowledge of data modelling techniques including slowly changing dimensions, aggregation, partitioning and indexing strategies
- Experience working with LLMs
- Ability to independently troubleshoot and performance tune large scale enterprise systems
- Excellent collaborator with experience working effectively with cross-functional teams such as leadership, product management and engineering, with a willingness to inspire other data engineers, data scientists and analysts
- Solid communication skills with the ability to communicate technical concepts to both technical and non-technical audiences