- Experience: Minimum of 2-3 years relevant experience
- Technical Requirement: Strong expertise in AWS
- Design, develop, and maintain scalable data pipelines for data ingestion, transformation, and storage.
- Build and manage ETL/ELT workflows to integrate data from multiple internal and external systems.
- Develop and maintain data warehouses, data lakes, and hybrid storage solutions.
- Monitor and optimize pipeline performance, reliability, and cost efficiency.
Data Quality & Reliability
- Implement data quality checks, validations, and monitoring mechanisms.
- Identify and resolve data issues, anomalies, and pipeline failures.
- Ensure data accuracy, completeness, and consistency across platforms.
- Support continuous improvement of data reliability and availability.
- Collaborate with data scientists, analysts, and business users to understand data requirements.
- Deliver curated, analytics-ready datasets to support reporting, dashboards, and advanced analytics.
- Partner with Data Architects and Enterprise Architecture teams to ensure alignment with architectural standards.
Data Security & Governance Support
- Implement data engineering solutions that comply with data governance, security, and privacy policies.
- Apply access controls, data masking, and other protective measures where required.
- Support metadata management, documentation, and lineage tracking.
Documentation & Automation
- Create and maintain documentation such as data flow diagrams, pipeline designs, and data dictionaries.
- Automate data processing and operational tasks to reduce manual intervention and errors.
- Document operational procedures and troubleshooting guidelines.
Technology Evaluation & Continuous Improvement
- Evaluate and adopt new data tools, platforms, and engineering practices.
- Stay current with trends in big data, cloud platforms, and data engineering frameworks.
- Recommend improvements to data infrastructure and processing efficiency.