Data Engineer (3-5 Yrs)

Job Description

Data Pipeline Architecture and Development:

1. Architect and optimize scalable data storage solutions, including data lakes, warehouses, and NoSQL databases, supporting large-scale analytics.
2. Design and maintain efficient data pipelines using technologies such as Apache Spark, Kafka, Fabric Data Factory, and Airflow, based on cross-functional team requirements.

Data Integration and ETL:

1. Develop robust ETL processes for reliable data ingestion, utilizing tools like SSIS, ADF, and custom Python scripts to ensure data quality and streamline workflows.
2. Optimize ETL performance through techniques like partitioning and parallel processing.

Data Modeling and Schema Design:

1. Define and implement data models and schemas for structured and semi-structured sources, ensuring consistency and efficiency while collaborating with data teams to optimize performance.

Data Governance, Security, and Compliance:

1. Establish and enforce data governance policies, ensuring data quality, security, and compliance with regulations, using tools like Microsoft SQL Server.
2. Implement access controls, encryption, and auditing to protect sensitive data and collaborate with IT to address vulnerabilities.

Infrastructure Management and Optimization:

1. Manage and optimize cloud and on-prem infrastructure for data processing, monitor system performance, and implement disaster recovery enhancements.
2. Leverage automation for provisioning, configuration, and deployment to improve operational efficiency.

Team Leadership and Mentorship:

1. Provide technical leadership, mentoring team members in best practices and cloud technologies, while aligning data engineering initiatives with strategic goals.

Skills Required

1. Bachelor's degree or higher in Software Engineering, Computer Science, Engineering, or a related field.
2. 3-5 years of experience in data engineering, with a proven history of designing and implementing complex data infrastructure.
3. Proficient in Python, Scala, or Java, with experience in scalable, distributed systems.
4. Strong knowledge of cloud computing platforms and related services like AWS Glue, Azure Data Factory, or Google Dataflow.
5. Expertise in data modeling, schema design, and SQL query optimization for both relational and NoSQL databases.
6. Excellent communication and leadership skills, with the ability to collaborate effectively with cross-functional teams and stakeholders.