Project Name: Developing a visual search-based clothing recommendation system
Funding Program: KOSGEB’s R&D, Product Development, and Innovation Support Program
Grant No: 68KIB
Role: Project Manager
Project Dates: 31.03.2023 – 31.07.2024
Project Owner Organization: Zoi Data
Project Summary:
In traditional e-commerce websites, users input keywords related to the product they’re searching for into the search box. In a typical scenario, after a user enters keywords for a product, the search algorithm matches these keywords with product labels in the database and presents relevant products to the user based on the entered keywords. The user then selects the desired product from the presented options and proceeds to the ordering steps. For text-based searches to be effective, it’s important for the customer to fully understand the product and know the appropriate keywords to enter into the search box. However, this is not always the case. This problem can be solved using visual search. During a visual search, customers search for a product using images instead of keywords, as they would in a traditional search. Therefore, in the proposed project, we aim to develop a visual search engine/solution for finding similar clothing items from extensive databases.
Project Name: Cloud Based Automatic Product Tagging via Image
Funding Program: Tubitak Teydeb 1507
Grant No: 7210160
Role: Project Manager
Project Dates: 01.09.2021 – 28.02.2023
Project Owner Organization: Zoi Data
Project Summary:
Within the scope of the project, automatic product tagging is performed for the fashion industry using deep learning and image processing techniques. During this product tagging, the boundaries of fashion products are determined through segmentation on the image, and their main categories are identified through classification. Subsequently, the dominant color is determined to decide the color of the product. Products whose main group and color have been determined enter other classifiers in the label tree according to their main product group (for example, skirt length, collar type, sleeve length, etc.), and the outputs obtained from these classifiers are combined in a grammatically coherent manner to create a long and comprehensive product label. The created product label can be presented in various languages such as Turkish, English, etc. Additionally, within the scope of the project, the issue of detecting licensed products from images has been addressed to monitor copyright payments. Thus, a product containing a licensed image can automatically be tagged with its corresponding license. This comprehensive product tagging system not only reduces the workload of e-commerce sites operating in the fashion industry and facilitates product tracking and control, but also enables the production of search engine-optimized labels.
Project Name: Automatic grading of student music exercise performance for use in online music education system design
Funding Program: TÜBİTAK ARDEB 1001
Grant No: 121E198
Role: Researcher
Project Dates: 15.12.2021 – 15.06.2023
Project Owner Organization: Izmir Democracy University
Project Summary:
We have entered a new era where education is largely conducted online. The number of music students following online courses is growing rapidly as well as the resources made available. In parallel to that, the human resource requirement for the grading of a large number of music students’ musical performances is also increasing. For this reason, the design of automatic systems for grading student performances becomes a necessity. Our project addresses the problem of “automatic assessment of student music performances“, for two types of musical exercises: melody repetition/imitation and rhythm repetition/imitation. In addition to this, we also studied the design and implementation of a system that automatically generates exercises at a targeted difficulty level.
Our main research goal is the implementation of a system that can automatically assess a student performance as accurately as a music instructor. Melody and rhythm dimensions are highly important in musical exercises. Hence, the results of this project will have large use in developing technologies supporting online education. For both of the tasks, we collected and publicly shared data sets that consist of recordings from real auditions for conservatory entrance exams in Turkey. All recordings are annotated/graded by three experts.
In the first study, we designed and implemented a system that automatically grades student vocal performances repeating melodic patterns. The system takes in a student performance recording and the reference piano recording (of the pattern repeated) as input and produces a grade as output, based on the melodic similarity/distance of these two recordings. Four sequential blocks are included in the system architecture. The first block estimates the fundamental frequency series and chroma feature matrices for the reference and performance recordings. The second block matches these series with different lengths in time using a dynamic time warping algorithm. In the third block, the statistical distribution of the distances between the aligned representations is computed. The fourth block is a machine learning model that takes in the distance distribution and automatically estimates a grade. The machine learning model is trained using a supervised learning approach. We present an analysis of inter and intra expert agreement and the machine learning experiment results for the automatic system proposed.
In the second study, we designed and implemented an automatic performance assessment system for a student’s rhythmic pattern imitation. The system compares the student’s performance recording with a teacher’s reference recording and assigns a grade between 1 and 4. The automatic assessment (grade assignment) task is considered as a regression problem and two approaches are tested and compared. The first approach applies classical regression methods to distance features. The second approach applies metric learning using a Siamese neural network to directly learn the most efficient feature representations from the onset positions. The best performance is achieved using the Siamese network.
In addition to the automatic assessment problem, we also studied the problem of automatic exercise creation at a target difficulty level. This technology is especially needed in artificial intelligence-driven personalized education and in situations where different exercises at similar difficulty levels are required (e.g. conservatory entrance exams). Due to lack of data, the potential for applying data-driven techniques is very limited for this problem. Hence, we implemented rule-based systems for automatic exercise creation of melodic pattern exercises and rhythmic pattern exercises. Difficulty level analysis of the generated patterns are studied using expert annotations on the generated patterns as well as student performances of these patterns.