VIRAS: Visually Impaired Robot-Assisted Shopping

Focus on the blind and visually impaired

The Institute for Lab Automation and Mechatronics (ILT) at the University of Applied Sciences of Eastern Switzerland has initiated the VIRAS (Visually Impaired Robot-Assisted Shopping) project. It includes the development of an innovative shopping aid for blind and visually impaired persons. VIRAS is a self-driving shopping cart that enables people to do their shopping in a supermarket on their own.

To clarify the need, a survey was conducted throughout Germany and Switzerland among various associations of the blind. The result clearly shows that within this group of people there is a very great interest in technical support when shopping in the supermarket. Currently, most sufferers avoid shopping for themselves. Often a third person is assigned to do this. The use of the VIRAS shopping cart meets with the approval of 87% of the respondents. With the possible provision of the shopping cart, almost 90% of visually impaired and blind people would use it to make their purchases independently.

4% of the Swiss population – with an upward trend – are affected by a visual impairment. In view of this, the project can not only contribute to inclusion but also lead to a significant improvement in the quality of life of those affected. By integrating autonomous navigation functions into the shopping cart, blind and visually impaired persons have the possibility to move independently through supermarkets. This significantly increases their independence and leads to a more self-determined life.

The Swiss Central Association for the Blind, the Swiss Federation of the Blind, the Swiss Caritas Action for the Blind and the Swiss Association for the Blind and Visually Impaired promote the development and realization of the VIRAS project through their financial and non-material support.

The VIRAS team is always in close contact with the associations and clarifies possible solutions. By involving the affected group of people, the development of the VIRAS shopping cart is realistically adapted to their corresponding needs.

Shopping challenges

The need for a shopping cart for blind and visually impaired persons results from several factors. The most important and at the same time the most demanding challenge when shopping without assistance is finding the exact products on the shelf. Particularly in the event of changes to the product range and product placement, those affected have considerable difficulty finding the desired product. Uneven lighting also makes orientation difficult. Weighing vegetables or fruit is almost impossible, as the scales cannot be operated without sufficient eyesight. Another hurdle is finding the cash register. In the supermarket, no information is provided for blind and visually impaired people on how to reach them. The VIRAS shopping cart aims to circumvent these obstacles and other limitations to provide a facilitated shopping experience for those affected.

Shopping with VIRAS

Barrier-free shopping will be possible with the VIRAS shopping cart. The autonomous system is equipped with a stereo camera and integrated speech recognition and voice output functions. Motorized, the shopping cart becomes self-propelled. In the future, this should independently guide the user to the desired products. To ensure safety, the shopping cart should avoid the people in the supermarket as well as obstacles. The stereo camera is not only used for 3D mapping of the supermarket, it also identifies the products as well as the user's hand and guides the user to the target product through acoustic instructions. The product recognition is based on AI object recognition. This innovative use eliminates the need to scan the product's barcode. This allows them to be reliably detected regardless of their orientation. If a shelf is empty, the user can be notified about it.

A shopping process for blind and visually impaired people could then look like this:

1. Localization of the shopping cart in the supermarket by acoustic signals.
2. Verbal input of the shopping list.
3. Navigation to the products with avoidance of obstacles or people.
4. Stopping the trolley when reaching the product.
5. Product recognition using AI object recognition.
6. Acoustic designation of the position of the product on the shelf.
7. Hand recognition of the user.
8. Audible indication of the position of the user's hand relative to the product.
9. Audible confirmation of the product name when the user has selected the correct product.
10. Navigation to the next product or checkout.

Visions

The VIRAS project is aiming for a number of exciting developments. In addition, the circle of users is to be expanded. In addition to blind and visually impaired people, the shopping cart is designed to assist people with limited mobility. Even people without limitations can use the VIRAS shopping cart as a navigation aid to find products quickly and efficiently. In addition, functions for inventory control are to be implemented. This includes the ability to check product availability in real time, and promotions can also be made available through the shopping cart.