Assistive solutions for a better shopping experience can
improve the quality of life of people, in particular also of
visually impaired shoppers. We present a system that visually
recognizes the fine-grained product classes of items on
a shopping list, in shelves images taken with a smartphone
in a grocery store. Our system consists of three components:
(a) We automatically recognize useful text on product
packaging, e.g., product name and brand, and build a
mapping of words to product classes based on the largescale
GroceryProducts dataset. When the user populates
the shopping list, we automatically infer the product class
of each entered word. (b) We perform fine-grained product
class recognition when the user is facing a shelf. We discover
discriminative patches on product packaging to differentiate
between visually similar product classes and to
increase the robustness against continuous changes in product
design. (c) We continuously improve the recognition accuracy
through active learning. Our experiments show the
robustness of the proposed method against cross-domain
challenges, and the scalability to an increasing number of
products with minimal re-training.