Computer vision technologies have attracted significant interest in precision agriculture in recent years. At the core of robotics and artificial intelligence, computer vision enables various tasks from planting to harvesting in the crop production cycle to be performed automatically and efficiently. However, the scarcity of public image datasets remains a crucial bottleneck for fast prototyping and evaluation of computer vision and machine learning algorithms for the targeted tasks. Since 2015, a number of image datasets have been established and made publicly available to alleviate the bottleneck. Despite this progress, a dedicated survey on these datasets is still lacking. To fill this gap, this paper makes the first comprehensive but not exhaustive review of the public image datasets collected under field conditions for facilitating precision agriculture, which include 15 datasets on weed control, 10 datasets on fruit detection, and 9 datasets on miscellaneous applications. We survey the main characteristics and applications of these datasets, and discuss the key considerations for creating high-quality public image datasets. This survey paper will be valuable for the research community on the selection of suitable image datasets for algorithm development and identification of where creation of new image datasets is needed to support precision agriculture.