A method to identify the picking point of the whole mature litchi fruit and its matching method based on binocular stereo vision under the structural environment is proposed in this paper. The mature litchi clusters in the extracted image were segmented by using the improved K-means clustering algorithm. The region of interest (ROI) of the Litchi stems is set based on the characteristics of the center of the litchi fruit region and the minimum circumscribed rectangle. Corner extraction from the main fruit bearing branch of litchi was carried out by Harris algorithm. The customized function of circle distance is used to determine the optimal corner, and then cumulative probability Hough linear detection of the Litchi stems is used to determine the straight line of the picking point. Finally, the two-dimensional pixel coordinates of the picking point are determined. Then the three-dimensional coordinates of litchi picking points are obtained by SGBM stereo matching algorithm. The experimental results show that the matching success rate of picking points can reach 86.63%. It provides the foundation for litchi harvesting robot to pick litchi fruit successful.

Keywords: k-means clustering algorithm; litchi harvesting robot, harris algorithm; hough, SGBM algorithm, ROI

In recent years, fruit and vegetable harvesting robots have become a research hotspot. Among them,¹ South China Agricultural University of China on litchi harvesting robot done a lot of research and developed a litchi picking robot which mainly divided into visual systems and execution systems from the structural point of view.^3,4 The visual system captures the scene in the real world through two industrial cameras just like the human eye, and then uses the developed software algorithm to identify the litchi fruit and its picking point. Finally, the three-dimensional coordinates of the point are calculated on the basis of the picking point in the matching left and right images.^3,4 The execution system picks up the litchi fruit by controlling the end-effector after acquiring the three-dimensional coordinates of the picking point from the visual system.⁵ It can be seen that the identification of litchi fruit and the matching of picking points by binocular vision system are the primary steps to pick litchi fruit successful.

The shape of litchi is a string of fruit, therefore it cannot be directly stereo matched by using the centroid or edge characteristics of circular fruit.^6,7 Based on the characteristics of the hue information of the litchi color image, the three-dimensional matching method based on the hue space is adopted to study the litchi picking point. Because the color information of the scene in the color image of litchi has the same characteristics, the method of stereo matching for the picking point of litchi was carried out by using the method of region stereo matching based on hue space. The method is not only computationally intensive, but also has good effect only on the calculation and matching research of picking point whose parent branch is regular and linear.⁸ A scheme of combined algorithms of Harris and improved SIFT to compute picking point of litchi and achieve its stereo matching was proposed. Corner extraction from the branch of litchi was carried out by Harris method.

The two-dimensional pixel coordinates of the picking point are calculated by combining with the centroid of litchi fruit area identified and the minimum circumscribed rectangle. Then, the computed picking point was described with a characteristic vector of SIFT, and its binocular stereo matching based on cosine distance similarity of SIFT was also proposed. In this paper, combined with K-means clustering algorithm, cumulative probability Hough linear detection, SGBM stereo matching algorithm and the usually used Harris algorithm which can quickly and effectively extract a large number of useful feature points in the region of rich texture,^9-11 a method to calculate and match picking points of litchi is proposed to solve the calculation and stereo matching of picking points on the litchi whose parent branch have complex shape.

Calculation of the picking points of litchi image

Identification of litchi fruit: After the S component of HSV color space is extracted, the S-component graph is segmented by improved K-means clustering algorithm which selects the gray value of the two highest peaks of the S component histogram to initialize the cluster centre in order to reduce the iteration times and time of the algorithm.^12,13 After the segmentation is completed, the function of area threshold is used to remove the small area background, and the voids are filled to obtain the binary image of litchi fruit as shown in Figure 1A.

The solution of the external rectangle of litchi image: The maximum connected domain is extracted from the identified litchi fruit image. The litchi outline image area is set to 1 (white), the remaining area pixel value is set to 0 (black), (0 for the background, 1 for the litchi fruit target). In order to solve the coordinates of litchi centroid,¹⁴ according to the definition of image centroid, the coordinates of litchi centroid can be solved by the following formula:

$\{\begin{array}{c}\overline{x}={\displaystyle {\sum }_{(x,y)ϵR}x}/{\displaystyle {\sum }_{(x,y)ϵR}f(x,y)}\\ \overline{y}={\displaystyle {\sum }_{(x,y)ϵR}y}/{\displaystyle {\sum }_{(x,y)ϵR}f(x,y)}\end{array}$ . (1)

Where $(\text{<mover accent="true"> x <mo>¯</mo> </mover>},\text{<mover accent="true"> y <mo>¯</mo> </mover>})$ is the coordinates of Centroid (x, y) is the coordinates of the pixel, $\text{f}\left(\text{x},\text{y}\right)$ is the pixel value of the image at the point (x, y).

The coordinates of litchi centroid are shown in Figure 1B. After obtaining the coordinates of centroid, the minimum circumscribed rectangle of the maximum connected domain is obtained, where the width of the rectangle is w and the height is h. In order to reduce the interference of the environment to locate the picking point,¹⁵ ROI of the branch was set according to the minimum circumscribed rectangle of the litchi centroid and the litchi profile where the width W of ROI was 0.4w and the height H was 0.3h. As shown in Figure 1C.

Figure 1 Obtaining the image of ROI.

The obtaining of calculation picking points: First, a function $C_i\left(x_i,y_i\right)$ named circle distance should be defined, where $C_i\left(x_i,y_i\right)$ is the number of all points that satisfy the condition $\left\{\left(x，\text{y}\right)|\sqrt{{\left(x-x_i\right)}^2+{\left(y-y_i\right)}^2}\le R\right\}$ , $\left(x_i,y_i\right)$ is the coordinates of each Harris corner detected, $\left(\text{x,}y\right)$ is the pixel coordinate whose pixel value is not zero, R is the threshold radius should be set manually.⁸ Using the Harris algorithm, combined with the above function, the process of calculation picking point is proposed in this paper as follows: The Harris corners of the litchi branch were detected in the ROI of the branch of litchi, whose coordinates is $\left({\text{x}}_{\text{i}},{\text{y}}_{\text{i}}\right)$ , the effect is shown in Figure 2A. According to the function definition to calculate the function value of each corners, and using the idea of the extreme value algorithm to obtain the maximum value of the function Ci whose coordinates is set to $\left({\text{x}}_{\text{a}}{\text{,y}}_{\text{a}}\right)$ , the effect is shown in Figure 2B.¹⁶ Then, the litchi branch was detected by using cumulative probability Hough linear detection in the ROI only if the ordinate of the starting point and ending points of the straight line is less than ya, the effect of the test is shown in Figure 2C. And the maximum length of each straight line is obtained. The slope of the longest straight line is K. Finally, $\left({\text{x}}_{\text{a}}{\text{,y}}_{\text{a}}\right)$ as a starting point, the slope of K for a straight line, the picking point is on the straight line. Binarization and Canny edge detection for the litchi branch to extraction the intersection point of the edge of the litchi branch and the line. The intersection point which the ordinate is the smallest is the picking point we need. The result of the picking point is shown in Figure 2D.

Figure 2 Extracting process of the picking point of litchi.

The stereo matching of picking points based on SGBM

The principle of SGBM algorithm: The core step of the SGBM algorithm is as follows: select matching primitives; construct the cost energy and function based on scan lines of multiple directions;¹⁷ obtain the optimal solution of energy cost and function. The basic process of the algorithm is:

1. Matching pixel-by-pixel. The function $d({\text{x}}_{\text{i}},{\text{y}}_{\text{i}},{\text{I}}_{\text{L}},{\text{I}}_{\text{R}})$ defines the possibility of the two points that are the same.

$d({\text{x}}_{\text{i}},{\text{y}}_{\text{i}},{\text{I}}_{\text{L}},{\text{I}}_{\text{R}})={\min }_{{\text{y}}_{\text{i}}-0.5\le \text{y}\le {\text{y}}_{\text{i}}+0.5}\left|{\text{I}}_{\text{L}}\left({\text{x}}_{\text{i}}\right)-{\text{I}}_{\text{R}}\left({\text{y}}_{\text{i}}\right)\right|$ . (2)

Where xi and yi are the image points on the left and right scanning lines. The gray scale at the point x on the left scan line is defined as IL(x); IR is obtained by the sample point linear interpolation of the right scanning line.

2. Using one-dimensional constraint to approximate two-dimensional constraint. Eight paths are set at intervals of 45° near the P. Calculate the minimum cost path Lr (p, d) through the 8 paths in order to approximate the two-dimensional constraint matching calculation.
3. The calculation of parallax. The calculation of disparity is obtained by the way of iterations. First, using the random parallax image to correct the right image and then match to generate a new parallax image.¹⁸ It is also possible to use a hierarchical calculation method to calculate the parallax image as an initial value in the case of a half resolution and then using the parallax image obtained to calculate recursively.
4. Delete the mistaken matching. When the matching of the same name point of left and right image is completed, using the point in the right image to match the same name point in the left image. If the two pairs of parallax are different,¹⁹ they are considered invalid.

The stereo matching of picking points: SGBM stereo matching is essentially an improved dynamic programming method, which improves the accuracy of matching results while preserving the high efficiency of dynamic programming. The SGBM algorithm is insensitive to the influence of illumination and has the strong robustness to noise. In this paper, SGBM algorithm is used to solve the three-dimensional coordinates of litchi picking point，which makes a good foundation for Litchi harvesting robot to work in the field. The stereo matching step of picking point is as follows Figure 3.

Figure 3 The stereo matching step of picking point.

The acquisition of matching images is based on a laboratory environment. The model of two cameras are the Chinese Microvision EM200C, the camera is installed and debugged as a relatively parallel state. Using Windows7+VS2013+OPENCV3.0 as the test platform.^18,19 It is need to calibrate the left and right camera calibration before taking the picture in order to obtain the camera parameters and distortion parameters, and then stereo calibration should be done to get the rotation matrix and translation matrix between the two cameras. Finally, stereo rectify is to get a complete row alignment image and make stereo matching more simple and reliable. Figure 4A & 4B are imaging situation in the left and right camera of the same litchi string.

Figure 4 The image in left and right camera and the image of picking point.

Select Figure 4A to calculate the picking point. Calculate the minimum circumscribed rectangle of the litchi fruit region and draw an external rectangle, the effect is shown in Figure 1B. Harris algorithm was used to extract the corners of the litchi stems. A total of 79 corners were detected. The effect is shown in Figure 2A. Count the circle distance function value of each corner, and obtain the corner (which is a corner of Harris) with the largest value of the circle distance function is $C_9\left(x_{9,}{y}_9\right)$ whose coordinates is (337, 146), the effect is shown in Figure 2B, using cumulative probability Hough linear detection to detect the litchi stems, finally obtain the two-dimensional pixel coordinates of litchi picking point (352, 119), the effect of the picking point in the original image is shown in Figure 4C. The calculation of picking points and SGBM stereo matching for the collected 76 images. The rate of matching is up to 86.63%, Which can meet the precision requirement of visual positioning of litchi harvesting robot.

This paper presents a method to Identify the litchi fruit based on K-means clustering algorithm, to detect the line based on cumulative probability Hough linear detection and to detect the corner based on Harris algorithm in the ROI of litchi stems, finally, the two-dimensional pixel coordinates of the picking point is obtained. Then using the SGBM stereo matching to match the picking point of litchi to obtain the three - dimensional coordinates of litchi picking point. The experiment result shows that the recognition success rate of litchi picking point was 86.63%.

This work was financially supported by the National Natural Science Foundation of China (No. 31571568), Supported by Science and Technology Planning Project of Guangdong Province.(No.2015A020209111), science and technology project of Huizhou city (No.2014B040008006).

The author declares no conflict of interest.

1. Chenglin Wang, Yunchao Tang, Xiangjun Zou, et al. A robust fruit image segmentation algorithm against varying illumination for vision system of fruit harvesting robot. Optik-International Journal for Light and Electron Optics. 2017;131:626–631.
2.  Xiangjun Zou, Haixin Zou, Jun Lu. Virtual manipulator-based binocular stereo vision positioning system and errors modeling. Machine Vision and Applications. 2012;23(1):43–63.
3. Xiangjun Zou, Min Ye, Chengyu Luo, et al. Fault-tolerant design of a limited universal fruit-picking end-effector based on visoin positioning error. Applied Engineering in Agriculture. 2016;32(1):5–18.
4. Chen Yan, Zou Xiangjun, Xu Dongfeng. Mechanism design and kinematics simulation of litchi harvesting robot. Mechanical Design. 2010;27(05):31–34.
5. Chenglin Wang, Xiangjun Zou, Yunchao Tang, et al. Localisation of litchi in an unstructured environment using binocular stereo vision. Biosystems Engineering. 2016;145:39–51.
6. Xiong Juntao, Zou Xiangjun, Chen Lijuan, et al. Fruit recognition and positioning technology of litchi picking manipulator. Journal of JIangsu University (Natural Science Eidt. 2012;33(1):1–5.
7. Xiong Juntao. Study on vision position of litchi picking manipulator on complex on complex environment. Guangzhou, China: South China Agricultural University; 2012.
8. Guo Aixia, Xiong Juntao, Xiao Deqin, et al. Computation of picking point of litchi and its binocular stereo matching based on combined algorithms of Harris and SIFT. Transactions of the Chinese Society for Agricultural Machinery. 2015;46(12):11–17.
9.  Krystian Mikolajczyk, Cordelia Schmid. Scale & affine invariant interest point detectors. International Journal of Computer Vision. 2014;60(1):63–86.
10. Lufeng Luo, Yunchao Tang, Xiangjun Zou, et al. Vision-based extraction of spatial information in grape clusters for harvesting robots. Biosystems Engineering. 2016;151:90–104.
11. Liu Jinxin. Research of Camera Calibration and Stereo Matching in Binocular Vision. Guangzhou, China: Guangdong University of Technology; 2012.
12. Xiang Juntao, Zou Xiangjun, Peng Hongxing, et al. Real-time identification and picking point localization of disturbance citrus picking. Journal of Agricultural Mechanization. 2014;45(8):38–43.
13. Luo L, Tang Y, Zou X, et al. Robust grape clusters detection in a vineyard by 2 combining Adaboost framework and multiple color 3 components. Sensors (Basel). 2016;16(12):E2098.
14. Rafael C Gonzalez, Richard E Woods. Digital Image Processing. 3rd ed. Beijing, China: Publish House of Ecectronics Industry; 2012. p. 542–543.
15. Luo Lufeng, Zou Xiangjun, Juntao Xiong, et al. Automatic positioning for picking point of grape picking robot in natural environment. Transactions of the Chinese Society of Agricultural Engineering. 2015;31(2):14–21.
16. Matas J, Galambos C, Kittler J. Robust detection of lines using the progressive probabilistic Hough transform. Computer Vision and Image Understanding. 2000;78(1):119–137.
17. Hirschmuller H. Accurate and Effcient Stereo Processing by Semi-Global Matching and Mutual Information. USA: IEEE San Diego; 2005.
18. Zhou Fang. Research on Stereo Matching in Binocular Vision and its Implementation. DaLian, China: Dalian University of Technology; 2003.
19. Yan Ruomei. Research on the Remote Sensing Stereo Pair Compression Algorithm based on area. Xian, China: Xidian University; 2010.