Click on the above “Beginner Learning Vision“, select to add “Star” or “Top“
Important content delivered at the first time
This solution will build a machine vision-based assembly line detection platform for mobile barcodes. The product is placed on the assembly line, and a high-speed industrial camera captures it one by one and transmits it to the computer for a series of processing, ultimately detecting whether the barcode is qualified and sorting out the unqualified barcodes, completing the function of automatic detection of mobile barcode quality.
Detection objects: Mobile product barcodes (including product codes, SN codes, IMEI codes)
Applicable range: Automatic production lines on assembly lines
Detection speed: 1 item/2 seconds (20 cm/second)
(Mobile product barcode)
2.1 Overall Structure Schematic
(Overall structure schematic)
It includes: assembly line conveying device, lighting device, high-speed camera image acquisition device, computer image processing part.
2.2 Hardware Design Plan
2.2.1 Conveyor Mechanism Design
In this solution, the quality of the conveyed objects is small, so there is no need to consider the load capacity of the conveyor belt too much, and it does not involve product loading and unloading. Therefore, a common belt conveyor structure is adopted, driven by a stepper motor to synchronize the belt with rollers, and a microcontroller is used to control the stepper motor for simple motion control to control the conveyor belt’s transmission switch and speed. Meanwhile, to ensure a clear distinction between the mobile product barcode and the background during image acquisition, the color of the conveyor belt is deep black.
(Belt conveyor structure diagram)
2.2.2 Industrial Camera Selection
The imaging system is the most important part of this solution. In this solution, the detection object is a barcode, which is relatively large and has obvious features, so a very high resolution is not required for the camera. However, since this solution is a real-time processing design, the speed of image acquisition and processing must be higher than the detection rate of the object, meaning the camera’s maximum frame rate must be high; additionally, since the imaging object is moving, to avoid issues such as blurring in the captured images, a camera with a global electronic shutter should be used; furthermore, the camera must support external trigger shutter and quick data transmission with the computer. Therefore, the industrial camera should have the following characteristics:
(1) High maximum frame rate
(2) Global electronic shutter frame exposure method
(3) Support for external trigger shutter
(4) Interface for quick data transmission with the computer
Considering these factors, this solution chooses the acA800-200gc/gm model industrial camera manufactured by Basler in Germany, with the following parameters:
(Industrial camera parameters)
This industrial camera meets the design requirements of this solution.
2.2.3 Photoelectric Sensor Selection
This solution uses a sensor-triggered method to control the camera shutter. In the design, the conveyor belt is black, and the barcode background is white, which have a significant difference in reflection ability. Therefore, a diffuse photoelectric sensor can effectively detect whether the product barcode has entered the camera’s shooting range. The photoelectric sensor is placed vertically above the conveyor belt and close to the camera. When the barcode reaches below the sensor, the reflected light changes, triggering the switch state, which indirectly controls the camera shutter, thus achieving the purpose of automatically capturing the barcode image.
2.2.4 Light Source Selection and Control
Common machine vision light sources include LED light sources, halogen lamps (fiber optic sources), and high-frequency fluorescent lamps. LED light sources are the most commonly used due to their long lifespan, and because LED light sources are composed of multiple LEDs arranged together, they can be designed into complex structures to achieve different light source irradiation angles. Since the barcode being detected in this solution is a smooth plane, a diffuse ring LED light source is used to illuminate it, which can effectively distinguish the product barcode from the black background of the conveyor belt. It is important to note that the height and angle of the light source must be properly adjusted until the images captured by the camera can perfectly distinguish the barcode from the background without any reflections.
(Diffuse ring illumination)
2.3 Software Platform Environment
The detection speed of this solution is about one item every two seconds, so the computer used for image processing can be a general office computer with a CPU frequency higher than 3GHz. MATLAB is used as the main image processing software.
2.3.1 Barcode Image Preprocessing
The barcode image obtained from the data interface by the computer is the unprocessed image directly captured by the high-speed camera. Due to the characteristics of CMOS and placement factors, there may be noise, tilt, and other issues, and the obtained image is a color image. Therefore, before detecting the barcode, the original image should be preprocessed. Preprocessing includes: filtering and denoising, binarization, and tilt correction.
2.3.2 Barcode Image Filtering and Denoising
The CMOS camera used in this solution has a high integration level, which leads to component interference noise. Therefore, it is necessary to remove random noise interference from these images. Experiments show that the noise in the images is mostly salt-and-pepper noise, which often appears as isolated pixel points with significantly different pixel grayscale compared to adjacent pixels, thus having a higher spectrum. To address the type of noise, median filtering is used, which has a good suppression effect on impulse interference and salt-and-pepper noise while preserving image edges well. From the characteristics of the barcode image, it consists of longitudinally arranged black and white bars. We filter the noisy image using templates of different sizes and compare their effects in removing noise interference. In MATLAB, the filter2 function can be used to call the created filter, and it has been verified that the 3×3 square filter has the best effect, so this solution uses a 3×3 median filter.
(Image with salt-and-pepper noise)
(Image after 3×3 median filtering)
2.3.3 Barcode Image Binarization
To facilitate subsequent processing of the image, it is necessary to perform binarization. In this solution, the product barcode is a rectangular sticker with a white bottom and a black conveyor belt background. The image obtained directly from the camera is a color image. After gray scaling the obtained image, binarization can be performed to facilitate the segmentation of the product barcode sticker. The MATLAB rgb2gray() function can be used to convert the color image to a grayscale image, and then the im2bw() function can easily perform binarization, with a threshold of 0.67, resulting in a better binarized image. It can be seen that the edges of the sticker are clearer after binarization.
(Original image) (Binarized image)
2.3.4 Barcode Image Tilt Detection and Correction
Since the position of the object on the conveyor belt is not fixed, the barcode area in the images captured by the camera may appear tilted. To facilitate subsequent barcode positioning and segmentation extraction, it is necessary to straighten the tilted barcode image.
Observing the binarized barcode image, it can be found that the paper area has a clear distinction from the background, while the significant feature of the tilted barcode image is that the paper area in its binary image is rotated at a certain angle, and the rotation angle cannot be too large. Therefore, to correct the tilted barcode image, the tilt angle must first be detected,
and then use the angle value to perform an inverse rotation to obtain a horizontally oriented barcode image containing the barcode. Considering the randomness of the image position and the complexity of the calculations, this solution decides to use the Hough transform method to detect the tilt angle of the barcode image. The implementation method is to first use the edge() function in MATLAB to obtain the edges of the binary image, and then use the relevant functions of Hough transform: hough(), houghpeaks(), houghlines() to obtain the starting and ending points of the longest edge. By using the slope formula, the tilt angle of the barcode image can be calculated, and finally, the imrotate() function is used to correct the tilt of the image. Tests have shown good results.
(Original tilted image) (Corrected image)
2.4 Barcode Segmentation
Since the barcode characters and other information on the sticker of the same mobile product are fixed in position, after segmenting the paper area in the image, it is easy to physically segment the barcode part by its fixed position.
2.4.1 Barcode Sticker Segmentation
In the binarized image, the sticker can be clearly distinguished from the background. After processing with the bwareaopen() function, small area points can be eliminated, resulting in a clearer binary image.
(Image processed by bwareaopen function)
After obtaining this image, the image is detected line by line from left to right to get the coordinates of the first point where the pixel changes from black to white, which is the first coordinate of the edge of the paper area, denoted as P1; then using the same method, the coordinates detected from right to left, top to bottom, and bottom to top are denoted as P2, P3, and P4 respectively. These four coordinates determine the position of the paper area. Finally, the imcrop() function is called to segment the upright image of the paper area according to these four coordinates.
(Segmented barcode image’s paper area)
2.4.2 Barcode Segmentation
As mentioned earlier, for the same mobile product, the positions of the barcode characters on the sticker are fixed. By calculating the position of the barcode on the sticker for the same mobile model, the imcrop() function can be called again to segment any part of the sticker.
(Segmented barcode)
2.5 Barcode Recognition and Detection
The processed barcode image ultimately results in a binary image consisting only of black and white bars, where the color information has already been defined during the threshold transformation, using 255 and 0 to represent white and black, respectively. The next task is to decode the numerical or alphabetical information contained in this image, that is, barcode reading.
2.5.1 Barcode Decoding Recognition Scheme
There are many algorithms for barcode recognition, commonly used ones include:
① Width measurement method: In the image-based decoding process, the width measurement does not use the traditional pulse measurement method, but determines the actual width of each bar or space by recording the number of pixels contained in it, thus determining the information represented by the entire barcode symbol.
② Average value method: Measure the entire width of the barcode symbol image from the start symbol to the end symbol, then divide by the standard width of 95 to obtain the pixel column width of the unit module, and then measure the actual width of each bar and space, calculating its width in units of the unit width.
③ Similar edge distance measurement method: The design idea of this method is to determine the logical value of characters by measuring the distance between similar edges of adjacent elements in the symbol, rather than by measuring the actual width of each element.
The first two methods have very high requirements for the barcode image because they measure the actual width of each element symbol and then obtain the represented code value through a lookup table. If there is any deviation between the actual measurement value and the standard value, correct decoding cannot be achieved. The third method effectively solves this problem; therefore, this solution adopts the similar edge distance measurement method to achieve the decoding function. Theoretically, the logical value of the barcode character should be judged by the actual width of the barcode, while the design idea of the similar edge distance method is to determine the logical value of characters by measuring the distance between similar edges in the symbol rather than by the actual width of the elements. The advantages of this method are that even if the barcode quality has defects, resulting in a significant deviation between the actual measurement value and the theoretical value that the barcode should have, it can still be correctly interpreted based on the distance between similar edges.
(Barcode character width schematic)
In the figure, C1, C2, C3, C4 represent the widths of four adjacent bars and spaces in each character, T represents the width of a character, and T1, T2 represent the distances between similar edges. By normalizing these values and looking them up in a table, the encoding values can be obtained.
(EAN-13 barcode characters and normalized values)
Using MATLAB to implement this method, the key is to detect the distance between similar edges. The barcode is scanned horizontally from left to right, using the points where the grayscale value changes as the coordinates for marking bars and spaces, thus calculating the widths of each bar and space. By calculating n=T/7, T=1C+2C+3C+4C, the width of the unit module can be obtained. The widths of each bar and space divided by the width of the unit module yield the normalized widths of the bars and spaces. For convenience in decoding, we remove the starting symbol of 3 bars and spaces, the middle separator of 5 bars and spaces, and the terminating symbol of 3 bars and spaces, and decode the remaining 48 bars and spaces in groups of four. The distances t1 and t2 between similar edges are calculated to obtain the normalized (T1, T2). Finally, looking up the table yields the barcode characters. Then, based on the odd and even nature of the six data symbols on the left, the prefix symbol can be calculated. The following is the decoding effect:
(Barcode recognition result)
Other code types (SN code, IMEI code) operate on similar principles.
2.5.2 Barcode Character Recognition
After obtaining the data from barcode recognition, it is necessary to compare it with the characters below the barcode to verify the accuracy of the barcode. There are many algorithms for character recognition. Since this solution only needs to recognize simple numbers and letters, a simple OCR algorithm is used to detect the numerical and alphabetical characters below the barcode, and then compared with the recognized barcode characters. If they match, it indicates that the product barcode detection has passed; if they do not match, the product code is recorded.
A character template database is established for character matching. In this solution, the characters that need to be recognized are only English letters and numbers, so the database is not large.
(Character database)
Then, physical location segmentation is used to segment the characters below the barcode.
(Characters below the barcode)
Next, a horizontal projection of the image is performed, so that the projection value where there are characters is high, while the projection value where there are no characters is low.
(Horizontal projection values)
Next, a scanning line is used to scan from bottom to top. This scanning line will intersect with the curves in the image,
and these intersection points will segment the peaks into individual regions. The barcode character consists of 13 digits, plus a few protrusions from the barcode, totaling 18 regions. Therefore, when the scanning line segments the peaks into 18 regions, it stops. Then, based on the coordinates of the projections of these 18 regions, the 13 (18) characters in the image can be segmented.
After segmenting the characters, the final step is to use the OCR neural network to match the segmented characters with the character images in the character library, ultimately recognizing the characters. By comparing the recognized string with the string recognized from the barcode, it can be determined whether the barcode is accurate.
3.1 Material Budget
3.2 Solution Limitations
This solution has areas that still need improvement:
1) The placement of the light source has high requirements, relying on the light source to obtain distinctly contrasting images, otherwise, there may be cases of failure to segment the barcode sticker from the background;
2) Although the probability is low, there still exists the possibility of failure in detecting a certain barcode. In this case, the approach taken by this solution is to discard that detection;
3) This solution only detects the encoding quality of the barcode, meaning it only checks whether the characters contained in the barcode correspond to the identification characters of the mobile product, without correcting any errors.
Discussion Group
Welcome to join the reader group of the public account to exchange ideas with peers. Currently, there are WeChat groups for SLAM, 3D vision, sensors, autonomous driving, computational photography, detection, segmentation, recognition, medical imaging, GAN, and algorithm competitions (these will gradually be subdivided). Please scan the WeChat ID below to join the group, with a note: “nickname + school/company + research direction”, for example: “Zhang San + Shanghai Jiaotong University + Vision SLAM”. Please follow the format, otherwise, it will not be approved. After successful addition, you will be invited to relevant WeChat groups based on research direction. Please do not send advertisements in the group, otherwise, you will be removed from the group. Thank you for your understanding~
