What Happens Inside Your Smartphone the Moment You Scan

Seven Processes Run in Just One Second

When you hold your smartphone camera up to a black-and-white pattern, the linked page opens in about one second. During this one second, seven processes run at tremendous speed inside the smartphone. (1) The camera captures a frame, (2) candidate regions for the pattern are detected in the image, (3) the three finder patterns are identified and the position and tilt are calculated, (4) the pattern is divided into a grid and each cell is judged as white or black, (5) the binary data is converted into a string, (6) damaged data is repaired by error correction, and (7) the string is recognized as a URL and the browser is launched.

These processes are shared between the smartphone's CPU and its image processing chip. On a typical smartphone, the camera captures 30-60 frames per second and tries to detect the pattern in real time in each frame. In other words, from the moment you hold up the camera, the process of "looking for the pattern" runs 30-60 times per second.

Steps 1-2: Capturing and Detecting Candidate Regions

An image captured by the camera is a collection of millions of pixels (picture elements). To find where the pattern is in it, the image is first converted into two colors, white and black (binarization). Next, regions where black pixels are densely clustered are detected as "pattern candidates."

What matters in this process is "contrast." If a black pattern is printed sharply on a white background, detection of the candidate regions is completed in an instant. However, in dark places or when the color of the pattern is pale, the boundary between white and black becomes blurred, and detection takes time or fails. The reason you are told to "scan in a bright place" is to raise the accuracy of this step.

Steps 3-4: Locating the Position and Dividing into a Grid

Once a candidate region is found, the next step is to look for the three finder patterns (large square markers). From the positional relationship of these three, the exact position, size, and tilt of the pattern are calculated. The reason it can be read even when scanned at an angle is that this calculation corrects the tilt.

Once the position is identified, the whole pattern is divided into a grid like a Go board, and each cell is judged as white or black. This judgment is not a simple "bright or dark" but is based on brightness relative to the surrounding cells. It is thanks to this relative judgment that the code can be read accurately even when the lighting is uneven.

Steps 5-6: Decoding and Error Correction

The results of the white-and-black judgment become a sequence of 0s and 1s (a bit string). Converting this bit string into a string of characters according to the pattern's standard is "decoding." The first few bits record the type of data (numbers, alphanumeric characters, kanji, and so on) and its length, so the decoder reads them before converting the main data.

At the same time as decoding, error correction processing is also carried out. If some cells could not be read correctly because of dirt or scratches, the original data is restored using spare data (error correction codes). This processing is based on a mathematical algorithm called the Reed-Solomon code, and under the standard, the setting with the highest error correction level can restore damage of up to 30%.

Instant Processing Creates the Experience

From pointing the camera at a code to the target page opening, we feel it as nothing more than an instant. Behind the scenes, however, a number of processes run in order, such as capturing, finding the pattern, identifying its position, reading light and dark, correcting errors, and recognizing the URL. It is precisely because these connect without a hitch that we hardly feel as if we are being kept waiting. If any one of the processes stumbles, scanning fails and has to be redone. The comfort of a casual instant is supported by a chain of processes accurately built up inside.

What Is Happening Behind the Scenes When It Cannot Be Read

When scanning does not go well, somewhere inside the smartphone a process has failed to complete and has stopped. The pattern cannot be found, the position cannot be fixed, light and dark cannot be distinguished, or so much information is missing that it cannot be fully corrected. The causes vary, but in every case the series of processes has stumbled somewhere. That is exactly why changing the distance a little, adjusting the brightness, or wiping dirt off the code lets the stopped process move forward and the scan succeeds. If you think of a failure to read as an interruption of the processing happening inside, it becomes easier to figure out what to do.

Step 7: Recognizing the URL and Launching the Browser

If the decoded string is a URL (a string starting with https://), the smartphone's OS recognizes it, launches the browser, and opens that page. For data other than URLs (phone numbers, email addresses, Wi-Fi information, text), the app that corresponds to each type is launched.

On iPhone, when the camera app detects a URL, a yellow banner appears at the top of the screen, and tapping it opens the browser. On Android, the behavior differs by model, but in many cases the browser launches automatically. The fact that everything, including this last step, is completed within one second shows how high the processing power of smartphones is.