Facial Recognition software and Plastic Surgery are on the rise - can they coexist?

When a person changes their name or address, they are required by law to update all forms of identification (such as driver's license, Social Security card). While some practices recommend their patients to do so, the same is not required post-plastic surgery. Before this wouldn't have been much of an issue but faces will soon replace fingerprints, turning people into walking ID cards thanks to facial recognition.

Facial recognition technology is slated to become a security measure available to everyone. This new wave in interest is thanks to Samsung's Galaxy S8, President Trump's executive order requesting the technology be used in airports, and rumors of the technology on Apple's iPhone 8. At the same time, plastic surgery is also becoming more and more common. Last year, Americans spent almost $16.5 billion on plastic surgery. With both plastic surgery and facial recognition being on the rise, is facial recognition technology advanced enough to identify someone after plastic surgery?

New faces everyday

Top five centers of plastic surgery in the U.S.

According to a study done by RealSelf, the " TripAdvisor for the body, face, and smile", and the analysis done by Insider Monkey, the top five cities with the highest plastic surgery rates are:

1. Miami, FL
2. Los Angeles, CA
3. San Francisco, CA
4. Austin, TX
5. Charlotte, NC

This list was made by comparing the top ten cities with the most plastic surgeons per 100,000 residents to the top ten cities with the most RealSelf searches per 100,000 residents. This study covers all kinds of plastic surgery but when it comes to facial procedures, these cities are also in the hearts of the regions popular for them.

Based on the 2016 Plastic Surgery Statistics Report, we found the top five facial procedures done in 2016 to be:

1. Botulinum Toxin Type A (typically called by its brand name such as Botox)
   1. It is a nonsurgical injection known to reduce/eliminate facial fine line and wrinkles. It works by blocking nerve signals to the injected muscle so it doesn't contract.
   2. Number of procedures: 7,056,255
   3. Total cost of procedures: $2,716,658,175
   4. Most popular in the West Coast
2. Soft tissue fillers (also known as dermal fillers)
   1. They're good for removing lines and restoring volume (fullness) in the face. It's a non-surgical rejuvenation option. Good for thicker lips.
   2. Number of procedures: 2,600,868
   3. Total cost of procedures: $1,727,231,356
   4. Most popular in the West Coast and Northeast
3. Chemical Peel and Microdermabrasion (They are grouped together because the procedures and results are similar)
   1. Chemical Peel:
      1. Like a peel-off face mask, a chemical peel is a chemical solution brushed onto the skin that usually begins to peel 48 hours after application. It is used for acne/scars, fine lines and wrinkles, irregular pigmentation, sun damage, and more.
      2. Number of procedures: 1,360,850
      3. Total cost: $915,852,050
      4. Most popular in the West Coast
   2. Microdermabrasion:
      1. Requires use of an instrument to lightly "sand" or remove the first layer of skin. The results are similar to that of a chemical peel; it is known to shrink enlarged pores, treat acne, and even skin color.
      2. Number of procedures: 775,014
      3. Total cost: $106,951,932
      4. Most popular in the Northeast
4. Rhinoplasty
   1. Also known as "nose reshaping" or a "nose job". The American Society of Plastic Surgeons website states that rhinoplasty can enhance the harmony of a face by making it more symmetrical.
   2. Number of procedures: 223,018
   3. Total cost: $1,125,348,828
   4. Most popular procedure the Northeast
5. Blepharoplasty
   1. A surgical procedure on the upper, lower, or both eyelids. The surgery can correct dropping or hooded upper eyelids and bags on the lower eyelids.
   2. Number of procedures: 209,020
   3. Total cost: $631,658,440
   4. Most popular in the Southeast

Each of these procedures, minus chemical peel and microdermabrasion, can result in the patient looking like a completely different person post-operation. The difference can, and most likely will, be drastic when patients combine more than one procedure. For example, say a patient combined a nose job with a chemical peel or microdermabrasion. Not only will the symmetry of their face be different, but their skin color may be lighter or darker. While it is said that machines are better at facial recognition than humans, is the technology able to recognize a person who's undergone plastic surgery?

The steps to facial recognition

While there are multiple variations in facial recognition, Georgia State University professor Jessica Cino summed it up best. There are "three steps: detection, faceprint creation, and verification or identification". The detection phase is just as it sounds; the system separates a face from the background. According to the Handbook of Face Recognition, after detection, the system will normalize the image of the face to have a better angle and better lighting. This allows for feature extraction which measures each feature, also known as nodal point, to give each face its own code. Such measurements include the distance between eyes, eyebrows, nose width, and more but this is where systems tend to differ. Some systems, such as Face++, are more detailed than others. While some measure 80 nodal points, Face++ Landmarks measures up to 106 points. A numerical code is created for each measurement that in turn becomes the subject's faceprint. The system's database then searches to find a match for that faceprint to identify or verify a person.

Snapchat's lenses, or 'filters', go through a similar process as the one described above. In its detection phase, the system finds a face using pixels and the color of them. The colors are important because when Snapchat's filter is looking for a face, it looks for areas of contrast. For example, eye sockets are normally darker than the forehead. Once it recognizes a face, it begins feature extraction. To extract features, the system uses an active-shape model. The model is an average face made of out a database of hundreds to thousands of faces whose features were manually marked by humans. The marking points are also nodal points. Since these are live filters, the average face is shifted to align with a person's face and features. After the nodal points are aligned, a mask is created that moves with the face to make live filters.

How well does it work?

Currently, facial recognition is under a lot of scrutiny. There is doubt concerning the accuracy of the technology especially after Samsung's Galaxy S8 was easily unlocked by a selfie rather than the real face. The newer generation of facial recognition is said to have "liveness" detection to fix the selfie loophole and infrared beams for better accuracy. Regardless of new improvements, is there a way facial recognition can tell if someone had plastic surgery? Even Snapchat's intricate system has failures. Some share the pain of having a filter register their lips as too small or their eyes or just not registering their face at all. The technology used to unlock phones, doors, and others of the sort will most likely have a backup unlocking method or the option to use other methods as the primary such as fingerprint scanning or passcode. It is surveillance facial recognition that raises concerns.

It's not the most accurate

According to The University of Washington, facial recognition systems have claimed to achieve near perfect accuracy rates. The problem with that is the dataset those tests used only had 13,000 images. That's not a good reflection of how the technology is currently being used when the airports in the cities with the most plastic surgery have some of the busiest airports in the US:

1. MIA is the 12th busiest - Miami, FL
2. LAX is the 2nd busiest - Los Angeles, CA
3. SFO is the 7th busiest - San Francisco, CA
4. AUS is the 34th busiest - Austin, TX
5. CLT is the 10th busiest - Charlotte, NC

To test the true accuracy against a large crowd, the University of Washington started an on-going challenge called the MegaFace Challenge. The results proved that there is still more work to be done to make facial recognition more accurate but in doing so, how will it combat the rising interest in plastic surgery? One solution may be to create multiple faceprints per person to allow for variations. Maybe faceprints that predict what that person would look like with a certain nose or eyes as we see when making characters in video games. But if the technology isn't that accurate now, it's safe to say its predictions would be unreliable.

Luckily, it is still optional for citizens to use the facial recognition option in airports but it is rumored that it may become the new ID method for all traveling in and out of airports. Before that happens, this question needs answers: How will facial recognition technology properly identify those who have undergone plastic surgery?