Open in app

Sign in

Write

Sign in

Building a Face Recognition App with the FaceOnLive SDK in Android

With Liveness Detection built-in

Shubham Panchal
FaceOnLive

--

Photo by Antoine Beauvillain on Unsplash

As an Android developer working with face recognition technologies, I’ve tried several methods and libraries, such as using FaceNet with MLKit’s Face detection, CameraX and dlib-android. Using server-side inference, where face images are sent to a remote API for verifying their similarity, is also a good strategy but requires an additional backend service and introduces latency in the app due to the network calls. Moreover, face recognition technology also needs to work on a live camera feed, which entirely rules out server-side inference.

I got a chance to try FaceOnLive’s Face and Liveness detection Android SDK and I would like to share my experience with the SDK and its integration in an existing Android app.

Where do we use face recognition technology?

Face recognition can be a means to authenticate a person’s identity by matching a picture with an existing image stored in a database. There are several use cases, which include:

  • Attendance monitoring in campuses and offices: A reliable face recognition app can replace finger-print or ID-card scanning mechanisms to mark the daily attendance of workers in a factory, employees in an office or for students on an education campus. Combined with live-ness detection, the chances of making a proxy attendance are reduced significantly. Moreover, the need to include special hardware devices (sensors or fingerprint scanners) is also eliminated.
  • eKYC (e Know Your Customer): It is the process wherein authorized organizations and agents verify and authenticate a customer’s personal data to a specific service available. Matching the face of customer in the eKYC process with picture given in the official documents can be performed easily across numerous documents easily with face recognition technology.
  • Customer Analytics: By identifying customers entering a shopping mall or a store, a number of services can be customized by analyzing their shopping history and presence in the store. As face recognition systems can also be used as a means of authentication, checkout-free solutions are also possible as they can directly determine the person’s payment methods by authenticating their identity through a picture of their face. Store owners can analyze frequent customers and tailor specific services which would maximum the retention and satisfaction of customers.
  • Video Surveillance with face recognition: It involves the use of security cameras and software to identify people by analyzing the video feed and their facial features. It can help in identifying suspects in criminal investigations or in monitoring customer behaviour in stores.

Features of FaceOnLive SDK

  • On-device inference, eliminating the need of making server calls on each face detection request. It also ensures the least amount of latency and improves overall user experience.
  • With built-in face liveness detection, we need not write separate logic for determining the liveness of the face detected in the camera frame. This also eliminates using a face proxy in critical applications.

Android Setup

In this section, we will discuss the steps needed to integrate FaceOnLive Face Recognition and Liveness Detection SDK in an Android app.

Getting Started with the SDK

Once we have received the API key, it should be added to the license file in app/src/main/assets ,

 BsTr9o4f4R/rM3TxbCWVb/hrOJuOIdz8ArQ/t2IgQFFUQzGHOLNNaMJiK/fUfr5zo005zoTA/cm6 
 VoZ6iGl+/hZGA3R5T/VWwhxekbw8JVz9sNesU6rMG5+1cNSN75trH2tpzdCPZ28ZDnZlttmiuUoC 
 9QazRe1xKi5tUXa+xgIxzL0vE6UW2dLKWaEXjn3fSJfLxXWw0q+UZP0hQAXb5Y9Yl/NVi7y3d0xT 
 Vq6/weuMQkgLcNdLqFRvQXup0M9W/pvuhaubySAxHCKVY8wToygN2iM78cOkyyAbGVwZeGQP0Jfd 
 46VZo+w+KCNw355j3osVVMghrOcVZnfbp1dNyg==

We can then activate the SDK in the onCreate method of our MainActivity by calling FaceSDK.setActivation by passing the API-key read from the license file in the assets directory,

 val license_str = application.assets.open("license").bufferedReader().use{ 
     it.readText() 
 } 
 var ret = FaceSDK.setActivation(license_str) 
 if (ret == FaceSDK.SDK_SUCCESS) { 
     ret = FaceSDK.init(assets) 
 }

ret represents the activation response from the SDK. It can assume several values based on the result of the SDK activation process, and should be handled accordingly,

if (ret != FaceSDK.SDK_SUCCESS) {(
    var message = "SDK initialization failed"
    if (ret == FaceSDK.SDK_LICENSE_KEY_ERROR) {
        message = "License key error!"
    } else if (ret == FaceSDK.SDK_LICENSE_APPID_ERROR) {
        message = "App ID error!"
    } else if (ret == FaceSDK.SDK_LICENSE_EXPIRED) {
        message = "License key expired!"
    } else if (ret == FaceSDK.SDK_NO_ACTIVATED) {
        message = "Activation failed!"
    } else if (ret == FaceSDK.SDK_INIT_ERROR) {
        message = "Engine init error!"
    }
}

The SDK is now ready to be used for face recognition. In the next step, we register faces (subjects) whose faces have to be recognized by the SDK.

Registering Faces for Recognition

Registering a user with the SDK (source: author)

The face recognizer requires an image, a ground-truth of the person is which has to be recognized from the live camera feed. This image is annotated with a user name or id provided by the application user and is stored on a local on-device database.

In real-time, while detecting faces from the camera feed, the detected face is compared against the faces that have been registered or provided as a ground-truth to the SDK. If the detected face matches an image present in the dataset with similarity given a certain threshold, the SDK concludes the detected face belongs to the person whose image had the greatest match.

// (1) Initialize face detection API
val faceDetectionParam = FaceDetectionParam()
faceDetectionParam.check_liveness = true
// liveness_level determines the model used for liveness detection
// 0 -> model v0
// 1 -> model v2-fast
faceDetectionParam.check_liveness_level = 0

// (2) Perform face detection
// bitmap -> input image containing faces
var faceBoxes: List<FaceBox>? = FaceSDK.faceDetection(bitmap, faceDetectionPar

// (3) Interpret the face detection results
if(faceBoxes.isNullOrEmpty()) {
    // No faces found in the image
} 
else if (faceBoxes.size > 1) {
    // Multiple faces found in the image
} 
else {
    // (4) Extract face templates (Features) from the input image
    val templates = FaceSDK.templateExtraction(bitmap, faceBoxes[0])

    // (5) Save `templates` locally, for instance in an SQLite database
    // The cropped face can also be obtained with Utils.cropFace
    // From the image, crop the region which bounds the face
    val faceImage = Utils.cropFace(bitmap, faceBoxes[0])
}

In the next step, we’ll explore how to integrate the face recognition SDK with CameraX to display a live camera feed (preview).

Creating a Live Preview with Face Detection

Face detection and Liveness detection with the SDK (source: author)

We create a live camera preview with Fotoapparat by providing it the necessary camera parameters and a CameraView object,

val PREVIEW_WIDTH = 720
val PREVIEW_HEIGHT = 1280
private lateinit var cameraView: CameraView
private lateinit var faceView: FaceView
private lateinit var fotoapparat: Fotoapparat
private var activeCamera: Camera = Camera.FrontCamera

override fun onCreate(savedInstanceState: Bundle?) {
    super.onCreate(savedInstanceState)
    setContentView(R.layout.activity_camera_kt)

    cameraView = findViewById(R.id.preview)
    faceView = findViewById(R.id.faceView)
    fotoapparat = Fotoapparat.with(this)
        .into(cameraView)
        .lensPosition(activeCamera.lensPosition)
        .frameProcessor(FaceFrameProcessor())
        .previewResolution { Resolution(PREVIEW_HEIGHT,PREVIEW_WIDTH) }
        .build()
        
    if (ContextCompat.checkSelfPermission(this, Manifest.permission.CAMERA)
        == PackageManager.PERMISSION_DENIED
    ) {
        ActivityCompat.requestPermissions(this, arrayOf(Manifest.permission.CAMERA), 1)
    } else {
        fotoapparat.start()
    }

    ...
}

We also need to write an implementation of FaceFrameProcessor which contains the process( frame: Frame ) method called on every frame produced by the camera, and we perform the following tasks,

A schematic flow of identifying a face using the SDK (source: author)
  • Convert the YUV image into a RGB image contained within a Bitmap object using FaceSDK.yuv2Bitmap . Let’s call it inputImage for our understanding.
  • Perform face detection on inputImage using FaceSDK.faceDetection which returns a list of FaceBox
Face Liveness detection (source: author)
  • Check if faceBox.liveness exceeds a certain threshold, if not, return from the process method as the face may not belong to a physical human being.
  • Extract face templates from the inputImage and the FaceBox using FaceSDK.templateExtraction calling them subjectTemplates
  • Read templates stored in the database and compare each of them with subjectTemplates using FaceSDK.similarityCalculation which returns a Float representing the similarity. If the similarity is greater than the given threshold, compare it with the maximum similarity we have found in earlier iterations.
inner class FaceFrameProcessor : FrameProcessor {

    override fun process(frame: Frame) {

        // (1) Convert the YUV frame received from the camera to a
        //     RGB bitmap
        val bitmap = FaceSDK.yuv2Bitmap(frame.image, frame.size.width, frame.size.height, cameraOrientation)

        // (2) Initiate face detection
        val faceDetectionParam = FaceDetectionParam()
        faceDetectionParam.check_liveness = true
        // liveness_level determines the model used for liveness detection
        // 0 -> model v0
        // 1 -> model v2-fast
        faceDetectionParam.check_liveness_level = 1
        val faceBoxes = FaceSDK.faceDetection(bitmap, faceDetectionParam)

        // (3) Provide faceBoxes to the faceView
        //     for rendering them on the screen
        runOnUiThread {
            faceView.setFaceBoxes(faceBoxes)
        }
        if(faceBoxes.size > 0) {
            val faceBox = faceBoxes[0]

            // Check if liveness score is above a given threshold
            if (faceBox.liveness > 0.7) {

                // (4) Compare faces stored in database
                //     and determine the greatest match
                val templates = FaceSDK.templateExtraction(bitmap, faceBox)
                var maxSimiarlity = 0f
                var maximiarlityPerson: Person? = null
                for (person in DBManager.personList) {
                    val similarity = FaceSDK.similarityCalculation(templates, person.templates)
                    if (similarity > maxSimiarlity) {
                        maxSimiarlity = similarity
                        maximiarlityPerson = person
                    }
                }
                if (maxSimiarlity > SettingsActivity.getMatchThreshold(context)) {
                    recognized = true
                    val identifiedPerson = maximiarlityPerson
                    val identifiedSimilarity = maxSimiarlity
                    // (5) Navigate the user to the next screen where the
                    //     similarity score can be displayed along with the 
                    //     detected face parameters
                }
            }
        }
    }
}

Reviewing Face Parameters

Along with the bounding boxes for the face and its liveness score, the SDK also returns the yaw, pitch and roll of the detected face. Let us discuss each of these parameters:

A diagrammatic representation of yaw, pitch and roll. Source
  • Yaw: It indicates the degree of rotation about an axis which goes straight up from the head. Turning the head sideways will change the yaw and make it positive if the head is turned left and negative if the head is turned right (assuming the face is in front of the viewer). For a straight gaze, the yaw is 0.
  • Pitch: Similar to yaw, it measures the degree of rotation about an axis passing through both ears. Looking upwards if make the pitch negative whereas looking downwards will make it positive.
  • Roll: It measures the degree of rotation about an axis passing through the nose and the back of the head. It measures the sideways tilt of the face.

For each FaceBox returned by FaceSDK.faceDetection , these parameters can be retrieved easily by property access,

val faceDetectionParam = FaceDetectionParam()
faceDetectionParam.check_liveness = true
faceDetectionParam.check_liveness_level = 1
val faceBoxes = FaceSDK.faceDetection(bitmap, faceDetectionParam)

val faceBox = faceBoxes[0]
val yaw = faceBox.yaw
val pitch = faceBox.pitch
val roll = faceBox.roll

Conclusion

I hope the blog was informative, and the readers will consider using the FaceOnLive SDK for face recognition in their Android apps. For doubts and suggestions, you can write a comment here on Medium, or connect with me directly. Have a great day ahead!

Android App Development
Face Recognition
Software Development
Programming
Kotlin

--

--

Shubham Panchal
FaceOnLive

Written by Shubham Panchal

1.4K Followers
Editor for

Android developer, ML and Math enthusiast. Mentorship, doubts, collaborations -.> https://topmate.io/shubham_panchal

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams