Camera SDK

Version : NUWA Camera SDK v1.1

Introduction

we can use the NUWA Camera SDK to resolve recognition people or objects, and all the function that is connected to camera service.

Due to all functions of camera SDK are async(Based on AIDL) design, App needs to register a callback to receive all kinds of notifications and results from Robot,and we need to point which function we want to use first, and before we use FACE_RECOGNITION function we need to use the face recognition app to add people's face(remember) as a family member, otherwise sdk can not know the people who it is seeing.

Please find NUWA Camera SDK sample code on

• NUWA GitHub Camera SDK Sample Code

Support product list : Kebbi Air series (AIR-H200、AIR-H201、AIR-H202、AIR-H203)

Add member to nuwa family system

There are two way to add people's face to face recognition list

1. Use 「Add Members」 to recognition face of new family member.
2. Lunch 「Add Members」by programming

Sample Code to lunch

​x
//Start App and request result
Intent intent = new Intent("com.nuwarobotics.action.FACE_REC");
intent.setPackage("com.nuwarobotics.app.facerecognition2");
intent.putExtra("EXTRA_3RD_REC_ONCE",true); //Boolean, true : only recognition one people
intent.putExtra("EXTRA_3RD_TITLE","My title"); //String, customize your recognition UI title
intent.putExtra("EXTRA_3RD_CONFIG_NAME","George"); //String, provide name directly
startActivityForResult(intent,1);
​
//Receive result data
protected void onActivityResult(int requestCode, int resultCode, Intent data) {
    super.onActivityResult(requestCode, resultCode, data);
    if(resultCode > 0){
        long faceid = data.getLongExtra("EXTRA_RESULT_FACEID",0);
        String name = data.getStringExtra("EXTRA_RESULT_NAME");
        //3rd app could save these data to database for more function prupose
    }
}
​

Register camera SDK service

Application should register receive which recognition result when start use NUWA Camera SDK

xxxxxxxxxx
mCameraSDK.register(mCameraSDKCallback, Constants.FACE_DETECTION | 
Constants.FACE_RECOGNITION | Constants.OBJ_RECOGNITION, this.getClass().getPackage().getName());

unregister camera SDK

After application finish or activity suspend, don't to forget to unregister NUWA Camera SDK.

xxxxxxxxxx
mCameraSDK.unregister(this.getClass().getPackage().getName());

Support Recognition Type

xxxxxxxxxx
* funtions we can use:
  1.FACE_RECOGNITION : detect face who register as family memeber
  2.FACE_DETECTION : Detect all face rectangle coordinate as a List
  3.OBJ_RECOGNITION : Recognitions result of object, return a List<Recognition> list.
  4.FACE_TRACK : Tacking single face location

use callback to get result data:

xxxxxxxxxx
private CameraSDK.CameraSDKCallback 
​
mCameraSDKCallback = new CameraSDK.CameraSDKCallback()
{
​
        @Override
        public void onConnected(boolean isConnected) {
            log.d(TAG, "onConnected:" + isConnected);
        }
​
        @Override
        public void onOutput(Map<Integer, OutputData> resultMap) {
            log.d(TAG, "onOutput:" + resultMap);
            final StringBuilder sb = new StringBuilder();
            for (Integer integer : resultMap.keySet()) {
                OutputData outputData = resultMap.get(integer);
                if (outputData != null && outputData.data != null && !"null".equals(outputData.data)) {
​
                    switch(type){
                        case Constants.FACE_DETECTION :{
                            //It allow find all face on preview screen, and get all rect.  (but no mask information)
                            ObjectMapper sMapper = new ObjectMapper();
                            List<Rect> list = null;
                            try {
                                list = sMapper.readValue(outputData.data, new TypeReference<List<Rect>>() {
                                });
                            } catch (IOException e) {
                                e.printStackTrace();
                            }
                            if (list != null && !list.isEmpty()) {
                                for (Rect rect : list) {
                                    if(rect != null)
                                        Log.d(TAG,"Got face rect:("+rect.bottom+","+rect.left+","+rect.right+","+rect.top+")");
                                }
                            }
                            break;
                        }
                        case Constants.FACE_RECOGNITION :{
                            //It allow recognition face in center of screen.
                            //We can get name、face rect、wear mask information、age、etc.....
                            //Example of get face data
                            // "@#$" is a special unknown result, we need to ignore it
                            FRData returnFace = mGson.fromJson(outputData.data, FRData.class);
                            if(returnFace != null && !"@#$".equals(returnFace.name) && !"null".equals(returnFace.name)){
                                Log.d(TAG,"Find FaceData user_name:"+returnFace.name );  //return name come from registered on Robot Family.
                                Log.d(TAG,"user face rect : "+returnFace.rect.toString());
                                Log.d(TAG,"user wear mask : "+returnFace.mask);
                                Log.d(TAG,"user age : "+returnFace.age);
                            }else{
                                if(returnFace != null)
                                    Log.d(TAG,"Unknown stranger "+returnFace.name);
                            }
                            break;
                        }
                        case Constants.FACE_TRACK :
                            break;
                        case Constants.OBJ_RECOGNITION : {
                            float mConfidence = 0.5f;//standard of confidence
                            ObjectMapper sMapper = new ObjectMapper();
​
                            List<Recognition> list = null;
                            try {
                                list = sMapper.readValue(outputData.data, new TypeReference<List<Recognition>>() {
                                });
                            } catch (IOException e) {
                                e.printStackTrace();
                            }
​
                            if (list != null && !list.isEmpty()) {
                                String title = null;
                                double confidence = 0;
                                //Find best confidence result.
                                for (Recognition recognition : list) {
                                    Debug.logD(TAG, "onOutput:" + recognition.title + ", confidence = " + recognition.confidence);
                                    if (recognition.confidence > confidence) {
                                        title = recognition.title;
                                        confidence = recognition.confidence;
                                    }
                                }
                                if (confidence > mConfidence) {
                                    Bundle bundle = new Bundle();
                                    bundle.putString("title", title);
                                    bundle.putDouble("confidence", confidence);
                                    bundle.putString("frameId", outputData.frameId);
                                }
                            }
                            break;
                        }
                    }
                }
​
            }
            
        @Override
        public void onPictureTaken(String path) {
            CsDebug.logV(TAG, "image path=" + path);
        }
    };
    

Parser outdata of each recognition type

FACE_DETECTION

Detect all face rectangle coordinate on camera.

We use ObjectMapper/TypeReference class to extraction list of face rectangle data.

Import library (build.gradle) :

xxxxxxxxxx
    implementation 'com.fasterxml.jackson.core:jackson-databind:2.9.5'
    implementation 'com.fasterxml.jackson.core:jackson-core:2.9.5'

Sample code :

xxxxxxxxxx
    import com.fasterxml.jackson.databind.ObjectMapper;
    import com.fasterxml.jackson.core.type.TypeReference;
​
​
    //It allow find all face on preview screen, and get all rect.  (but no mask information)
    ObjectMapper sMapper = new ObjectMapper();
    List<Rect> list = null;
    try {
        list = sMapper.readValue(outputData.data, new TypeReference<List<Rect>>() {
        });
    } catch (IOException e) {
        e.printStackTrace();
    }
    if (list != null && !list.isEmpty()) {
        for (Rect rect : list) {
            if(rect != null)
                Log.d(TAG,"Got face rect:("+rect.bottom+","+rect.left+","+rect.right+","+rect.top+")");
        }
    }

Reference library :

xxxxxxxxxx
 * https://javadoc.io/doc/com.fasterxml.jackson.core/jackson-databind/2.3.1/com/fasterxml/jackson/databind/ObjectMapper.html
 * https://fasterxml.github.io/jackson-core/javadoc/2.2.0/com/fasterxml/jackson/core/type/TypeReference.html

FACE_RECOGNITION

It allow recognition face in center of screen.

We can get name、face rect、wear mask information、age、etc..... Import library (build.gradle) :

xxxxxxxxxx
implementation 'com.google.code.gson:gson:2.7'

Sample code :

xxxxxxxxxx
    import com.google.gson.Gson;
    import com.google.gson.JsonArray;
​
    //Example of get face data
    // "@#$" is a special unknown result, we need to ignore it
    FRData returnFace = mGson.fromJson(outputData.data, FaceRecData.class);
    if(returnFace != null && !"@#$".equals(returnFace.name) && !"null".equals(returnFace.name)){
        Log.d(TAG,"Find FaceData user_name:"+returnFace.name );  //return name come from registered on Robot Family.
        Log.d(TAG,"user face rect : "+returnFace.rect.toString());
        Log.d(TAG,"user wear mask : "+returnFace.mask);
        Log.d(TAG,"user age : "+returnFace.age);
    }else{
        if(returnFace != null)
            Log.d(TAG,"Unknown stranger "+returnFace.name);
    }

If you did not found FaceRecData to parser json format of recognition result, please declare this class on your project.

xxxxxxxxxx
public class FaceRecData {
    public static final String UNKNOWN_NAME = "other";
    /**
     *
     */
    public int idx;
    public String conf = null;
    /**
     * name of face which register on family member. (CameraSDK v1)
     */
    public String name = null;
    /**
     * rectangle of face coordinate on screen
     */
    public Rect rect = null;
    /**
     * age which AI recognized
     */
    public String age;
    /**
     * gender which AI recognized
     */
    public String gender;
    /**
     * unique face id
     */
    public long faceid;
​
    public FaceRecData() {
    }
}

FACE_TRACK

Use to get face move coordinate on screen smoothly.

Sample code : ongoing

OBJ_RECOGNITION

OBJ_RECOGNITION use NUWA AI model to recognition what object is.

Pre-training data list : https://developer-docs.nuwarobotics.com/sdk/CameraSDK/obj/obj_list.html

We use ObjectMapper/TypeReference class to extraction list of face rectangle data.

Import library (build.gradle) :

xxxxxxxxxx
    implementation 'com.fasterxml.jackson.core:jackson-databind:2.9.5'
    implementation 'com.fasterxml.jackson.core:jackson-core:2.9.5'

Sample code :

xxxxxxxxxx
    import com.fasterxml.jackson.databind.ObjectMapper;
    import com.fasterxml.jackson.core.type.TypeReference;
    
    //class define on NuwaSDK
    import com.nuwarobotics.lib.visual.model.Recognition;
    
    
    float mConfidence = 0.5f; //standard of confidence
    ObjectMapper sMapper = new ObjectMapper();
    
    List<Recognition> list = null;
    try {
        list = sMapper.readValue(outputData.data, new TypeReference<List<Recognition>>() {
        });
    } catch (IOException e) {
        e.printStackTrace();
    }
    
    if (list != null && !list.isEmpty()) {
        String title = null;
        double confidence = 0;
        //Find best confidence result.
        for (Recognition recognition : list) {
            Debug.logD(TAG, "onOutput:" + recognition.title + ", confidence = " + recognition.confidence);
            if (recognition.confidence > confidence) {
                title = recognition.title;
                confidence = recognition.confidence;
            }
        }
        if (confidence > mConfidence) {
            Bundle bundle = new Bundle();
            bundle.putString("title", title);
            bundle.putDouble("confidence", confidence);
            bundle.putString("frameId", outputData.frameId);
        }
    }