A face-clustering toolkit for MacOS (Silicon)
The intro panel gives quick access to two options: load video or load project.
A project is a file in .fcproject format generated by this application, which resumes the face-clustering networks created and edited by the user. A .fcproject file is manually generated by saving a project in the editor.
To start a new project, the user will have to select a video with the "load video" option. Directly supported formats are .mp4 and .mov, but technically all formats compatible with the AVFoundation utility should be usable (tweaking might be necessary for esoteric formats, though).
Successful decoding of a video file will direct the user to the import setting page. It gives the user two import settings: frame extraction interval ("analyze every") and frame scaling. Frame extraction interval is used to reduce the number of the frames used for face recognition, which improves stability for large video processing. It also supports two units for deciding the interval: second and frame. Frame scaling decides if the analyzed frames should be scaled down to provide further acceleration.
This application stores all faces as objects with multiple attributes. Seven basic attributes are generated upon the face's being detected. These are frame index (integer), face box (4D vector), confidence (decimal), face rotation (3D vector generated by Apple's face API), Path (string, the relative path where the face is stored), cluster (string), and position (point variable).
All attributes align with one of the six types: integer, decimal, integer vector, decimal vector, string, and point. In addition to the basic attributes, the user can add new attributes to a network or remove the added attributes (basic attributes cannot be removed, but "position" can be replaced by other point attributes). The point type is a special type that is used for face clustering. In the editor's Overview panel, the user can decide an attribute of point type to be the "display positioning attribute". This action will make the faces in the Network panel rearranged according to the new position attribute, and the distances between faces calculated based on the positioning attribute are used for dividing all faces into clusters.
After loading a previous project or creating a new project with a video, the user will be navigated to the editor. The editor provides five panels for manipulating and interpreting the statistics generated by face recognition and clustering, including Network, Frames, Overview, Project, and Timeline.
The Network panel is the default view of the editor. It provides a GUI-empowered interface for editing and fine-tuning the results of face clustering. In this page all faces are positioned based on the point attribute set as the "display positioning attribute", while editing occurring in this panel is directly saved to that attribute's values.
The user can switch between two modes using the Preview/Edit Button in the toolbar. In the Preview mode, the button writes "Edit", and the user can drag the canvas to navigate the network preview. Mouse wheel is used to zoom in/out the canvas. The Cluster button shown in this mode allows the user to regenerate all clusters based on the current positions of all faces and a given distance. In the Edit mode, however, dragging will allow the user to move the positions of faces. The Cluster button is replaced with a slider that determines the range of dragging. if the slider is at the minimal value, only one face can be dragged at once, but if a higher range is picked, multiple faces within the cursor can be relocated together.
In both modes, the user can deactivate certain faces by right-clicking them. Deactivated faces are not included in data analysis and clustering but still stored within the network and can be recovered by another right-click in the Network panel.
There are also three modes for displaying the cluster results: none, lines, and polygons. The polygon mode is used by default, but it has an issue that when there are only two faces in a cluster, the cluster cannot be displayed. The lines mode addresses this issue by drawing lines between all pairs of faces in a cluster.
The Frames panel is used to control the results of facial recognition at frame-level accuracy.
The user can click a frame in the grid on the left to edit the faces in this frame. The buttons on top of the grid also allows the user to delete or import single frames as images.
In the panel on the right, the user could re-detect faces for the chosen frame. The user can also select a face in the frame by clicking the boxes on the image preview or the columns in the list below. They can delete the selected face or look at it in details by selecting a "aligning display" mode. This image shows the three modes of aligning display: landmark lines (left), landmark points (middle), and aligned face (right):
The difference between aligned face and the original picture is mild here because the shown image is frontal.
The Overview panel offers an interface to quickly review all attributes of the loaded faces in current network. The user can assign any attributes of Point type as the network display positioning attribute here, but updating the positioning attribute will not automatically replace existing clusters.
The list display below shows all attributes of all faces in the loaded network (aka the active network of the project). Except for the first six attributes, all attributes are editable in this list. By double clicking an attribute, a text box will pop up below the list view and the user can input a new value for the attribute in the text box. If the input value is not properly formatted for the expected data type, however, the value will not be updated.
The Overview panel also provides several utilities in the right-top toolbar including:
(1) Export Full Face Images: to save all face images at full resolution (the original resolution cropped from the video file). This function requires that the loaded media file is stored at the original location and has been unchanged since the creation of current network.
(2) Export CSV: to save the network as a CSV file. Three types of CSV outputs are supported: empty template (a table with the attribute headers only), filled template (a table with attribute headers and sample inputs), and full table (a table with all attributes and their values).
(3) Conditional Selection: to allow the filtering of faces based on the values and integrity of an attribute.
(4) Disable Face (
): to disable all selected faces.
(5) Edit Selected: to assign values to an attribute for currently selected faces. This values can be a constant or a variable borrowed from other attributes.
(6) Delete Face (
): Remove all selected faces from network (irreversible).
(7) Create Attribute: (7.1) Facenet512: to generate a 512-D vector for all faces that are not deactivated (7.2) Custom CoreML: to load a CoreML model from device and use it to analyze the faces (7.3) T-SNE: to perform dimension reduction on a vector attribute that has more than 2 dimensions and generate a 2D Point attribute (or 1D decimal and 3D vector) for the faces based on the selected vector (can be slow; requires that a vector has been selected in the attribute header row) (7.4) Import CSV: to import values from a CSV file (7.5) Create Empty: to create an empty attribute for the current network and (optionally) assign a default value for this attribute.
(8) Remove Attribute: to remove the attribute selected in the attribute header row.
Multi-network projects can be instable in current version
The project panel is typically used for multi-network projects. While all projects are one-network be default, this panel allows the user to import other networks into a project. Other networks are all saved under the path Documents/Face Cluster Toolkit by default as folders. The support for multiple networks is to enhance the timeline visualization component, which can be useful for reducing the demand of computational power for long videos. By dividing a long video into clips, importing them as separate networks, and combining them in the project, the user can perform face clustering for a long video while keeping each network at acceptable size.
After selecting a network, the user can also review all clusters in that network. Selecting a cluster displays all faces it contains. In the timeline, all clusters with the same name from different networks will be identified as pointing to the same character.
Face texture display in timeline can be instable in current version
The timeline panel provides a quick visualization for the statistics. In the current version this component can be unstable, and exporting the data as CSV to be visualized with more established tools such as Plotly and SPSS is recommended.
The timeline panel is useful for researching certain patterns in relation to the video's structure. The X-Axis option is specifically designed to enhance the visualization of multi-network projects. The merge mode means that all faces from all networks in the project are placed at the exact x location indicating their appearance in the timeline of the original video file. The collapsed mode will collapse all networks based on their order stored in the project file. This order can be changed in the Project view. The Active mode means that only faces in the active network (the one being edited in the Network and Overview sections) will be places in the timeline, whilst other faces are places left to the timeline before media time 00:00.
| Referenced Projects | Content |
|---|---|
| Facenet | Facial attribute recognition model (Facenet512) |
| MTCNN-Caffe | Face alignment models (MTCNN) |
| Swift-TSNE | T-SNE implemented for Swift (CPU) |
Manually adding jpg and jpeg files with the same file name will result in unexpected errors with face image identification. Using the import utility in the Frame view will avoid such issue by forcing all jpeg files to be imported as jpg format.
Renaming clusters in the project page may cause the application to crash, but the renaming will be automatically saved and can be resumed after reloading.
- Improve T-SNE implement
- Clean unused codes
- Improve the stability of multi-network project editing