This README.txt file was generated on <20210707> by ------------------- GENERAL INFORMATION ------------------- 1. Title of Dataset: Supporting data for "Development of Generative Adversarial Tri-model (GAT model) Methods for Multi-modal Sensing" 2. Author Information First Author Contact Information Name: Song Wang Faculty: Engineering Email: songwang@connect.hku.hk --------------------- DATA & FILE OVERVIEW --------------------- Directory of Files: Human Motion Recognition with 3D CNN A. Filename: train data maker.py Short description: This script file is used to generate the "train_data.npy" and "train_label.npy" for network training purpose. B. Filename: test data maker.py Short description: This script file is used to generate the "test_data.npy" and "test_label.npy" for network test purpose. C. Filename: motion recognition.py Short description: This script file is used to define and train the 3D CNN. D. Filename: 180408_d3cnn.pkl Short description: This file stores the trained network, with the accuracy shown in the thesis. Directory of Files: Fisheye Camera Calibration A. Filename: wide 43 calibration with photo.py Short description: This script file is used to calibrate the fisheye camera based on the checkboard images in the same directory. B. Filename: calibration_data1920x1440.npz Short description: This file stores the fisheye camera calibration result, namely, camera parameters. Directory of Files: GAT Model for Sensor Calibration/balance sensor calibration raw data A. Filename: data maker.py Short description: This script file is used to tranform the experiment image patches in the same directory to numpy format data "balance sensor experimental pixel values for training.npy", "balance sensor experimental pressure values for training.npy", "balance sensor experimental pixel values for test.npy", and "balance sensor experimental pressure values for test.npy". "balance sensor experimental pixel values for training.npy" contains all the image patches used for training, so it has size 800x225. This data is used as input during training. "balance sensor experimental pressure values for training.npy" contains the experimental pressure values corresponding to those training image patches, so it is one-dimensional and has length 800. This data is used as label during training. "balance sensor experimental pixel values for test.npy" contains all the image patches for test, so it has size 190x225. This data is used as input during test. "balance sensor experimental pressure values for test.npy" contains the experimental pressure values corresponding to those test image patches, so it is also one-dimensional and has length 190. This data is used as label during test. The experiment image patches are named as such: There are 5 groups of repeated experiment. If the image patch is in group 1 and captured under pressure value 0.2kg/cm^2, then this image patch is named as "1_0d2kg_cm2_calibrated.png". The rest image patches are named in the same manner. Here the suffix "_calibrated" means the image patches are undistorted with the fisheye camera calibration result. Directory of Files: GAT Model for Sensor Calibration/skin sensor calibration raw data A. Filename: FOV_size_extraction.m Short description: This script file is used to calculate the FOV size of the experiment images in the same directory. The experiment images are named as such: There are also 5 groups of repeated experiment. If the image is in group 1 and captured under contact force of 7gf, then this image is named as "7g_1.jpg". The rest images are named in the same manner. The extracted FOV size values, along with the corresponding contact force values are stored in the "experiment data.mat" file. B. Filename: transfer experiment data to npy form.py Short description: This script file is used to transform the experiment data in the "experiment data.mat" file into numpy format data "skin sensor experimental FOV size values.npy" and "skin sensor experimental contact force values.npy". "skin sensor experimental FOV size values.npy" contains all possible FOV size values corresponding to all experimental FOV images, so it has size 65x5. This data is used as input. "skin sensor experimental contact force values.npy" contains the experimental contact force values (in unit gf) corresponding to all FOV images, so it is one-dimensional and has length 65. This data is used as label. Directory of Files: GAT Model for Sensor Calibration A. Filename: balance sensor ideal pixel values for calculating pressure-pixel relationship.npy Short description: As stated in the thesis, the way to calculate the pressure value corresponding to some pixel value from network is simply setting all 225 inputs equal to this pixel value. Therefore, for convenience, we made the "balance sensor ideal pixel values for calculating pressure-pixel relationship.npy", which has size 180x225. This file contains 180 noise-free image patches with identical pixel values inside each image patch. The pixel values inside the first image patch are all 0. The pixel values inside the second image patch are all 1 and so on. The pixel values inside the last image patch are all 179. If we input these man-made image patches into network, we can easily generate the pressure-pixel relationship for pixel value from 0 to 179. B. Filename: GAT model with Type 1 network for balance sensor calibration.py Short description: This script file generates a pressure-pixel relationship from the Type 1 network with the GAT model. "Type 1 net_params transferred from balance sensor network.pkl" contains the network parameters from the trained Type 1 network, for transfer learning purpose. C. Filename: GAT model with Type 2 network for balance sensor calibration.py Short description: This script file generates a pressure-pixel relationship from the Type 2 network with the GAT model. "Type 2 net_params transferred from balance sensor network.pkl" contains the network parameters from the trained Type 2 network, for transfer learning purpose. D. Filename: brightness_pressure relationship.csv Short description: This spreadsheet contains the trained pressure-pixel relationship from the Type 2 network with the GAT model. E. Filename: GAT model with Type 1 network for skin sensor calibration.py Short description: This script file generates a force-FOV relationship from the Type 1 network with the GAT model. F. Filename: GAT model with Type 2 network for skin sensor calibration.py Short description: This script file generates a force-FOV relationship from the Type 2 network with the GAT model. G. Filename: GAT model with Type 1 network and transfer learning for skin sensor calibration.py Short description: This script file generates a force-FOV relationship from the Type 1 network with the GAT model and transfer learning. H. Filename: GAT model with Type 2 network and transfer learning for skin sensor calibration.py Short description: This script file generates a force-FOV relationship from the Type 2 network with the GAT model and transfer learning. I. Filename: Type1 net_params from skin sensor calibration.mat Short description: This file stores the Type 1 network parameters for the force-FOV relationship trained with the GAT model and transfer learning, for skin elasticity measurement purpose. Directory of Files: Comprehensive Skin Properties Analysis/experiment analysis for skin pore size and roughness/experiment of skin* A. Filename: calculate_skin_pore_size.m Short description: This script file defines the matlab function to calculate skin pore size. B. Filename: calculate_skin_roughness.m Short description: This script file defines the matlab function to calculate skin roughness. C. Filename: skin_properties_analysis.m Short description: This script file invokes the "calculate_skin_pore_size.m" and "calculate_skin_roughness.m" to calculate the skin pore size and roughness from those captured images in the same directory. Directory of Files: Comprehensive Skin Properties Analysis/experiment analysis for skin elasticity A. Filename: calculate_skin_elasticity.m Short description: This script file defines the matlab function to calculate skin elasticity. B. Filename: cheek_elasticity_analysis.m Short description: This script file invokes the "calculate_skin_elasticity.m" to calculate the skin elasticity of cheek from the captured images of cheek in the same directory. C. Filename: eyebag_elasticity_analysis.m Short description: This script file invokes the "calculate_skin_elasticity.m" to calculate the skin elasticity of eyebag from the captured images of eyebag in the same directory. D. Filename: mouthcorner_elasticity_analysis.m Short description: This script file invokes the "calculate_skin_elasticity.m" to calculate the skin elasticity of mouthcorner from the captured images of mouthcorner in the same directory. E. Filename: temple_elasticity_analysis.m Short description: This script file invokes the "calculate_skin_elasticity.m" to calculate the skin elasticity of temple from the captured images of temple in the same directory. F. Filename: Type1 net_params from skin sensor calibration.mat Short description: This file stores the Type 1 network parameters for the force-FOV relationship trained with the GAT model and transfer learning, which is loaded in the "calculate_skin_elasticity.m". Directory of Files: GAT Model for Human Balancing Evaluation A. Filename: calibration_data1920x1440.npz Short description: This file stores the fisheye camera calibration result, namely, camera parameters. B. Filename: single video calibration.py Short description: This script file is used to tranform the original distorted videos into undistorted ones. The generated videos will be in grayscale and named with suffix "_calibrated". C. Filename: pressure_brightness.mat Short description: This file stores the trained pressure-pixel relationship from the Type 2 network with the GAT model, same as that in the "brightness_pressure relationship.csv". D. Filename: solve human motion with GAT model.py Short description: This script file solves human motion with the GAT model for the four human experiments in Chapter 5 of the thesis. The used balance sensor data is stored in the "videos" subfolder. Additional Notes on File Relationships, Context, or Content (for example, if a user wants to reuse and/or cite your data, what information would you want them to know?): N/A File Naming Convention: N/A ----------------------------------------- DATA DESCRIPTION FOR: N/A ----------------------------------------- 1. Number of variables: N/A 2. Number of cases/rows: N/A 3. Missing data codes: N/A 4. Variable List A. Name: N/A Description: Value labels if appropriate B. Name: N/A Description: Value labels if appropriate -------------------------- METHODOLOGICAL INFORMATION -------------------------- 1. Software-specific information: N/A Name: N/A Version: N/A System Requirements: N/A Open Source? (Y/N): N/A (if available and applicable) Executable URL: N/A Source Repository URL: N/A Developer: N/A Product URL: N/A Software source components: N/A Additional Notes(such as, will this software not run on certain operating systems?): N/A 2. Equipment-specific information: N/A Manufacturer: N/A Model: N/A (if applicable) Embedded Software / Firmware Name: N/A Embedded Software / Firmware Version: N/A Additional Notes: N/A