Knowledge Technology Corpora

This dataset is composed of Youtube videos which are about a minute in length and are annotated taking into consideration a continuous emotional behavior. The videos were selected using a crawler technique that uses specific keywords based on long-term emotional behaviors such as "monologues", "auditions", "dialogues" and "emotional scenes".

After the videos were selected, we created an algorithm to identify if the video has at least two different modalities which contribute for the emotional categorization: facial expressions, language context, and a reasonably noiseless environment. We selected a total of 420 videos, totaling around 10 hours of data.

Annotations and Labels

We used the Amazon Mechanical Turk platform to create an utterance-level annotation for each video, exemplified in the Figure below. We assure that for each video we have a total of 5 different annotators. To make sure that the contextual information was taken into consideration, the annotator watched the whole video in a sequence and was asked, after each utterance, to annotate the arousal and valence of what was displayed. We provide a gold standard for the annotations. This results in trustworthy labels that are truly representative of the subjective annotations from each of the annotators, providing an objective metric for evaluation. That means that for each utterance we have one value of arousal and valence. We then calculate the Concordance Correlation Coefficient (CCC) for each video, which represents the correlation between the annotations and varies from -1 (total disagreement) to 1 (total agreement).

Evaluation Protocol

We release the dataset with the gold standard for arousal and valence as well as the individual annotations for each reviewer, which can help the development of different models. We calculate the final CCC against the gold standard for each video. We also distribute the transcripts of what was spoken in each of the videos, as the contextual information is important to determine gradual emotional change through the utterances. The participants are encouraged to use crossmodal information in their models, as the videos were labeled by humans without distinction of any modality.
We also make available to the participating teams a set of scripts which will help them to pre-process the dataset and evaluate their model during the training phase.

References 

Barros, P., Churamani, N., Lakomkin, E., Siqueira, H., Sutherland, A., & Wermter, S. (2018). The omg-emotion behavior dataset. arXiv preprint arXiv:1803.05434. 

More Information

To have access to this corpus, please visit our website:   https://github.com/knowledgetechnologyuhh/OMGEmotionChallenge

We recorded 60 object-hand interactions, at 30 frames per seconds from the perspective of the NICO robot. 

NICO (Neuro-Inspired COmpanion) is a developmental robot, which is used in research on multi-modal human-robot interaction and neuro-cognitive, and includes two HD RGB cameras and interaction capabilities of a 3.5-year-old child. In interactions with objects, the robot's hands are naturally introducing huge occlusions, making the tracking of object-manipulation-effects from the visual perspective difficult.

The recordings include different push, pull, grasp, and lift actions on a broad range of toy objects that show diverse behaviour when interacted with, such as rolling away, bouncing on the table, or changing their morphology. Compared to other SOTA object-tracking datasets, the conditions on object scale and in-plane-rotation are particularly difficult.

Annotations

The dataset is annotated with a ground truth for the object position (rectangular bounding box) with object-tracking tasks.

License:

This corpus is distributed under the Creative Commons CC BY-NC-SA 3.0 DE license. If you use this corpus, you have to agree with the following items:

• To cite our reference in any of your papers that make any use of the database. The references are provided at the end of this page.
• To use the corpus for research purpose only.
• To not provide the corpus to any second parties.

Reference:

When using this dataset please cite:

Heinrich, S., Springstübe, P., Knöppler, T., Kerzel, M., & Wermter, S. (2019). Continuous convolutional object tracking in developmental robot scenarios. Neurocomputing, Vol. 342, pp. 137-144.

Data:

https://todo

 

Gestures constitute a crucial element in human communication, as well as in human-robot interaction, thus, gesture recognition has been a field of particular interest in computer science. More specifically, dynamic gesture recognition is a challenging task, since it requires the accurate detection of the body parts involved in the gesture, their tracking and the interpretation of their sequential movement.

To evaluate our systems on a dynamic command gesture recognition task, we designed, recorded and organized the TsironiGR corpus. This corpus contains nine command gestures for Human-Robot Interaction (HRI): Abort, Circle, Hello, No, Stop, Turn Right, Turn Left, and Warn. Six different subjects were recorded, each one performing at least ten times the same gesture. We recorded a total of 543 sequences. Each of the gesture sequences is segmented and labeled with one of the nine classes. The gestures were captured by an RGB camera with a resolution of 640x480, recorded with a frame rate of 30 FPS.

The gesture descriptions are provided below:

• Abort - 57 Sequences -It is a gesture that requires the motion of the hand in front of the throat with the palm facing downwards.
• Circle - 60 Sequences - This gesture is a cyclic movement starting from the shoulder with the arm and the index finger being stretched out. The movement is taking place in front of the body of the user while he/she is facing the direction of the capturing sensor.
• Hello - 59 Sequences - As its name denotes, it is the typical greeting gesture, with the hand waving while facing the direction of the robot. The waving starts from the elbow rotation.
• No - 62 Sequences - To perform this gesture the arm and the index finger need to be stretched out towards the direction of the robot and then the repeating wrist rotation alternately to the right and the left.
• Turn Left - 62 Sequences - As the name of gesture denotes, the arm is pointing to the left.
• Turn Right - 60 Sequences - In the same sense as Turn Left, this gesture consists of the arm pointing to the right.
• Stop - 60 Sequences - The gesture includes raising the arm in front of the body with the palm facing the robot.
• Turn - 63 Sequences - This gesture is a cyclic movement starting with a rotation from the elbow including the simultaneous rotation of the wrist with the index finger stretched pointing downwards. The signature of this movement seems like a circle.
• Warn - 60 Sequences - This gesture firstly requires raising the hand in front of the body having an angle between the upper and the lower arm greater than 90 degrees and then rotating the elbow while the palm is being stretched out.

License:

This corpus is distributed under the Creative Commons CC BY-NC-SA 3.0 DE license. If you use this corpus, you have to agree with the following itens:

• To cite our reference in any of your papers that make any use of the database. The references are provided at the end of this page.
• To use the corpus for research purpose only.
• To not provide the corpus to any second parties.

Contact:

To have access to this corpus, please send an e-mail with your name, affiliation and research summary to: Pablo Barros

Reference:

Tsironi, E., Barros, P., Weber, C., & Wermter, S. (2017). An analysis of convolutional long short-term memory recurrent neural networks for gesture recognition. Neurocomputing, 268, 76-86.

To evaluate computational models in a realistic human-robot interaction scenario, we designed and recorded a database using the camera of a small, 58cm tall, NAO robot with four different hand postures. 

For each hand posture between 400 and 500 images where collected, each one with a dimension of 640x480 pixels.  In each image, the hand was is present in different
positions, not always in the centralized, and sometimes with occlusion of some
fingers.

Evaluation Protocol

A random training, validation and testing split must be generated using 70% of the images in each posture for training the model and 30% as testing. We calculate the mean and standard deviation of the F1-score for 30 experiments.

License:

This corpus is distributed under the Creative Commons CC BY-NC-SA 3.0 DE license. If you use this corpus, you have to agree with the following items:

• To cite our reference in any of your papers that make any use of the database. The references are provided at the end of this page.
• To use the corpus for research purpose only.
• To not provide the corpus to any second parties.

References:

Jirak, D., Wermter, S. (2018). Sparse Autoencoders for Posture Recognition. In Proceedings of International Joint Conference on Neural Networks,  (pp. 2359-2548)   

Barros, P., Magg, S., Weber, C., & Wermter, S. (2014). A multichannel convolutional neural network for hand posture recognition. In International Conference on Artificial Neural Networks (pp. 403-410) 

Contact:

To have access to this corpus, please send an e-mail with your name, affiliation and research summary to: Pablo Barros