Human Activity Recognition with Metric Learning

Du Tran, Alexander Sorokin, and David Forsyth


This paper proposes a metric learning based approach for human activity recognition with two main objectives: (1) reject unfamiliar
activities and (2) learn with few examples. We show that our approach outperforms all state-of-the-art methods on numerous standard
datasets for traditional action classification problem. Furthermore, we demonstrate that our method not only can accurately label activities
but also can reject unseen activities and can learn from few examples with high accuracy. We finally show that our approach works well on
noisy YouTube videos.

Video demonstrations:

     


Source code: Motion context descriptor (MATLAB code with 2 sample sequences 64 MB)

Computed features: for UIUC1 (56MB), Weizmann (8MB), UMD (21MB), IXMAS (303MB)

Datasets

UIUC1

Section 1:

Walking: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,

Running: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,

Jumping: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,

Waving: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,

Jumping_jacks: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,

Clapping: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,

Jump_from_situp: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,

Raise_1_hand: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,

Stretching_out: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,

Turning: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,

Sitting_to_standing: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,

Crawling: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,

Pushing_up: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11,

Standing_to_sitting: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,

Annotations: forground masks, bounding boxes

Section 2:

Walking: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14,

Running: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14,

Jumping: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14,

Waving: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14,

Jumping_jacks: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14,

Clapping: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14,

Jump_from_situp: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14,

Raise_1_hand: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14,

Stretching_out: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14,

Turning: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14,

Sitting_to_standing: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14, 15,

Crawling: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14,

Pushing_up: 01, 02, 03, 04, 05, 06, 07, 08,

Standing_to_sitting: 01, 02, 03, 04, 05, 06, 07, 08, 09, 10, 11, 12, 13, 14,

Annotations: forground masks, bounding boxes

Section 3:

Walking: 01, 02, 03, 04, 05,

Running: 01, 02, 03, 04, 05,

Jumping: 01, 02, 03, 04, 05,

Waving: 01, 02, 03, 04, 05,

Jumping_jacks: 01, 02, 03, 04, 05,

Clapping: 01, 02, 03, 04, 05,

Jump_from_situp: 01, 02, 03, 04, 05,

Raise_1_hand: 01, 02, 03, 04, 05,

Stretching_out: 01, 02, 03, 04, 05,

Turning: 01, 02, 03, 04, 05,

Sitting_to_standing: 01, 02, 03, 04, 05,

Crawling: 01, 02, 03, 04, 05,

Pushing_up: 01, 02, 03, 04, 05,

Standing_to_sitting: 01, 02, 03, 04, 05,

Annotations: forground masks, bounding boxes

UIUC2

UIUC2: badmindton sequences (452 MB)


Publications
Du Tran, Alexander Sorokin, Human Activity Recognition with Metric Learning, ECCV08, Marseille, France.
Du Tran, Alexander Sorokin, David Forsyth, Human Activity Recognition with Metric Learning, UIUCDCS-R-2008-2952.