System and method for the detection and counting of repetitions of repetitive activity via a trained network

11727,725

17/173720

February 11, 2021

A technique and system for counting the number of repetitions of approximately the same action in an input video sequence using 3D convolutional neural networks is disclosed. The proposed system runs online and not on the complete video. It analyzes sequentially blocks of 20 non-consecutive frames. The cycle length within each block is evaluated using a deep network architecture and the information is then integrated over time. A unique property of the disclosed method is that it is shown to successfully train on entirely synthetic data, created by synthesizing moving random patches. It therefore effectively exploits the high generalization capability of deep neural networks. Coupled with a region of interest detection mechanism and a suitable mechanism to identify the time scale of the video, the system is robust enough to handle real world videos collected from youtube and elsewhere, as well as non-video signals such as sensor data revealing repetitious physical movement.

Wolf, Lior (Herzlia, IL); Levy, Ofir (Jerusalem, IL)

1. A system for detecting and counting repetitive activity over time, comprising: a signal input to transfer, from outside to inside the system, signal information of synthetic unrealistic data in a series of training examples from which repetitive activity is detected and counted; wherein a processor utilizes a deep neural network classifier to learn from said plurality of training examples; one or more detectors for detecting repetitive activity from a plurality of time windows of fixed length, where the time windows contain said signal information; said processor utilizes said deep neural network classifier for determination of and outputting an estimation of a cycle length of each repetition of the repetitive activity within each time window; and at least two registers to integrate the estimation provided by the processor and to produce a count of the number of the repetitions.

2. The system of claim 1 , wherein, the time windows have more than one fixed length.

3. The system of claim 2 , wherein uncertainty in estimating the cycle length of the repetitive activity is used to select an appropriate window length for producing the count of the activity.

4. The system of claim 1 , wherein the processor learns from a plurality of training examples.

5. The system of claim 1 , wherein the plurality of training examples includes synthetic unrealistic data.

6. A system for discerning whether repetitive activity is occurring over time, comprising: a signal input to transfer, from outside to inside the system, signal information of synthetic unrealistic data from which it may be discerned whether repetitive activity has occurred; one or more detectors for detecting repetitive activity from a plurality of time windows of fixed length, where the windows contain said information; and a processor that utilizes a deep neural network classifier for determination of and outputting an estimation of a cycle length of each repetition of the repetitive activity within each time window.

7. The system of claim 6 , wherein the detection of whether repetitive activity is occurring is based on uncertainty in estimating a cycle length as determined by the processor.

8. The system of claim 6 , wherein said processor outputs an estimation of a cycle length of said repetitive activity.

9. The system of claim 8 , wherein said processor determines said estimation of said cycle length by classifying said repetitive activity into a classification that is characterized by said cycle length.

10. The system of claim 1 , wherein said processor determines said estimation of said cycle length by classifying said repetitive activity into a classification that is characterized by said cycle length.

11. The system of claim 1 , wherein said signal information comprises a video stream.

12. The system of claim 6 , wherein said signal information comprises a video stream.

13. The system of claim 11 , wherein said registers produce said count online while said video stream is still incomplete.

14. The system of claim 12 , wherein said processor produces said outputs online while said video stream is still incomplete.

15. The system of claim 11 , wherein said processor automatically detects, within each frame of said video stream, a region-of-interest containing said repetitive activity.

16. The system of claim 12 , wherein said processor automatically detects, within each frame of said video stream, a region-of-interest containing said repetitive activity.

17. The system of claim 1 , wherein said processor is configured for determining said count-start and count-stop based on an entropy of probabilities assigned by said deep neural network classifier.

18. The system of claim 6 , wherein said processor is configured for determining whether a repetitive activity is occurring within each time window based on an entropy of probabilities assigned by said deep neural network classifier.

6795567 September 2004 Cham
7035435 April 2006 Li
7526101 April 2009 Avidan
7675655 March 2010 Marshall
7899208 March 2011 Kondo
8619187 December 2013 Garg
8699758 April 2014 Almbladh
9183466 November 2015 Siskind
9269158 February 2016 Ukil et al.
10460194 October 2019 Wolf et al.
20030018475 January 2003 Basu
20040258307 December 2004 Viola
20060018516 January 2006 Masoud
20060112028 May 2006 Xiao
20060165258 July 2006 Avidan
20090175496 July 2009 Kondo
20120008819 January 2012 Ding
20120008836 January 2012 Bobbitt
20120128212 May 2012 Almbladh
20120169890 July 2012 Tian
20130016239 January 2013 Cho
20140185876 July 2014 Almbladh
20140369596 December 2014 Siskind
20170017857 January 2017 Wolf et al.
102281385 December 2011
102281385 March 2013
103761746 April 2014
106663325 May 2017
WO 2015/132665 September 2015
WO 2019/000412 January 2019

International Preliminary Report on Patentability dated Sep. 22, 2016 From the International Bureau of WIPO Re. Application No. PCT/IB2015/000728. cited by applicant
International Search Report and the Written Opinion dated Sep. 21, 2015 From the International Searching Authority Re. Application No. PCT/IB2015/000728. cited by applicant
Notice Of Allowance dated Jun. 19, 2019 From the US Patent and Trademark Office Re. U.S. Appl. No. 15/124,047. (9 pages). cited by applicant
Notice of Allowance dated Sep. 29, 2020 from the US Patent and Trademark Office Re. U.S. Appl. No. 16/666,390. (12 pages). cited by applicant
Official Action dated Feb. 25, 2019 From the US Patent and Trademark Office Re. U.S. Appl. No. 15/124,047. (12 pages). cited by applicant
Official Action dated Sep. 25, 2018 From the US Patent and Trademark Office Re. U.S. Appl. No. 15/124,047. (15 Pages). cited by applicant
Official Action dated Jul. 8, 2020 From the US Patent and Trademark Office Re. U.S. Appl. No. 16/666,390. (9 pages). cited by applicant
Second Notice of Allowance dated Oct. 20, 2020 from the US Patent and Trademark Office Re. U.S. Appl. No. 16/666,390. (8 pages). cited by applicant

edspgr.11727725