Neural network systems and methods for target identification from text

11017,177

16/454771

June 27, 2019

Neural network systems are provided that comprise one or more neural networks. The first neural network can comprise a convolutional neural network (CNN) long short-term memory (LSTM) architecture for receiving a primary data set comprising text messages and output a primary data structure comprising a text pattern-based feature. The second neural network can comprise a CNN architecture for receiving a secondary data sets derived from the primary data set and output a plurality of secondary data structures. The third neural network can combine the data structures to produce a combined data structure, and then process it to produce a categorized data structure comprising the text messages assigned to targets. The primary data set can comprise hate speech and the categorized data structure can comprise target categories, for example, hate targets. Methods of operating neural network systems and computer program products for performing such methods are also provided.

CONDUENT BUSINESS SERVICES, LLC (Florham Park, NJ, US)

Conduent Business Services, LLC (Florham Park, NJ, US)

1. A neural network system comprising: a computer readable medium comprising: a first neural network comprising a convolutional neural network (CNN) long short-term memory (LSTM) architecture, the first neural network configured to receive a primary data set comprising text messages and output a primary data structure, a second neural network comprising a CNN architecture, the second neural network configured to receive a plurality of secondary data sets derived from the primary data set and output a plurality of secondary data structures, wherein the plurality of secondary data sets comprises a graph-based feature, a semantic feature, or both, and a third neural network comprising a deep neural network (DNN) architecture, the third neural network configured to: combine the primary data structure and the plurality of second data structures to produce a combined data structure, and process the combined data structure to produce a categorized data structure comprising the text messages assigned to targets; and a processor configured to operate the first, second, and third neural networks.

2. The neural network system of claim 1 , wherein at least one of the first, second, and third neural networks comprise a max-pooling layer, a dropout layer, or both.

3. The neural network system of claim 1 , wherein the plurality of secondary data sets comprises at least two graph-based features.

4. The neural network system of claim 1 , wherein the second neural network comprises a plurality of channels, and each channel of the plurality of channels comprises a different data set of the plurality of secondary data sets.

5. The neural network system of claim 4 , wherein the second neural network comprises a convolution layer and a filter length of the convolution layer differs between channels.

6. The neural network system of claim 5 , wherein: the plurality of channels comprises at least three channels comprising a first channel, a second channel, and a third channel; the filter length of the convolution layer is different in each of the three channels; the plurality of secondary data sets comprising a first data set, a second data set, and a third data set; the first channel comprises the first data set, the second channel comprises the second data set, and the third channel comprises the third data set; and the first data set comprises a first graph-based feature, the second data set comprises a second graph-based feature, and the third data set comprises a semantic feature.

7. The neural network system of claim 1 , wherein: the text messages comprise hate speech; the categorized data structure comprises a plurality of target categories; and the target categories comprise hate targets; and the hate targets comprise two or more of behavior, religion, ethnicity, class, nationality, race, sexual orientation, disability, gender, and morphology.

8. The neural network system of claim 1 , wherein: the text messages comprise language relating to an event, a product, an individual, a hobby, music, a location, an activity, a health issue, a utility issue, a safety issue, a weather phenomenon, a complaint, or an emotion, or any combination thereof; the categorized data structure comprises a plurality of target categories; and the target categories comprise events, products, individuals, hobbies, music genres, songs, locations, activities, health issues, utility issues, safety issues, weather phenomena, complaints, or emotions, or any combination thereof.

9. The neural network system of claim 1 , wherein the output of the first neural network comprises a text pattern-based feature.

10. The neural network system of claim 1 , wherein the third neural network is configured as a classifier comprising a plurality of binary classifiers configured to operate as a one versus all classifier.

11. The neural network system of claim 1 , further comprising a user interface configured to enable a user to interact with the first, second, and third neural networks.

12. A method of operating a target identification system, the method comprising: receiving a primary data set comprising text messages; generating a plurality of secondary data sets from the primary data set, the generation comprising production of a graph-based feature data set and a semantic feature data set; processing the primary data set using a first convolutional neural network (CNN) comprising long short-term memory (LSTM) to produce a primary data structure comprising a text pattern feature; processing the plurality of secondary data sets using a second CNN to produce a plurality of secondary data structures; combining the primary data structure and the plurality of secondary data structures to produce a combined data structure; and processing the combined data structure using a deep neural network (DNN) configured as a classifier to output a categorized data structure comprising the text messages assigned to targets.

13. The method of claim 12 , wherein the processing of the primary data set comprises embedding the primary data set in the first CNN, and the processing of the plurality of secondary data sets comprises embedding the plurality of secondary data sets in the second CNN.

14. The method of claim 12 , wherein the combining comprises concatenating the primary data structure and the plurality of secondary data structures, and the method further comprises flattening the primary data structure and the plurality of secondary data structures prior to the concatenation.

15. The method of claim 12 , wherein the generating comprises: constructing a graph comprising nodes corresponding to words in the text messages and edges connecting nodes based on occurrence within a predetermined distance; identifying words biased by predetermined keywords in the graph to produce the graph-based feature data set, the graph-based feature data set being a first graph-based feature data set; and identify words having a high load determined by a number of shortest path passes using a node corresponding to a word to produce a second graph-based feature data set of the secondary data set.

16. The method of claim 12 , wherein the second CNN comprises a plurality of channels comprising a first channel configured to process the graph-based feature data set and a second channel configured to process the semantic feature data set; the method further comprising applying a different length filter to each filter.

17. The method of claim 12 , wherein: the text messages comprise language relating to hate, an event, a product, an individual, a hobby, music, a location, an activity, a health issue, a utility issue, a safety issue, a weather phenomenon, a complaint, or an emotion, or any combination thereof; the categorized data structure comprises a plurality of target categories; and the target categories comprise hate targets, events, products, individuals, hobbies, music genres, songs, locations, activities, health issues, utility issues, safety issues, weather phenomena, complaints, or emotions, or any combination thereof.

18. The neural network system of claim 1 , wherein the plurality of secondary data sets comprises a graph-based feature data set, or a semantic feature data set, or both.

19. The neural network system of claim 1 , wherein the plurality of secondary data sets comprises a graph-based feature data set and a semantic feature data set.

20. A computer program product comprising a non-transitory computer readable medium, wherein the non-transitory computer readable medium stores a computer program code for operating a neural network system, wherein the computer program code is executable by one or more processors of an application server of the system to: receive a primary data set comprising text messages; generate a plurality of secondary data sets from the primary data set, the generation comprising production of a graph-based feature, a semantic feature, or both; process the primary data set using a first convolutional neural network (CNN) comprising long short-term memory (LSTM) to produce a primary data structure comprising a text pattern feature; process the plurality of secondary data sets using a second CNN to produce a plurality of secondary data structures; combine the primary data structure and the plurality of secondary data structures to produce a combined data structure; and process the combined data structure using a deep neural network (DNN) configured as a classifier to output a categorized data structure comprising the text messages assigned to targets.

8670526 March 2014 Clawson
9037464 May 2015 Mikolov et al.
9195990 November 2015 Hahn et al.
9311599 April 2016 Attenberg et al.
10255269 April 2019 Quirk
10521587 December 2019 Agranonik
2016/0078339 March 2016 Li
2016/0099010 April 2016 Sainath
2018/0025721 January 2018 Li
2019/0166670 May 2019 Alfier

Tian, Chujie, et al. “A deep neural network model for short-term load forecast based on long short-term memory network and convolutional neural network.” Energies 11.12 (2018) (Year: 2018). cited by examiner
Sosa, Pedro M. “Twitter sentiment analysis using combined LSTM-CNN models.” Eprint Arxiv (2017): 1-9. (Year: 2017). cited by examiner
Pitsilis, Georgios K., Heri Ramampiaro, and Helge Langseth. “Effective hate-speech detection in Twitter data using recurrent neural networks.” Applied Intelligence 48.12 (2018): 4730-4742. (Year: 2018). cited by examiner
Huang, Qiongxia, et al. “Deep sentiment representation based on CNN and LSTM.” 2017 International Conference on Green Informatics (ICGI). IEEE, 2017. (Year: 2017). cited by examiner
Badjatiya, P. et al., “Deep Learning for Hate Speech Detection in Tweets,” IW3C2 WWW 2017 Companion; Perth Australia ACM 978-1-4503-4914-7/17/04 pp. 759-760 (2017). cited by applicant
Davidson, T. et al., “Automated Hate Speech Detection and the Problem of Offensive Language,” 11th International Conference ICWSM pp. 512-515 (2017). cited by applicant
Del Vigna, F. et al., “Hate me, hate met not: Hate speech detection on Facebook,” ITASECI7; Venice, Italy pp. 86-95 (2017). cited by applicant
El Sherief, M. et al., “Hate Lingo: A Target-Based Linguistic Analysis of Hate Speech in Social Media,” Proc. of the 12th International AAAI Conf. on Web and Social Media Assoc. for the Advancement of Artificial Intell (www.aaai.org) pp. 42-51 (2018). cited by applicant
El Sherief, M. et al., “Peer to Peer Hate: Hate Speech Instigators and Their Targets,” Proc. of the 12th International AAAI Conf. on Web and Social Media (ICWSM) pp. 52-61 (2018). cited by applicant
Gamebäck, B. & Sikdar, U., “Using Convolutional Neural Networks to Classify Hate-Speech,” Proc. of the 1st Workshop on Abusive Language Online; Vancouver, Canada pp. 85-90 (2017). cited by applicant
Go, A. et al., “Twitter sentiment classification using distant supervision.” CS224N Project Report; Stanford 1(12) (2009). cited by applicant
Goh, K.-I. et al., “Universal Behavior of Load Distribution in Scale-Free Networks,” Physical Review Letters 87(27) 278701-1 to 278701-4 (Dec. 31, 2001). cited by applicant
Kalchbrenner, N. et al., “A Convolutional Neural Network for Modelling Sentences,” Proc. of the 52nd Annual Mtg of the Assoc. for Computational Linguistics; Baltimore, Maryland, pp. 655-665 (2014). cited by applicant
Kim, Y., “Convolutional Neural Networks for Sentence Classification.” arXiv preprint arXiv:1408.5882 (2014). cited by applicant
Mihalcea, R. & Tarau, P., “Textrank: Bringing order into text.” Proc. of the 2004 Conf. on Empirical Methods in Natural Language processing. (2004). cited by applicant
Newman, M. E. J., “Scientific Collaboration Networks. II. Shortest Paths, Weighted Networks, and Centrality,” Physical Review E 64:016132-1 to 016132-7 (2001). cited by applicant
Nobata, C. et al. “Abusive Language Detection in Online User Content.” Proc. of the 25th International Conf. on World Vide Web. International World Wide Web Conferences Steering Committee (2016). cited by applicant
Park, J.-H. & Fung, P., “One-Step and Two-Step Classification for Abusive Language Detection on Twitter,” arXiv preprint arXiv:1706.01206 (2017). cited by applicant
Silva, L. et al. “Analyzing the Targets of Hate in Online Social Media.” 10th International AAAI Conf. on Web and Social Media. (2016). cited by applicant
Tai, K. S. et al., “Improved Semantic Representations from Tree-Structured Long Short-Term Memory Networks,” arXiv preprint arXiv:1503.00075 (2015). cited by applicant
Wang, J. et al., “Dimensional Sentiment Analysis Using a Regional CNN-LSTM Model” Proc. of the 54th Annual Meeting of the Assoc. for Computational Linguistics (vol. 2: Short Papers) (2016). cited by applicant
White, S. & Smyth, P., “Algorithms for Estimating Relative Importance in Networks.” Proc. of the 9th ACM SIGKDD International Conf. on Knowledge Discovery and Data mining. ACM (2003). cited by applicant
Yin, W. & Schütze, H., “Multichannel Variable-Size Convolution for Sentence Classification,” arXiv preprint arXiv:1603.04513 (2016). cited by applicant
Zhang, H. et al., “YNU-HPCC at SemEval 2017 Task 4: Using a Multi-Channel CNN-LSTM Model for Sentiment Classification,” arXiv preprint arXiv:1603.04513 (2016). cited by applicant
Zhang, Z. et al. “Detecting Hate Speech on Twitter Using a Convolution-GRU Based Deep Neural Network,” European Semantic Web Conference. Springer Intl Pub AG; Cham, CH (2018). cited by applicant

edspgr.11017177