Language Models for Document Understanding ; Modèles de Langages pour les Compréhension de Documents

Every day, an uncountable amount of documents are received and processed by com-panies worldwide. In an effort to reduce the cost of processing each document, thelargest companies have resorted to document automation technologies. In an idealworld, a document can be automatically processed without any human intervention:its content is read, and information is extracted and forwarded to the relevant ser-vice. The state-of-the-art techniques have quickly evolved in the last decades, fromrule-based algorithms to statistical models. This thesis focuses on machine learningmodels for document information extraction.Recent advances in model architecture for natural language processing haveshown the importance of the attention mechanism. Transformers have revolution-ized the field by generalizing the use of attention and by pushing self-supervisedpre-training to the next level. In the first part, we confirm that transformers withappropriate pre-training were able to perform document understanding tasks withhigh performance. We show that, when used as a token classifier for informationextraction, transformers are able to exceptionally efficiently learn the task comparedto recurrent networks. Transformers only need a small proportion of the trainingdata to reach close to maximum performance. This highlights the importance ofself-supervised pre-training for future fine-tuning.In the following part, we design specialized pre-training tasks, to better preparethe model for specific data distributions such as business documents. By acknowl-edging the specificities of business documents such as their table structure and theirover-representation of numeric figures, we can target specific skills useful for themodel in its future tasks. We show that those new tasks improve the model’s down-stream performances, even with small models. Using this pre-training approach,we are able to reach the performances of significantly bigger models without anyadditional cost during finetuning or inference.Finally, in the last part, we address ...