Document and Figure Linking

Electronic documents are used to store and share information about scientific research across time and space. Most library systems have the ability to handle the network of referencing among documents while treating the document entity as a whole. The local relationships between different parts or components of these documents, however, is often not made explicit. This project is focused on extracting relationships between figures and text within each scientific documents.

We have first developed a dataset by utilizing the PMC public dataset. We parse the XML representation of the document from raw dataset to extract the figures, captions and direct reference sentences (sentences that have the label such as “Fig. 2”). The BRAT annotation tool is then used to clean up the references and accelerate the process of annotating indirect reference sentences that are not explicit in the system. Manual annotation of indirect and spatial references is still on-going.

Using this dataset, we are addressing performing sentence classification based on pre-trained BERT models. A multimodal transformer based model is also being explored for image captioning and image retrieval tasks.