Artiwise-Nested Named Entity Recognition via Language Model Based Neural Architecture

Founded in 2015, Artiwise has been helping companies improve their customer satisfaction by providing actionable insights. By consolidating all customer communication channels in a single platform, Artiwise enables companies to understand the sentiment of their customers. Artiwise is the first company to measure customer satisfaction by analyzing the emotions of customers and offers tools such as root-cause analysis, aspect-based sentiment analysis, instant customer insights, category-based customer satisfaction scores, and periodic comparison reports to help companies identify the factors that contribute to customer dissatisfaction and churn. By providing a holistic view of customer feedback, Artiwise helps companies take a proactive approach improving the customer experience.

             

Technical/scientific Challenge

In natural language processing (NLP), information extraction (IE) is the process of converting unstructured textual data into a structured format in order to extract the information it contains. Entity extraction is one method of information extraction that aims to identify the entities present in a document. Named entity recognition is a common approach to entity extraction that identifies predefined entity classes in a text. These classes may be dependent on or independent of each other, and the goal of NER is to extract the relationships between the entities in the text, including only the information about the entities of interest. Most NER models focus on flat entities, which are concepts that can only be associated with one class, but nested entities, which are a natural part of language structure, can provide a more complete view of the information in a text. For example, a flat NER model might classify “Chelsea Football Club” as an organization, but in some contexts “Chelsea” may refer to a location. A model using nested entities would be able to capture this distinction. Flat NER models may have limitations in their ability to handle complex situations like this.

Solution

This research presents a method for identifying named entities in economic news articles written in Turkish. We used a deep learning model based on transformer-based language models and created a large dataset of annotated articles. The data required cleaning before it could be used, so we implemented a data-cleaning pipeline. Multiple annotators were able to label the data at the same time using our annotation tool, Artiwise Analytics, and a guide was used to ensure consistency in the annotations. The model was trained and evaluated on the TRUBA GPU clusters, achieving an 82% micro-F1 measure. These results suggest that the model is suitable for use in industrial settings and may be able to address various challenges in these domains. We were grateful for the assistance of Dr. Sefer Baday, our advisor, who provided valuable academic insight. The use of TRUBA’s GPU clusters was crucial to the success of the project.

Figure 1: An example for Nested-Ner problem[1]

Figure 2: Application UI exapmple. (https://www.bloomberght.com/celebi-hindistan-da-iki-ihaleye-katildi-2303994)

Figure 3: Application UI Example 2 (https://dailyhodl.com/2022/12/25/former-alameda-ceo-admits-to-using-ftx-customer-deposits-to-repay-loans-on-risky-investments-report)

Business impact

The current flat-NER models available to our customers can extract meaningful sequences from text, but they do not have the depth of understanding needed for more complex tasks. We have developed a model that can handle a variety of tasks and have tested it on two different datasets in our project. We believe this model can be used in various industries. It is integrated into the Artiwise Analytics platform, making it easy for users to upload their data, tag the relevant categories, and train the model using user-friendly interfaces. The model is based on a large language model and requires significant hardware resources, such as a GPU cluster, to train. Thanks to the support of EuroCC, we were able to test our model on a GPU-based cluster and have decided to invest in our infrastructure by adding more GPUs to increase our capabilities.

Benefits

  • Operational efficiency thanks to replacing human operations with machine-based systems.
  • Precise analysis than using the flat type of NER models.
  • Understanding a document is key to extending the operation to different departments in the banking domain.

Keywords

  • Keywords: Nested Ner, Named Entity Recognition, Machine Reading Comprehension, Information Extraction
  • Industry sector: Finance/Insurance, Automative, Public Services,
  • Technology: Natural Language Processing, Deep Learning, Machine Learning.

[1] Ju, Meizhi, Makoto Miwa, and Sophia Ananiadou. “A neural layered model for nested named entity recognition.” Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 2018.