In our Natural Language Processing (NLP) blog series, we analyzed the importance of NLP in customer care and how to do it in 4 fundamental steps. This blog focuses on an NLP technique called “Topic Modeling” to understand customer data. Read more to discover how you can use this technique to help your business deliver what your prospects and customers need.

Topic Modeling, what is it?

Imagine a scenario where a company frequently receives text documents such as emails, product reviews, or even customer complaints. It would be highly time-consuming for a single person to go through them and then define the underlying topics. A biased and error-prone analysis is therefore unavoidable. Fortunately, using Topic Modeling, you can make it faster and, to some extent, better. The primary output of this technique is that it structures unstructured data according to some latent patterns, called “topics.”

Benefits of using Topic Modeling

Topic Modeling combines topics into a single, understandable structure. It groups them into broader concepts that make sense for a particular business or issue. Let’s say you want to find your customers’ biggest challenges. However, you only have this unstructured data that you gather from combing through your customer service emails. Using Topic Modeling, you can structure this data and highlight your customers’ critical issues. In short, this technique helps you:

  • Mining relevant information at scale

  • Getting real-time analysis

  • Building a consistent understanding of data

The challenges

There exist several topic modeling techniques, such as Latent Semantic Indexing (LSA), Latent Dirichlet Allocation (LDA), Probabilistic Latent Semantic Analysis (PLSA) and Non-Negative Matrix Factorization (NMF). All the models mentioned above have their pros and cons. However, they all share specific challenges:

Choosing the optimal number of topics

All techniques require us to define how many topics we want to extract from our unstructured collection of documents. This requirement represents a big drawback as it is impossible to know the ideal number in advance. Indeed, if we knew how many topics we needed, we wouldn’t need to analyze our data at all. Thankfully, we can solve this problem by building a model for varying numbers of topics, typically from 1 to 100, and then measuring each model’s quality. Even though it is a brute force approach, it is still frequently used. However, since it is required to run the model multiple times, it is imperative to use techniques that can train fast. NMF fits in that category.

Evaluating the quality of the topics found

The second challenge is related to the first; it is necessary to use a metric to measure the quality of each model to find out which one is best. There is no consensus on which metric is best. Still, the most popular ones are the Silhouette index and the Pointwise Mutual Information (PMI). To remedy these flaws, it is wise to manually go through a sample of the data and define which documents belong together and which ones don’t. This approach helps us identify which models tend to agree with our judgment.

Having a “human- understandable” description of a topic

All the topic models work either with probabilities or linear algebra techniques, resulting in words becoming numbers and losing meaning. The model’s output is a collection of words, which the model recognized as significant for that topic. For instance, in a call center scenario, a topic could be described by the following terms: delivery, missing, reimbursement, item, lost. While it gives a general idea, it can sometimes be too abstract. Describing topics remains an active field of study, and multiple approaches are being investigated. One of them is the wikification algorithm, which connects the words to Wikipedia articles, which helps the user by providing more context.

It all starts with the data you collect…

Being data-driven is crucial to improve your business continually. Text data can help you improve your decision-making, but it can only do that once it has been structured. Many machine learning techniques can help you with that, and Topic Modeling is, without a doubt, one of them. By now, you should have a basic understanding of what Topic Modeling is and how it can help make sense of unstructured data. Do you want to know more about what Topic Modeling and NLP can offer to your business? Our team of Data Science experts is always ready to help you!