What are the main limitations of using LLMs for topic modeling in qualitative research, as discussed in this chapter?

The chapter identifies several main limitations: topic instability, where each execution produces different topics and assignments; hallucinations that fabricate plausible-looking but nonexistent topics; failure to retrieve topics actually present in the data; model bias reflected in the output; and a lack of genuine interpretability and contextual understanding. These limitations mean LLM-generated topics should be treated as a starting point and require careful human oversight.

According to the chapter, one main limitation is topic instability, which stems from the stochastic behavior of LLMs. Running the same code multiple times generates different sets of topics and descriptions, and topic assignment varies even when the same list of topics is provided. A second limitation is the risk of model hallucinations, where the model produces nonexistent topics that appear real and may describe topics based on top words that do not actually exist in the data. A third limitation is the risk of failing to retrieve topics that genuinely present in the dataset, meaning some real themes may be missed. A fourth limitation is the potential for model bias to appear in the output. Finally, the chapter emphasizes that LLMs lack interpretability and contextual understanding: while they can identify patterns, they do not have genuine interpretive abilities or human-like understanding, and they struggle with subjective meanings and contextual factors that are important in qualitative research.