Skip to content

About

Choosing the number of LDA topics with Louvain community detection on word co-occurrence graphs, plus Guided LDA seeded with the community keywords. Tested on PubMed, research abstracts and news.

Topics

Resources

Stars

0 stars

Watchers

1 watching

Forks

Latest commit

 

History

2 Commits

Folders and files

Repository files navigation

Topic Modelling with Word Co-occurrence Communities

Finds the number of topics in a text corpus with graph community detection and uses the result to drive LDA and Guided (seeded) LDA.

Choosing the number of topics k is one of the hardest parts of LDA. Coherence-score sweeps are slow and often pick a k that doesn't match the real structure of the data. This project builds a word co-occurrence network, splits it into communities with the Louvain algorithm, and uses:

  • the number of communities as the number of topics, and
  • the most central words in each community as topic labels and as seed words for Guided LDA.

Approach

Raw documents
   │
   ▼
Preprocessing ── tokenize · remove stopwords · keep nouns/verbs (POS filter)
   │             lowercase · strip punctuation · lemmatize · merge WordNet synonyms
   │             drop rare words · TF-IDF vocabulary filter (min_df=2, max_df=0.8)
   ▼
Word co-occurrence graph (words = nodes, co-occurring in a document = edge)
   │
   ▼
Louvain community detection ──► number of communities = k
   │
   ├──► Eigenvector centrality in each community ──► top-10 words per topic
   │
   ├──► LDA with k topics (number of topics from the communities)
   │
   ├──► LDA with k chosen by the best c_v coherence (baseline, k = 1…39)
   │
   └──► Guided LDA seeded with each community's top words

Datasets

Notebook Corpus Documents used
PubMed.ipynb PubMed abstracts from 4 search topics: deep learning, COVID-19, human connectome, virtual reality 1,000 (250 per topic)
Research topics.ipynb Research article abstracts (ID, TITLE, ABSTRACT) 1,000
Topic modelling Code.ipynb News articles: Harvard Dataverse, doi:10.7910/DVN/GMFCTR Full dataset
GLDA_Ex_5_RANLP.ipynb Preprocessed PubMed-domain corpus (experiment_5.pickle) 2,229

The datasets are not included in this repository.

Results

Number of topics found by community detection vs. by the best coherence score:

Corpus Louvain communities Best-coherence k
PubMed (4 known topics) 4 2
Research abstracts 6 3
News articles 9 30

On the PubMed corpus, where the true number of topics is known to be 4, community detection found exactly 4 clearly separated topics. The top words by eigenvector centrality were:

Community Top words
0 – Human connectome connectivity, brain, analysis, region, function, imaging, cortex
1 – Deep learning learning, model, network, resolution, method, data, image
2 – COVID-19 covid19, patient, disease, coronavirus, case, health, sarscov2
3 – Virtual reality reality, participant, vr, virtual, environment, time

Guided LDA

Seeding Guided LDA with the community keywords (seed_confidence=0.15) gave clean, separated topics. Example output:

Topic 0: electrode film pedot polymer electrochemical surface pedotpss layer high composite
Topic 1: brain network connectivity functional study patient region cortex control cortical
Topic 2: method data model learning network image feature deep performance neural

Repository structure

├── PubMed.ipynb                 # Community detection + LDA on PubMed abstracts
├── Research topics.ipynb        # Same pipeline on research article abstracts
├── Topic modelling Code.ipynb   # Same pipeline on news articles
└── GLDA_Ex_5_RANLP.ipynb        # Guided (seeded) LDA using community keywords

Getting started

Requirements

pip install pandas numpy scikit-learn nltk gensim networkx python-louvain pyLDAvis matplotlib seaborn
pip install guidedlda   # only needed for GLDA_Ex_5_RANLP.ipynb
import nltk
nltk.download(['stopwords', 'punkt', 'wordnet', 'averaged_perceptron_tagger'])

guidedlda is no longer maintained and may fail to install on recent Python versions. Python 3.7–3.9 is recommended for that notebook.

Run

  1. Download the dataset for the notebook you want to run.
  2. Change the pd.read_csv(...) path at the top of the notebook to point to your copy.
  3. Run all cells.

Tech stack

Python · NLTK · Gensim · scikit-learn · NetworkX · python-louvain · GuidedLDA · pandas

Author

Intesur Ahmed: GitHub

About

Choosing the number of LDA topics with Louvain community detection on word co-occurrence graphs, plus Guided LDA seeded with the community keywords. Tested on PubMed, research abstracts and news.

Topics

Resources

Stars

0 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages