Finds the number of topics in a text corpus with graph community detection and uses the result to drive LDA and Guided (seeded) LDA.
Choosing the number of topics k is one of the hardest parts of LDA. Coherence-score sweeps are slow and often pick a k that doesn't match the real structure of the data. This project builds a word co-occurrence network, splits it into communities with the Louvain algorithm, and uses:
- the number of communities as the number of topics, and
- the most central words in each community as topic labels and as seed words for Guided LDA.
Raw documents
│
▼
Preprocessing ── tokenize · remove stopwords · keep nouns/verbs (POS filter)
│ lowercase · strip punctuation · lemmatize · merge WordNet synonyms
│ drop rare words · TF-IDF vocabulary filter (min_df=2, max_df=0.8)
▼
Word co-occurrence graph (words = nodes, co-occurring in a document = edge)
│
▼
Louvain community detection ──► number of communities = k
│
├──► Eigenvector centrality in each community ──► top-10 words per topic
│
├──► LDA with k topics (number of topics from the communities)
│
├──► LDA with k chosen by the best c_v coherence (baseline, k = 1…39)
│
└──► Guided LDA seeded with each community's top words
| Notebook | Corpus | Documents used |
|---|---|---|
PubMed.ipynb |
PubMed abstracts from 4 search topics: deep learning, COVID-19, human connectome, virtual reality | 1,000 (250 per topic) |
Research topics.ipynb |
Research article abstracts (ID, TITLE, ABSTRACT) |
1,000 |
Topic modelling Code.ipynb |
News articles: Harvard Dataverse, doi:10.7910/DVN/GMFCTR | Full dataset |
GLDA_Ex_5_RANLP.ipynb |
Preprocessed PubMed-domain corpus (experiment_5.pickle) |
2,229 |
The datasets are not included in this repository.
Number of topics found by community detection vs. by the best coherence score:
| Corpus | Louvain communities | Best-coherence k |
|---|---|---|
| PubMed (4 known topics) | 4 | 2 |
| Research abstracts | 6 | 3 |
| News articles | 9 | 30 |
On the PubMed corpus, where the true number of topics is known to be 4, community detection found exactly 4 clearly separated topics. The top words by eigenvector centrality were:
| Community | Top words |
|---|---|
| 0 – Human connectome | connectivity, brain, analysis, region, function, imaging, cortex |
| 1 – Deep learning | learning, model, network, resolution, method, data, image |
| 2 – COVID-19 | covid19, patient, disease, coronavirus, case, health, sarscov2 |
| 3 – Virtual reality | reality, participant, vr, virtual, environment, time |
Seeding Guided LDA with the community keywords (seed_confidence=0.15) gave clean, separated topics. Example output:
Topic 0: electrode film pedot polymer electrochemical surface pedotpss layer high composite
Topic 1: brain network connectivity functional study patient region cortex control cortical
Topic 2: method data model learning network image feature deep performance neural
├── PubMed.ipynb # Community detection + LDA on PubMed abstracts
├── Research topics.ipynb # Same pipeline on research article abstracts
├── Topic modelling Code.ipynb # Same pipeline on news articles
└── GLDA_Ex_5_RANLP.ipynb # Guided (seeded) LDA using community keywords
pip install pandas numpy scikit-learn nltk gensim networkx python-louvain pyLDAvis matplotlib seaborn
pip install guidedlda # only needed for GLDA_Ex_5_RANLP.ipynbimport nltk
nltk.download(['stopwords', 'punkt', 'wordnet', 'averaged_perceptron_tagger'])
guidedldais no longer maintained and may fail to install on recent Python versions. Python 3.7–3.9 is recommended for that notebook.
- Download the dataset for the notebook you want to run.
- Change the
pd.read_csv(...)path at the top of the notebook to point to your copy. - Run all cells.
Python · NLTK · Gensim · scikit-learn · NetworkX · python-louvain · GuidedLDA · pandas
Intesur Ahmed: GitHub