TSNE Plot运行耗时咨询：Amazon食品评论数据集Colab运行异常求助

Hey there! Let's break down your t-SNE plot issue—waiting half an hour for 5000 rows is definitely not normal. Under typical circumstances, this should finish in 1-10 minutes max, depending on a few key factors.

Why it's taking so long #

Here are the most likely culprits slowing things down:

• High-dimensional features: If you're feeding raw text embeddings (like uncompressed TF-IDF vectors with thousands of dimensions) directly into t-SNE, the pairwise distance calculations get computationally expensive fast.
• No multi-threading enabled: Scikit-learn's TSNE class defaults to using a single CPU core. Colab gives you access to multiple cores, so not utilizing them wastes a lot of time.
• Suboptimal parameters: Setting a very high perplexity (way above the recommended 5-50 range for your dataset size) or an excessive n_iter value can drag out runtime unnecessarily.
• Using CPU instead of GPU: Standard t-SNE implementations are CPU-bound, but GPU-accelerated versions can cut runtime from minutes to seconds.

Fixes to speed things up #

Try these steps to get your plot generated quickly:

1. Pre-reduce feature dimensions: Run PCA first to shrink your feature space to 50-100 dimensions before passing to t-SNE. Example code:

   from sklearn.decomposition import PCA
   from sklearn.manifold import TSNE
   
   # Assume X is your high-dimensional feature matrix
   pca = PCA(n_components=50)
   X_pca = pca.fit_transform(X)
   tsne = TSNE(n_components=2, n_jobs=-1, random_state=42)
   X_tsne = tsne.fit_transform(X_pca)
   

2. Enable multi-threading: Add n_jobs=-1 to your TSNE initialization to use all available CPU cores in Colab.
3. Use GPU-accelerated t-SNE: Colab supports RAPIDS, a GPU-accelerated ML library. Install it and use cuml.TSNE instead—for 5000 rows, this should finish in a few seconds. Quick snippet:

   !pip install cuml-cu11 --extra-index-url=https://pypi.nvidia.com
   from cuml.manifold import TSNE
   
   tsne = TSNE(n_components=2, random_state=42)
   X_tsne = tsne.fit_transform(X)
   

4. Tweak t-SNE parameters: Stick to perplexity=30 (the default) and n_iter=1000 unless you have a specific reason to adjust them.

内容的提问来源于stack exchange，提问作者Ajay