AWS EC2 p2xlarge实例无法运行Tensorboard的问题求助

Hey there, let's work through these TensorBoard issues one by one—since your training/evaluation runs fine, the problems are isolated to TensorBoard's dependencies and environment settings. Here's what to do:

1. Resolve the locale.Error (Critical Crash Cause) #

This is the final error that's killing TensorBoard, so let's fix it first:

• Check your current locale settings by running this in your EC2 terminal:

  locale
  

• You'll likely see some entries marked LC_ALL="" or a locale that's not supported. Set a UTF-8 locale temporarily to test:

  export LC_ALL=en_US.UTF-8
  export LANG=en_US.UTF-8
  

• Try launching TensorBoard again. If it works, make this change permanent by adding those two lines to your ~/.bashrc file (or /etc/environment for system-wide effect), then run:

  source ~/.bashrc
  

2. Fix Numpy Version Mismatch RuntimeError #

Your numpy downgrade for PIL is conflicting with TensorBoard's requirements. Here's how to find a compatible middle ground:

• First, check what numpy version TensorBoard expects. In your tensorflow_p27 environment, run:

  pip show tensorboard
  

  Look for the Requires line to see the numpy version range.
• Install a numpy version that works for both PIL and TensorBoard. For example, if TensorBoard needs numpy>=1.16.0 but PIL works with 1.15.4, try:

  conda install numpy=1.15.4
  

  If that doesn't work, try uninstalling PIL first, upgrading numpy to meet TensorBoard's needs, then reinstall a PIL version compatible with the newer numpy:

  pip uninstall pillow -y
  conda install numpy=<tensorboard-compatible-version>
  pip install pillow
  

  Alternatively, install PIL without dependencies to avoid automatic numpy downgrades:

  pip install pillow --no-deps
  

3. Eliminate Matplotlib Duplicate Key Warning #

This is just a warning, but cleaning it up will make your logs cleaner:

• Find where matplotlib's config files live by running:

  python -c "import matplotlib; print(matplotlib.matplotlib_fname())"
  

• Open the resulting file in a text editor, search for duplicate keys (look for lines that repeat the same setting, like backend: TkAgg appearing twice), and remove the duplicate entries.
• If you have a user-specific config at ~/.config/matplotlib/matplotlibrc, check that file too for duplicates.

Once you've worked through these steps, TensorBoard should launch without issues while keeping your training pipeline intact.

内容的提问来源于stack exchange，提问作者gustavz