如何在Amazon EC2中永久配置PySpark相关环境变量并查看所有环境变量

Got it, let's tackle your two questions one by one—this is a super common scenario when setting up Spark with Jupyter on EC2!

Since you're using Ubuntu (the default AMI for EC2 uses bash shell), there are two main ways to make these environment variables stick across terminal sessions:

For Your Current User (Recommended) #

This method only applies to your user account and doesn't require admin privileges:

1. Open the .bashrc file in your home directory with a text editor like nano:

   nano ~/.bashrc
   

2. Scroll to the very end of the file and paste your three export lines:

   export SPARK_HOME=/home/ubuntu/spark-3.0.1-bin-hadoop3.2
   export PATH=$SPARK_HOME/bin:$PATH
   export PYTHONPATH=$SPARK_HOME/python:$PYTHONPATH
   

3. Save and exit nano: press Ctrl+O, hit Enter to confirm the save, then Ctrl+X to close.
4. Make the changes take effect immediately (no need to restart your terminal):

   source ~/.bashrc
   

From now on, every time you log into your EC2 instance or open a new terminal, these variables will load automatically.

Quick note: If you're using a different shell (like zsh instead of bash), edit the corresponding config file (e.g., ~/.zshrc) instead of .bashrc. But Ubuntu's default shell is bash, so this shouldn't be an issue here.

For All Users on the System (Requires Admin Rights) #

If you want every user on the EC2 instance to have access to these Spark variables, edit a global config file:

1. Open /etc/profile with sudo (since it's a system-wide file):

   sudo nano /etc/profile
   

2. Add the same three export lines at the end of the file, save, and exit.
3. To apply the changes immediately in your current terminal, run:

   source /etc/profile
   

Keep in mind this affects all users, so only use this if you need it—most of the time, the user-specific method is sufficient.

There are a few handy commands to check your environment variables, depending on what you need:

• List all environment variables: Use env or printenv—both will show every active environment variable and its value:

  env
  # Or alternatively
  printenv
  

• Check a specific variable: If you just want to verify one variable (like SPARK_HOME), use echo:

  echo $SPARK_HOME
  echo $PATH
  echo $PYTHONPATH
  

• Filter for specific variables: When the list is too long, pipe the output to grep to narrow it down. For example, to find all Spark-related variables:

  env | grep SPARK
  

• View all shell variables (including local ones): Use the set command, but note this will show more than just environment variables—it includes shell functions and local variables too. Stick with env or printenv if you only care about environment variables.

内容的提问来源于stack exchange，提问作者Hamidreza Ahady Dolatsara