Desktop Linux with NFS Home Directories

Something no one does anymore, apparently.

January 19, 2023 AD

I manage multiple Rocky Linux workstations that automount users’ home directories via kerberized NFS. Unfortunately, I don’t think this is a common setup anymore–I encountered a few bugs and performance issues that needed non-obvious workarounds.


1. Things break when you log in from two places at once

If you can somehow restrict your users to a single GNOME session at any given time, you’ll probably be fine. However, as soon as someone leaves his desktop running and logs into another workstation, strange things begin to happen. Here are some oddities I’ve observed:

2. It’s slow

I/O-heavy tasks, like compiling and grepping, will be much slower over NFS than the local disk. Browser profiles stored on NFS (~/.mozilla, ~/.cache/chromium, etc.) provide a noticeably poor experience.

File browsing is also painful if you have lots of images or videos. Thumbnails for files stored on NFS will be cached in ~/.cache/thumbnails, which is also stored on NFS!

Solution: Move stuff to local storage

The XDG Base Directory Specification lets you change the default locations of ~/.cache, ~/.config, and the like by setting some environment variables in the user’s session. We can solve most of these problems by moving the various XDG directories to the local disk.

Automatically provision local home directories

First, let’s write a script that automatically provisions a local home directory whenever someone logs in:


# /usr/local/sbin/

# Log all output to syslog.
exec 1> >(logger -s -t $(basename "$0")) 2>&1

PAM_UID=$(id -u "${PAM_USER}")

if (( PAM_UID >= 1000 )); then
  install -o "${PAM_USER}" -g "${PAM_USER}" -m 0700 -d "/usr/local/home/${PAM_USER}"

Of course, it needs to be executable:

chmod 755 /usr/local/sbin/

Next, we modify the PAM configuration to execute our script whenever anyone logs in via GDM or SSH:

--- /etc/pam.d/gdm-password
+++ /etc/pam.d/gdm-password
@@ -1,5 +1,6 @@
 auth     [success=done ignore=ignore default=bad]
 auth        substack      password-auth
+auth        optional /usr/local/sbin/
 auth        optional
 auth        include       postlogin

--- /etc/pam.d/sshd
+++ /etc/pam.d/sshd
@@ -15,3 +15,4 @@
 session    optional
 session    include      password-auth
 session    include      postlogin
+session    optional /usr/local/sbin/
A note on SELinux

If you’re using SELinux, you’ll need a separate copy of the create-local-homedir script for use with GDM, labeled with xdm_unconfined_exec_t:

ln /usr/local/sbin/create-local-homedir{,-gdm}.sh
semanage fcontext -a -t xdm_unconfined_exec_t /usr/local/sbin/
restorecon -v /usr/local/sbin/

Be sure to modify /etc/pam.d/gdm-password appropriately.

Set XDG Environment Variables

We need to tell the user’s applications to use the new local home directory for storage. We have to do this early in the PAM stack for GDM, because $XDG_DATA_HOME must be set before gnome-keyring gets executed.

Edit your PAM files again, adding one more line:

--- /etc/pam.d/gdm-password
+++ /etc/pam.d/gdm-password
@@ -1,6 +1,7 @@
 auth     [success=done ignore=ignore default=bad]
 auth        substack      password-auth
 auth        optional /usr/local/sbin/
+auth        optional conffile=/etc/security/pam_env_xdg.conf
 auth        optional
 auth        include       postlogin

--- /etc/pam.d/sshd
+++ /etc/pam.d/sshd
@@ -16,3 +16,4 @@
 session    include      password-auth
 session    include      postlogin
 session    optional /usr/local/sbin/
+session    optional conffile=/etc/security/pam_env_xdg.conf

Then, create the corresponding pam_env.conf(5) file:

# /etc/security/pam_env_xdg.conf

XDG_DATA_HOME    DEFAULT=/usr/local/home/@{PAM_USER}/.local/share
XDG_STATE_HOME   DEFAULT=/usr/local/home/@{PAM_USER}/.local/state
XDG_CACHE_HOME   DEFAULT=/usr/local/home/@{PAM_USER}/.cache
XDG_CONFIG_HOME  DEFAULT=/usr/local/home/@{PAM_USER}/.config

Hacks for Non-XDG-Compliant Apps

Unfortunately, since a majority of open source developers follow the CADT model, there are many apps that ignore the XDG specification. Sometimes these apps have their own environment variables for specifying their storage locations. Otherwise, symlinks can provide us with an escape hatch.

Create a script in /etc/profile.d for these workarounds. Scripts in this directory are executed within the context of the user’s session, so we can freely write inside his NFS home directory using his UID (and kerberos ticket, if applicable).

# /etc/profile.d/

if (( UID >= 1000 )); then
  # Building code is *much* faster on the local disk. Modify as needed:
  export PYTHONUSERBASE="/usr/local/home/${USER}/.local"  # python
  export npm_config_cache="/usr/local/home/${USER}/.npm"  # nodejs
  export CARGO_HOME="/usr/local/home/${USER}/.cargo"      # rust
  export GOPATH="/usr/local/home/${USER}/go"              # golang

  # Firefox doesn't provide an environment variable for setting the default profile
  # path, so we'll just symlink it to /usr/local/home.
  mkdir -p "/usr/local/home/${USER}/.mozilla"
  ln -sfn "/usr/local/home/${USER}/.mozilla" "${HOME}/.mozilla"

  # Flatpak hardcodes ~/.var, so symlink it to /opt/flatpak.
  ln -sfn "/opt/flatpak/${USER}" "${HOME}/.var"

If you use any Flatpak apps, each user will need his own local Flatpak directory. The Flatpak runtime appears to shadow the entire /usr using mount namespaces, so any /usr/local/home symlinks will disappear into the abyss. Luckily, /opt appears to be undefiled. Modify your original script like so:

--- /usr/local/sbin/
+++ /usr/local/sbin/
@@ -6,4 +6,5 @@

 if (( PAM_UID >= 1000 )); then
   install -o "${PAM_USER}" -g "${PAM_USER}" -m 0700 -d "/usr/local/home/${PAM_USER}"
+  install -o "${PAM_USER}" -g "${PAM_USER}" -m 0700 -d "/opt/flatpak/${PAM_USER}"

Closing Thoughts

Most of my users are nontechnical, so I’m pleased that these workarounds do not require any manual intervention on their part.

I am sad that $XDG_CONFIG_HOME can’t be shared between multiple workstations reliably. When I change my desktop background or add a new password to gnome-keyring, it only affects the local machine.

Initially, I tried symlinking various subdirectories of ~/.config to the local disk individually as I encountered different bugs (e.g. ~/.config/pulse). Unfortunately this proved brittle, as I was constantly playing whack-a-mole with apps that abused $XDG_CONFIG_HOME for storing local state. In the end, it was less of a headache to just dump the whole thing onto the local disk.

I suppose if you verified an app behaved properly with multiple simultaneous NFS clients, you could always symlink /usr/local/home/$USER/.config/$APP back onto NFS!