Saltstack troubleshooting

Invalid Jinja

Check the minion's /var/log/salt/minion to get the full python stacktrace.

Salt Modules won't Execute

Here there be dragons.
If you're lucky, the salt modules documentation will stipulate a requirement. For example, see the note on salt.modules.gpg.
In most cases, a problem can be pinned down by reading stacktraces in /var/log/salt/minion.

Here are some modules that I bang my head against frequently.

gpg

• install gpg
• install python-gpg to saltstack's python interpreter

pip

Most of my problems here come from:

1. salt using the global python installation rather than a virtualenv
2. my package manager installing a newer package than salt supports.

Unfortunately this is a moving target, and pinning packages becomes a bit of a headache.
(I did try salt in a virtualenv, but gave after hours debugging issues with salt's protocol decoding)

Verify using logs: sudo less /var/log/salt/minion.

Failed to import states pip_state, this is due most likely to a syntax error:

Sample Resolutions

Archlinux, Package Downgrade

# uninstall pkg while still required by salt
pacman -Rsd --nodeps python-pyzmq

# find/download old version of package
xdg-open https://archive.archlinux.org/packages

# install old package
pacman -U ~/Downloads/python-pyzmq-19.0.1-4-x86_64.pkg.tar.zst

# pin the package version so it doesn't get updated
# /etc/pacman.conf
IgnorePkg = python-pyzmq

If that doesn't work, confirm that the same python install is being used.
You ma/var/log/salt/miniony need to downgrade saltstack, or pip install ${PKG} into saltstack's python interpreter.

find /usr/lib -type d -name '*zmq*'
find /usr/lib -type d -name '*salt*'

/usr/bin/python -m pip install 'pyzmq<=20.0.0'

Archlinux, Deps

pkg_resources, required by pip.installed, now requires psutil.

pacman -S python-psutil

Bugs

KeyError: 'config.option', pkg_resources.DistributionNotFound: The 'contextvars' distribution was not found and is required by salt

• See https://github.com/saltstack/salt/issues/65110
• And https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=272027

If you don't need salt-ssh, you can fix by removing contextvars from /usr/local/lib/python3.9/site-packages/salt-3006.2-py3.9.egg-info/requires.txt