Saltstack troubleshooting

Invalid Jinja

Check the minion's /var/log/salt/minion to get the full python stacktrace.

Salt Modules won't Execute

Here there be dragons.
If you're lucky, the salt modules documentation will stipulate a requirement. For example, see the note on salt.modules.gpg.
In most cases, a problem can be pinned down by reading stacktraces in /var/log/salt/minion.

Here are some modules that I bang my head against frequently.

gpg

• install gpg
• install python-gpg to saltstack's python interpreter

pip

Most of my problems here come from:

1. salt using the global python installation rather than a virtualenv
2. my package manager installing a newer package than salt supports.

Unfortunately this is a moving target, and pinning packages becomes a bit of a headache.
(I did try salt in a virtualenv, but gave after hours debugging issues with salt's protocol decoding)

Verify using logs: /var/log/salt/minion

Resolve

Archlinux

# uninstall pkg while still required by salt
pacman -Rsd --nodeps python-pyzmq

# find/download old version of package
xdg-open https://archive.archlinux.org/packages

# install old package
pacman -U ~/Downloads/python-pyzmq-19.0.1-4-x86_64.pkg.tar.zst

# pin the package version so it doesn't get updated
# /etc/pacman.conf
IgnorePkg = python-pyzmq

If that doesn't work, confirm that the same python install is being used.
You may need to downgrade saltstack, or pip install ${PKG} into saltstack's python interpreter.

find /usr/lib -type d -name '*zmq*'
find /usr/lib -type d -name '*salt*'

/usr/bin/python -m pip install 'pyzmq<=20.0.0'