Did you properly configure the PrivacyIDEA-ADFS provider config.xml file specifying your PI server in it before you installed it on the ADFS server? If you have to make a config change, that config file isn’t referenced after installation, so you’d have to uninstall the provider then reinstall it.
Do you have a load balancer in front of the PI server? I had issues with my lab setup going through a virtual kemp. Never worked on getting that to play nice though…
Enable debug logging in your pi.cfg file and run through an auth attempt, there’s usually something in there to help.