I responded to you with this in Discord, but pasting it here for others that may happen across this thread...
If you use SteamNetworking, it basically uses Steam servers like a VPN. All traffic goes through Steam before it gets to your server, and makes the same path in the opposite direction. This increases latency incredibly.
SteamNetworking should only be used if you are unable to properly forward the ports, or for troubleshooting. In absolutely every other situation it should be disabled.
Disabling LiteNetLib will have a severe impact on network performance of the server.