It's on the LAN and CPU spikes can continiue for some time, up to a few minutes for stupid vlookup in Excel or similar cpu intensive operations.
But it seams like the high load is disturbing the View Agent somehow, looking at the event log I see several SMB disconnect events occuring at the same time as these problems.
We are not using SSL btw.