Are you talking about the snappy transient noise? It's pretty typical for saw-based sounds to have a harsh high end, and if your filter modulation (pluck) is exposing those frequencies, it might add harshness. Also if you are closing the filter really quickly, it will probably exacerbate any harshness.
What I've done in the past to reduce this problem is some combination of:
(a) decrease the cutoff frequency of the filter you're modulating
(b) increase the decay time of the filter
(c) soften the high end with EQ (& possibly add in noise to fill the frequency space you cut out, depending on the sound you want)
(d) dial down any distortion/compression on the sound, if there is any contributing to the issue
(e) change the waveform in the synth