Always wanted to try this and I just did with ...less than perfect results.
Basically the scream effect is mimicking a resonant peak, only instead of putting the peak right at the cutoff, you can move the peak around relative to the cutoff. The extra peak created by the scream knob is also going under some fairly heavy distortion, and I think that distortion is only on the peak itself.
To recreate this in Serum, I simply whipped up a LPF on a standard saw, then went into the FX section, turned on the EQ and Distortion, then moved the distortion underneath the EQ. Choosing either band in the EQ, set it to peak mode, crank the gain, turn the Q to be around the middle, and then modulate the frequency of it with the same modulator as the cutoff for the LPF. After that, try something like Hard Clipper or Tube distortion and crank the drive.
The result is a little gimicky, and not quite the same, most likely due to the fact that the distortion is acting upon the whole sound, and also i think when i did it i just needed some tweaking to the actual modulation, but I believe that's pretty much the idea.
In all honesty, it's probably better to just use Massive whenever you need the sound of it's scream filter. You can always resample if need be, and I think to get the sound exactly the same would be way more effort than it's worth.
