Like you said, depends on the track!
A lot of my basic drumwork is often done in MIDI with Maschine, but there are plenty of times I'll just drag the sample from my library into the arrangement and just manually build it out. When I'm writing musical ideas I'll usually start by opening a piano sampler and just noodle around on my keyboard/draw in some notes, but I'll also start chopping up some loops or samples in my library and see how I can create a melody by pitching and reversing and all that fun stuff.
That being said, a great workflow to develop is bouncing and resampling so you can jump back and forth between audio and MIDI. Make your basic idea in MIDI, then bounce it all to audio so you can do fine tuned editing, then adding more MIDI to add more polish and detail, bouncing that, etc. Or design a sound, record a few notes, start chopping and messing with them and processing them, bounce that out to a new file and put that into a sampler so you can play a whole new musical idea with it. Or the classic "synths in MIDI, FX and drums in audio" split.