Hey Steve,
I doubt you will see a WOW differance on coyote from a 17hmr to a 22wmr but, I have only shot woodchucks,coons and a few fox for testing. IMO- they are six to one one half dozen the other and shot placement is key with either. At this point your not 100% sure it was hit let alone a hit to the vidal with it being night etc. If they bring out the 5mm in 2009 that may be the WOW differance in rim mag????
As Glen noted try the 20 gr hp, it will certainly penetrate. Or my favorite the TN'T if your gun likes them.
Sounds like your on the right track with the Mossy. For #4 buck patterning most including myself have found the Federal Premium patterns much better than the value stuff. You will have to order it as no shops around here carry it, or I can pick you up some when in Buffalo on business.
I would also look hard at Remington hevi shot in BB, both in 3" btw.
IMO- that "60 yards" is the tough choice mark/range. Its out of shotgun range and certainly with in kill power for rim mag. I think the advantage belongs to the rim mag if the set has open enough area for bullets and the shotgun if its brushy. With the shotgun try to use cover/coyote expected entrance area to your advantage with hopes it shows inside 30-40 yards. Though I know you will skin him.......you and I are not really what I would call "fur hunters".............so don't hesitate to cycle that pump and put another in him for good measure.
I am currently after Steelhead with the flyrod & spey rod as you know

It should not be much longer and I will have got this out of my system and be ready to hunt again.
PS- If you whack a Coyote I will take the tail (if not mangy) to tie some Steelhead streamers out of..........unless you want it for the same.............it makes great hairwings and really swims well.
Glen,Red,AMMOe,Jerry,marine, others...............Hi

Hope you and your families are well.
Take care buddy..........talk to you soon.
PS - I just remembered. I have a bunch of 20 gr hp and TN'T's in HMR still..........swing by and get them if you want them.