modw220, keep it simple. Pull the four runs of ethernet to a central point. That means that if you have CATV & telephone all where the breaker panel is, that means pull your ethernet there also.
I have both my telephone & catv coming in where my breaker panel is, and have a 3 foot A/V rack that I have two shelves attached to, and a patch panel. All of my drops are to the patch panel, and on a shelf near the patch panel on the rack, sits my RG for ATT U-Verse & the Netgear GS-108 switch.
If all it is is four drops period, get a Single gang box, that you can place a six Keystone wall plate on to, and that way you can use the keystones as your "patch" panel. Put in two "blanks" that there are not the two keystone jacks. From the "Patch Panel", then you can run patch cords to a switch, or use the patch panel if you are going to use HDMI over ethernet.
I tend to pull everything to a central point. Only time that I would place a switch up on another floor, is if I wanted to not pull very long runs down to the main panel, so that everything is kept simple. That would mean if it is two or three floors, you would have a switch on each floor, then from there a run of ethernet would be from that floor to the main switch that has no other feeds attached to it. Now of course the catch is, you would be using "smart" switches for all of this, not "dumb" switches which are typical in the majority of home installs.
For yours, use a single dumb switch, and pull everything to a central point.