Lousy case, no SSD, cheap components.
Are you really saving money?
You need to decide what you want. Do you want a generic budget tower PC sitting in your living room looking like a generic budget tower PC, or do you want something in a component-style case, with good components, a solid state drive, etc.
By the time you get it the way you want, you won't be saving anything and you won't have as good a system.
You mentioned putting it in a different case, but to buy a new case and to add a solid state drive means adding $200 to the cost of that refurbished Gateway..