For a 3 inch fish in 5 gallons you definatly need filtration, his problems will be stemming from bad water quality, you need to turn on the filter again, if its too strong and pushes him about, try putting plants/ornaments in the way to direct water flow.
Goldfish also need at least 10 gallons each to thrive so id look at upgrading the tank too