Setting the Temperature in a Refrigerator?

I have a question on refrigerator temp settings. I have a top freezer fridge in my garage. The freezer side works perfectly, but sometimes, the fridge side does not. Since it doesn't have a temp display, I have placed manual temp gauges inside, one in the freezer and one in the fridge section. Anyway, my understanding is that the fridge side needs to be no more than 37 degrees to keep cooling. The dials controlling the temp are set to 5 (0-9) where 5 is the 'normal' setting.

My question is if I want to drop the temp on the fridge side, is a higher number (6-9) colder or the lower (1-4). Please note it's sitting in a garage. If I leave it closed overnight, the temp remains around that 37 degrees. If I open it a few times, it gets into the warm range (40+). I am just looking for a little advice here. Thanks.