You should check the seal on your refrigerator every so often. You can do this by closing the door on a dollar bill. If you can pull the bill out easily, it's time to replace the gaskets. You can buy these replacement kits from either an appliance dealer or a home repair center.
This page contains the following solutions.
To test the seal of your refrigerator, put a piece of paper on the edge of the rubber gasket and close the door. If the paper can be pulled out easily then warm air can find it's way in. The gasket should be replaced.