Riding Mower Won't Turn Over?

I have a JD LT133. I have just replaced the battery in it. It will not crank or turn over unless I use a rocking motion to the entire mower. Once cranked it runs smooth and steady, but once shutting it off, it will not re-start or turn over. What can be the issue with this mower?

By Howard C