The phrase originates from the following story.
A man has a beautiful house overlooking the ocean. He will be out of town for a few weeks and tells a friend he can have the full use of the house. Tells him that he can swim in the pool, watch the TV, sleep in the master bedroom, eat all the food he wants from the well stocked kitchen, and drink all the liquor he wants from the well stocked bar. The guy can do whatever he wants with one exception: don't screw the pooch.
So to "screw the pooch" means to really mess up a sweet deal. Also from "The Right Stuff", one astronaut is telling the other about bringing some souvenirs into space (a practice NASA forbid). The other says: "Well just be careful you don't screw the pooch". In other words, don't do anything so bad that you are going to mess up a sweet deal.