problem with starter???

when i don't start my car for a week or two the starter solenoid clicks but will not turn over engine . I installed a button switch with relay on it. If hit the button a few times the starter turns over and engine starts up. Problem is much worse when cool weather sets in.
when the the starter is warm or over the summer the starter works fine. Ithink wire to starter is problem.