I know exactly what you are talking about...I had the same problem a few months ago.
There is a spring with a ball bearing on the end. If you take apart the unit and the ball bearing falls out you will crack the slider piece when you go to adjust the temp. w/o the ball bearing it will stick very badly.
I had to replace the slider piece with a used one...you can't buy it seperatly from Toyota (at least that is what I was told) ...I was quoted ~$450 for the assembly.