My little boy did the same. If he was just whining, I would get down low and look him in the eye and say something like "mommy doesn't want to hear that whiny voice ... when you're done, let me know, and then we can talk together." Then I would ignore him or remind him that I didn't want to hear that voice and wanted to hear his "good-boy voice" ... Or if he threw himself down, I would say "I don't want to watch this kind of behavior, so I'm going in to (the living room/kitchen/etc.). Let me know when your done with your tantrum, and we can talk together" and then I would leave.
This is of course harder to do at the store or out in public, but usually, in those instances, I would say "We will go home if this behavior continues" and if it does, we really do go home. (Most of the good training happens at home though.) If there were times when his whining/tantrums just wouldn't stop, we would take time out and "face the wall" for 2 minutes at age 2, 3 minutes at age 3, etc. We would walk over to the wall spot, I would get down low, face him to me, and look him in the eye and tell him why he had to face the wall (for whining, hitting, throwing, not listening, etc.) and then tell him he had to face the wall as punishment. When his time was up AND he had quieted down, I would go to him, stand him up and tell him again why he had had to face the wall, and then ask for him to say "I'm sorry for whining/hitting/throwing/not listening/etc." and then I would ask for a hug. If he had offended anyone else (daddy, grandma, grandpa, etc.) he would have to go tell them he was sorry too.
A friend of mine established the "whining step" ... the lowest step on the stairs for their little girl to go for time out when her whining got out of control.
We've been consistent, and now we have a fairly nicely behaved boy. Not perfect, but he can get himself under control ... now he'll say after a bit of whining and a warning from me, "my tears are all gone now, mommy! I'm all better!" It's tough, but stick with it. They need and appreciate the boundaries of what is acceptable behavior.