Take a pinch grip at the front of the hat's crown and flick it up and forwards. Turn the hand round so the palm is facing forwards to catch the hat in another pinch grip at the back. After you've got a feel for this try doing it without looking at the hat.
Either place it straight back on the head or reverse the process to return it.
If you use a bowler hat make sure you take a really firm pinch to crease the top of the hat when removing it, otherwise you'll be left trying to catch the smooth round surface of the back of the hat.
Once you've mastered this one it's a trivial task to start by pinching the back of the hat and flicking it to a catch on the front.