Using <Esc> as a modifier key is a very unusual thing in my experience. When I read about the mode transitions I didn't understand it and I rather tried them as a key combo (like <Esc>, release, and then an arrow key).
The idea is fine, but I think it should be explained clearly in the Usage docs, maybe even highlight it with an info box.