I have found the main thing messing things up in terms of the context-menu not being shown at all on consecutive right clicks. I have also immensely improved the fallback heuristics.
In addition, since it still might not be perfect, in certain scenarios, thus I consider this an ongoing (long-term?) topic to be discussed and researched over here.
Update: I have played around little bit more and now I believe the current results might be considered as solid, under normal usage scenarios.