Laggy ssh with clear network

Recently, after reboot, my ssh sessions to remote servers started behaving sluggishly, sometimes taking several seconds for typed characters to appear. The lag was especially noticeable with last character that got typed. Nature of ssh sessions it to type several characters and checking the output. You can imagine the frustration seeing most of keypresses delivered, but getting feedback after some delay. Ping to the server was showing reliable, speedy responses with not a single packet loss. Network was in excellent shape, no torrents or other heavy uploads. I did not know where to look or how to google for the problem. I then remembered that I waited several months for the reboot to check an X-related crash and found that I disabled all X acceleration in the /etc/X11/xorg.conf file:

    Option "DRI" "false"
    Option "shadow" "true"
    Option "NoAccel" "True"

Desperate about the whole issue, I enabled back all X acceleration options, restarted X and voila! The ssh is fast again! Now, I don't know whether it is a bug or which subsystem it is in. So I decided to describe the problem sprinkled with keywords I used while googling for the problem in hope it helps someone. Write in comments if it did. :)