Implement modal interface in a major mode

So, following @Tyler advice I came with this:

;; Major mode
(defvar pletter-mode-keymap (make-sparse-keymap))
(define-key pletter-mode-keymap (kbd "C-q") 'move-pletter-mode)

(define-derived-mode pletter-mode fundamental-mode "pletter"
  "pletter-mode is a major mode for playing with letters and words.
\{pletter-mode-map}"
  (use-local-map pletter-mode-keymap) ;; not needed
  )

;; Minor mode
(defvar move-pletter-mode-keymap (make-sparse-keymap))
(define-key move-pletter-mode-keymap (kbd "<left>") 'movechar-to-left)
(define-key move-pletter-mode-keymap (kbd "<right>") 'movechar-to-right)

(define-minor-mode move-pletter-mode
  "Minor mode to toggle arrows bindings for pletter-mode."
  nil " move" move-pletter-mode-keymap
  (message "Toggle move-pletter"))

(provide 'pletter-mode)

which seems to work for me.

You could create a minor mode that binds the arrow keys to the functions you want. Then your 'special' key only needs to toggle the minor mode on and off. You would bind the 'special' key in your major mode map, and make the minor mode buffer-local, so it would only change the behaviour in files in that major mode.