As per ffmpeg x264 encoding guide, you can pass -tune animation option.

Alternatively…

Adapting the options for x264 suggested here to mencoder results in these settings for encoding (except for the turbo thing, see below).

You can input these options directly from commandline, but for later use, you may find it more convenient to edit ~/.mplayer/mencoder.conf and add these

[gnuanimex264p1]
profile-desc="GNU/Anime x264 Pass 1"
lavdopts=threads=2
ovc=x264=yes
x264encopts=turbo=2:bitrate=300:pass=1:frameref=5:mixed_refs=yes:fast_pskip=no:bframes=3:b_pyramid=normal:weight_b=yes:me=umh:subme=8:direct_pred=auto:trellis=2:partitions=all:8x8dct=yes:threads=auto
oac=mp3lame=yes
lameopts=vbr=3:mode=1:q=4
of=lavf=yes
lavfopts=format=avi
sws=9

[gnuanimex264p2]
profile-desc="GNU/Anime x264 Pass 2"
lavdopts=threads=2
ovc=x264=yes
x264encopts=bitrate=300:pass=2:frameref=5:mixed_refs=yes:fast_pskip=no:bframes=3:b_pyramid=normal:weight_b=yes:me=umh:subme=8:direct_pred=auto:trellis=2:partitions=all:8x8dct=yes:threads=auto
oac=mp3lame=yes
lameopts=vbr=3:mode=1:q=4
of=lavf=yes
lavfopts=format=avi
sws=9

[gnuanimex264]
profile-desc="GNU/Anime x264"
lavdopts=threads=2
ovc=x264=yes
x264encopts=crf=26:frameref=5:mixed_refs=yes:fast_pskip=no:bframes=3:b_pyramid=normal:weight_b=yes:me=umh:subme=8:direct_pred=auto:trellis=2:partitions=all:8x8dct=yes:threads=auto
oac=mp3lame=yes
lameopts=vbr=3:mode=1:q=4
of=lavf=yes
lavfopts=format=avi
sws=9

Now you can use these profiles like this

(2-pass, ratecontrol: bitrate)
mencoder -profile gnuanimex264p1 $1" -o /dev/null
mencoder -profile gnuanimex264p2 "$1" -o "$2"

(1-pass, ratecontrol: crf)
mencoder -profile gnuanimex264 "$1" -o "$2"

You may like to alter various parameters here. turbo might be the first thing you'd like to change (excerpt from mencoder's manual page:

turbo=<0-2>
  Fast first pass mode.  During the first pass of a two or more pass
  encode it is possible to gain speed by disabling some options with
  negligible or even no impact on the final pass output quality.
    0    disabled (default)
    1    Reduce subq, frameref and disable some inter-macroblock
    partition analysis modes.
    2    Reduce subq and frameref to 1, use a diamond ME search and
    disable all partition analysis modes.
  Level 1 can increase first pass speed up to 2x with no change in the
  global PSNR of the final pass compared to a full quality first pass.
  Level 2 can increase first pass speed up to 4x with about +/- 0.05dB
  change in the global PSNR of the final pass compared to a full
  quality first pass.

and the obvious parameter “threads” (of -lavdopts). “format” parameter (of -lavfopts) is another obvious one. The bitrate (for video) is 300 but you might wanna play around with other ratecontrol systems as well. Output audio is VBR mp3lame with q=4, though Opus would be a better option for low rates ¹⁾.

FFmpeg should be able to do the same job, albeit with slightly different options for x264.

You can make use of use the preset options ani and anihq fromh264enc project.

ani=force_cfr:frameref=8:mixed_refs:bframes=5:b_adapt=2:b_pyramid=normal:noweight_b:weightp=1:direct_pred=auto:aq_mode=2:aq_strength=0.5:me=hex:subq=10:nombtree:psy_rd=0.3,0.0:chroma_me:trellis=1:cabac:deblock:8x8dct:partitions=p8x8,b8x8,i8x8,i4x4:nofast_pskip:nodct_decimate
anihq=force_cfr:frameref=10:mixed_refs:bframes=5:b_adapt=2:b_pyramid=normal:noweight_b:weightp=2:direct_pred=auto:aq_mode=1:aq_strength=0.5:me=umh:me_range=24:subq=7:mbtree:psy_rd=0.3,0.0:chroma_me:trellis=1:cabac:deblock:8x8dct:partitions=p8x8,b8x8,i8x8,i4x4:nofast_pskip:nodct_decimate

or use the script itself –it's a very handy one with a bunch of options, but beware, it's heavily interactive.