Sorry, this post is nothing that a set of links!
It is not a good idea to compile packages right on Raspberry Pi. That could take hours.
Since, I am currently developing my own VoIP image, I need a tool for fast image creation and deployment.
Here is good introduction:
http://blog.kmp.or.at/2012/05/build-your-own-raspberry-pi-image/
http://www.kmp.or.at/~klaus/raspberry/build_rpi_sd_card.sh
And more links about cross compilation and QEMU (QEMU for emulation and compilation on "Raspberry Pi" environment):
http://ffmpeg.org/trac/ffmpeg/wiki/How%20to%20compile%20FFmpeg%20for%20Raspberry%20Pi%20(Raspbian)
http://www.raspbian.org/RaspbianDocumentation
http://hertaville.com/2012/09/28/development-environment-raspberry-pi-cross-compiler/
http://www.raspberrypiforums.com/forums/tutorials/article/16-a-raspberry-pi-emulated-environment-on-osx-lion/
http://xecdesign.com/qemu-emulating-raspberry-pi-the-easy-way/
http://www.raspberrypi.org/phpBB3/viewtopic.php?f=29&t=37386
http://igor.gold.ac.uk/~mas01mjy/wp/?p=95
http://elinux.org/Rpi_kernel_compilation#Cross_compiling_from_OSX