vim ~/.ssh/config Host 192.168.x.x ForwardAgent yes
Inside your vagrant file, add the line:
config.ssh.forward_agent = true
Then, inside your host machine, run
to start the ssh agent on your host machine.
Then run ssh-add on the vagrant box.
You can even automate the ssh-add on your provisioning scripts with the following:
echo "ssh-add" >> /home/vagrant/.bash_profile
This way, every time you vagrant up your machine will be ready to ssh into any system you have configured in your host without the need for passwords (think deploying to staging servers)