If you're lazy, just give the audio a +8dB volume boost, and enable the DRC at +8dB, and you should get an acceptable result for most multi-channel sources.
Select Dolby Prologic II as format, and make sure when not running Nero for AAC you're using 16 bit, and not higher, since the ffmpeg encoding engine doesn't like all 24 bit /32 bit or float sources. You can look at the settings in these screen shots set for 24 bit output:
AC3Filter main screen AC3Filter mixer screen
When encoding a multi-channel audio file to a stereo file, something will have to give.
When going from a 640 kbps AC3 file, to a 224kbps AAC file, some is gained by better compression of AAC, but quite some dynamics are lost. To make it even worse, during the mixing of the channels, the output will be matched so that maximum volume on the multi-channel source, will not results in a clipped signal.
In practice this means that multi-channel encodes to 2 channel formats (especially in a stereo surround encode like Prologic I or II) tends to be a lot lower in volume then the source., although surround effects can be completely overwhelming.
Since Firmware 2.0 PS3 users can even just set AC3filter, if they use Nero for AAC encoding, to 5.1 output, and they get full 5.1 surround AAC files in their MP4s. If you do not have a AAC decoding amplifier (AC3, DTS and DTS Neo are more the norm) , the sound will be a simple mix-down though.
(please verify if your receiver/pre-amp support this, I don't have the equipment to test this. To test it with TOSLINk, enable AAC output in the PS3 sound options, with HDMI, it should be automatically selected.)
Here is an example of an amp that support AAC: NHT Controller (always welcome for testing ;) )
If you do encode to 2 channels, and have a 5.1 sound system, it is best to encode to Dolby Pro Logic II and set the receiver/amplifier to the Dolby Pro logic II Movie setting. This will cause the receiver to try to extract the surround information from the stereo source, and will provide surround sound. (only not as good as true 5.1 output).
There are quite some things to improve this, but remember that it will always be a trade-off in quality compared to the source file.
The easiest steps to improve the mixing are to increase the source volume and enable dynamic range compression (you can get a good article on this from the ac3filter site: http://ac3filter.net/guides/dynamic_range_compression
As the article mentions, setting the DRC too high will cause ticks and clicks in the encode.
What the article doesn't mention is that by enabling DRC, you are losing a lot of the bit-depth of the original sound, and the audio can start to sound 'flat' (basically, the same thing that is happening with current popular music)
So as with everything in life, experiment as much as you feel like to get the optimal settings you enjoy, but be careful not to overdo it.