Mira quizás sea que la config se te ha puesto mal o algo, en la carpeta del 3GP converter hay un archivo que se llama Transcoding.ini, lo abres y cambias el Item3 y Item4 por estas configs
[Item3]
Title=AVC 480x272 768 kbps 1-pass
TitleE=AVC 480x272 768 kbps 1-pass
Command0=""<%AppPath%>\cores\ffmpeg" -y -i "<%InputFile%>" -title "<%Title%>" -timestamp "<%TimeStamp%>" -bitexact -vcodec h264 -coder 1 -bufsize 128 -g 250 -s 480x272 -r 29.97 -b 345 -acodec aac -ac 2 -ar 48000 -ab 64 -f psp "<%OutputPath%>MAQ<%RandomNumber5%>.MP4""
Command1=""<%AppPath%>\cores\ffmpeg" -y -i "<%InputFile%>" -f image2 -ss 5 -vframes 1 -s 160x120 -an "<%OutputPath%>MAQ<%RandomNumber5%>.THM""
[Item4]
Title=AVC 480x272 768 kbps 2-pass
TitleE=AVC 480x272 768 kbps 2-pass
Command0=""<%AppPath%>\cores\ffmpeg" -y -i "<%InputFile%>" -title "<%Title%>" -timestamp "<%TimeStamp%>" -bitexact -vcodec h264 -coder 1 -bufsize 128 -g 250 -s 480x272 -r 29.97 -b 768 -passlogfile "<%OutputFile%>" -pass 1 -acodec aac -ac 2 -ar 48000 -ab 64 -f psp "<%OutputPath%>MAQ<%RandomNumber5%>.MP4""
Command1=""<%AppPath%>\cores\ffmpeg" -y -i "<%InputFile%>" -title "<%Title%>" -timestamp "<%TimeStamp%>" -bitexact -vcodec h264 -coder 1 -bufsize 128 -g 250 -s 480x272 -r 29.97 -b 310 -passlogfile "<%OutputFile%>" -pass 2 -acodec aac -ac 2 -ar 48000 -ab 64 -f psp "<%OutputPath%>MAQ<%RandomNumber5%>.MP4""
Command2="rm "<%OutputFile%>-0.log""
Command3=""<%AppPath%>\cores\ffmpeg" -y -i "<%InputFile%>" -f image2 -ss 5 -vframes 1 -s 160x120 -an "<%OutputPath%>MAQ<%RandomNumber5%>.THM""
Luego en la PSP eliges una de las dos opciones o bien "AVC 480x272 768 kbps -1pass" o el siguiente.
Saludos