Average is best, especially in the 4-6 inches range. Small can be tricky, but I prefer it to OMG huge, because nobody really likes having their cervix battered. Every now and then, someone (usually a black guy, so that stereotype has some truth to it) comes in with a porn-sized penis and tells me that he sees escorts because normal women run screaming from his elephant dong. When that happens, it's time for a long, intense blowjob and very careful, gentle sex.
I will say that, if you're far from the average and know it, it's a good idea to bring your own condoms.