It is wise to avoid directories that are larger than necessary, for the same reason that you should avoid large files; directories are a special type of file, and when a directory grows too large any process that searches it becomes slower.
You should be aware of several special sizes. A directory that contains entries for up to 62 files (plus the required . and ..) fits in a single disk block and can be searched very efficiently. A directory can have up to 638 entries and still be viable, as long as it is used only for data storage; anything larger is usually a disaster when used as a working directory. The figures 62 and 638 apply to filenames of 14 characters or less. As filename lengths increase, up to a maximum of 255 characters, the number of files that fit on a single disk block decreases, thus reducing the optimum number of files in a directory.
It is especially important to keep login directories small, preferably one block at most. Note that, as a rule, directories never shrink. This is very important to understand, because if your directory ever exceeds either the 62 or 638 thresholds, searches will be inefficient; furthermore, even if you delete files so that the number of files is less than either threshold, the system will still continue to treat the directory inefficiently.