swap serves basically two purposes. It allows the system to continue to operate when physical memory runs out at a performance cost - run out of physical memory without it, you get crashes, lockups, and processes being killed with out of memory errors the second they ask for more memory than the system has. The reduced performance in this case is a symptom of being forced to use it for active processes, rather than a symptom of having too much swap.
It also lets physical memory be used more efficiently, by moving less-used pages in memory to disk until they are needed again. This frees up memory for cacheing purposes, which is usually a more efficient use of space than having infrequently used segments of program memory just sitting there locked in physical ram.
A long-standing best practice has been to size swap space at twice the physical memory, in other words, if you have 1GB of ram, devote 2GB to swap. This is still good advice, but in practice more modern systems with 4GB or more of physical ram can
usually drop this to the same amount of swap as the system has of physical memory.
There are a few things you can do to improve performance when using swap. If you have multiple drives, moving swap to a faster or less used hard drive is recommended, and on a very IO-bound system, you may get significant performance increases by doing this. For traditional hard drives, moving swap closer to the center of the physical disk may help, as seek times are generally shorter at the center due to less travel of the drive heads.
Having swap on an SSD can help significantly as well, but I'd caution you that this may create a lot of wear and tear on a SSD, and will give it a shorter lifespan.
Of course, the best solution to improving memory performance is usually to throw more RAM into the box, and if you look at your memory usage and see heavy usage of swap along with little or no free memory, it's a good indication that it's time to invest in more ram.