The problem here is probably your commit rate. I can tell that you have a very high commit rate, because the volume of log generation is much higher (about 6 times higher than a linear increase) when you run 9 processing threads. The only thing that could cause that is excessive redo wastage. High redo wastage means that your LGWR is overactive, and with a large log buffer, that means that you are committing much too often.
The log buffer space waits that you are seeing can be dismissed as a side effect of the fast log switching. You probably also have some contention for the redo allocation latch, and VERY high log file sync waits.
If you cannot reduce the commit frequency, then you will get some relief from the log file sync waits by running LGWR as a real time process, and making sure that you follow the recommendations in the Ixora web tip on Disk Configuration for Online Log Files. Longer term you may want to consider partitioning the workload between multiple nodes of a parallel server database so that you can get multiple independent threads of redo.
I have a question for you, may be it is very simple for you to help me out. I am running Oracle 8.1.6 on Sun Solaris 2.6 (Sun E10K, 16CPU, 8GB RAM, EMC storage, 3 controllers with fiber channel, powerpath, DMP disabled, Vetitas Volume Manager, 3 controllers dual path for fail-over). Our stripe width is 64K. We are getting data from click stream and currently we are doing single thread of processing. We want to go for 9 thread of parallel processing to get 9X increase in processing speed and throughput. But with all the different scenario's we are getting 3 threads is optimal. After 3 parallel threads the throughput is starting to fall drastically. Is it related to number of controllers?
I noticed the log switch is happening for each minute while doing 9 parallel thread and it is for 1 hour in case of single thread (4M log buffer and 512M log file). So I increased the log buffer to 30M and log file size to 2GB. So now log switch is happening for every 5 min while doing 9 parallel threads. Redo log files are in non-stripped disk. I am noticing log buffer busy wait very much while doing 9 parallel threads. _log_io_size is set to 1/3 rd of log_buffer. Still I am not getting good results. I am running out of ideas. I hope you may have some great tips to solve this. Thanks in advance.