Peter wrote:This is exactly how I would do it (still wondering why no attention seems to go toward FPGA for multiprocessing, where the CPU cores could be on a single chip and coupled very efficiently).
I don't have your skills with FPGAs. Not even elementary skills. If I can get this functional with lean hardware, it would make sense later on to re-implement it in an FPGA. It would at least have driver software - the seemingly insurmountable hurdle.
Peter wrote:While technically interesting, I just doubt that multiprocessing would help the practical speed issues much. That's why I hesitate to go that road.
You're the only person who has released any work in "High Performance QLing"
I think you have exposed a point in throughput where the fine balancing act of the OS finally tumbles. High speed IO is a problem, and going forward it will only get worse and show up in more places. In days of old we would unburden the system of these issues by using DMA, which removed the transfers as an OS-managed task and converted them to an OS triggered, hardware-managed task. However, because the only performance benefits DMA brought to the table were overcoming OS weaknesses, it became more economical to code around it and improve those areas of the OS.
So there's our challenge as a community: as developers are in such short supply, how do we resolve the same issue knowing what the entire industry has done since then? Do we find a way to fix the OS, or do we throw hardware at the problem to make it go away? Both require developers - but OS developers are a very rare breed.
It's probably the most serious bottleneck the QL community faces.
Peter wrote:For example, more CPUs won't ease the pains of slow mass storage buffering nor overly memory-copying GUI...
Ah, but we can do a lot to ameliorate the problems. I'm not predicting they'll go away, but there are solutions that work for some parts of the equation that might be surprisingly simple.