What you're referring to is index printing. It uses spot colors like simulated process, but instead of regular halftones using round dots at a specific frequency and angle, it uses square dots that are randomly placed to achieve tints, gradients, and so on.
My only experience with it is outputting an index print through QuikSeps Pro, which has a set of actions to do this. One thing about index printing is that it often takes a lot of colors to get good results. I've also tried a simple white ink sep using indexing, and it came out pretty good. The one multi-color job I tried out of QuikSeps Pro, oddly, produced a moire in the underbase. I changed the dpi to 150 (from 200) and it eliminated it. Index printing is generally assumed to eliminate moire since the dots are random and not in a fixed "grid" like a traditional halftone screen, but as I found out, anything is possible. You can create your own index seps out of Photoshop by essentially assigning spot colors using the color range selection feature to create new channels under Image/Mode/Indexed Color. QuikSeps Pro recommends using 200 dpi as a dot size, and it works really well. The one sep set I did was a 4 spot color job that came out surprising well, except that for some reason, a drop shadow had a lot of blue in it and gave it a sort of exaggerated color look to it. The rest of the image came out great.