Over the last fifteen years or so, I have installed numerous brick and stone patios, walkways and driveways. Here in Michigan, most paving is dry-laid, that is, there is no concrete or mortar involved in the installation process. Because concrete and mortar tend to break up over time due to the freeze thaw cycles in this cold climate, it makes sense to use a flexible base and joint filler. Until a few years ago, I used either baby sand, which tends to washed out too easily, or rough-graded sand to fill between concrete or clay pavers. I used chips n dust or stone dust or slag sand to fill between natural flagstone. Rough sand and the fillers I used between flagstones worked fairly well for keeping the joints filled and the pavers from shifting. (Of course aluminum or plastic edging is necessary to keep the edges of each project from shifting outward, but that is another subject). The real downside of these joint-filling materials is that they don't keep weeds down. And yes, there is some washout over time.