yep, plenty of people have done it.

personally, I prefer to space out. if it's easy with your axle, maybe 2mm spacers on each side so you only squeeze 2mm down in total.

helps to keep the dropouts parallel down the length