I generally go down a scale for buildings if I'm gaming anything other than 1:1 skirmish.
Visually I think the slightly smaller buildings look better for whatever reason. I have some of the Wargames Atlantic 10mm Japanese sengoku sets and I plan to use 6mm scenery for them.
Not Japanese but these give an idea about going a scale down:

3mm farm with 6mm figures

10mm figures with 6mm buildings