Seems like sound reasoning. Through time cadres have been the backbone of most military formations.
I am working on a USCT brigade - and perhaps a 'legion' for my ACW collection, one full brigade (4 regiments) of infantry, at least one cavalry regiment and one artillery battery.
One of the great injustices here in the USA is the continued mistreatment of people of color after the Civil War. Almost certainly, their presence in numbers turned the tide in favor of the Union (though the Union was going to win anyway just based on industrial output - in my opinion). The south was a left-over feudal society with chattel slavery rather than feudal obligations which were little better than a form of slavery.