It is tough for James. He doesn't hate you he's just angry at you for causing his brother to leave. That's the way he see's it anyway. He doesn't understand about "taking breaks". And probably couldn't care less.
To him the fighting probably resembles what he see's and hears between you and his mom. He may feel like "here we go again" and fears that nothing is ever going to get better. But this time, the consequences of you and your gf fighting results in him loosing his brother. He's pissed!
This is a very important time in his life, he's a "Big Brother" he probably also feels protective of Brandon, and may feel that he'll never see his baby brother again. He feels Helpless.
What would concern me though is how he expresses his feelings. It looks like he's learned that closing down by ignoring you, is the only thing he could do to make his point.
I hope you encourage him to keep talking about this, and that he can tell you ANYTHING even if you don't like what he says. That means you have to listen to all the stuff that's in his head. It may be about you and it may not be pleasant.
If he's afraid you will get mad at him, he'll never open up.
As far as his mother goes, your relationships are your business just like hers are hers. And I wouldn't be surprised if James doesn't tell her at all, he probably knows it will just cause problems.
I hope it all works out.