-EDIT
TLDR: If the Barbarians and Romans have different play styles it would encourage raiders to settle and the defenders to drive off intruding armies. Skip to fourth paragraph.
-EDIT
Yes, yes. I've been keeping up to date with development on your blog. Think I might do a 7th century Byzantine campaign with them.
Oh, I've been having a quick think about your conundrum and I haven't come up with much. Only stuff that seems better suited to larger or longer campaign games, not a 3-battle system (if that still applies to these rules, I'm working under a lot of assumptions).
When I've used your rules it seemed like the 'defending' army spent most of the time driving off the intruding army rather than actually defending against it, so in my Slav's V. Byzantines game I allowed the Slavs to raid settlements and the local countryside for activity points by spending at least 2 uninterrupted months in any given place, certain settlements might give more points after longer 'sieges' i.e. Constantinople. I found this gave an imperative to actually fight battles, because in another campaign both armies just sat around for a year and swelled to 16 units each before fighting one another.
So this is my solution to your problem. The two sides function differently. The defending side functions normally, 'harvesting' activity points while raiders have to raid to get their points. If a raider army hangs around long enough it colonises that region and can start harvesting activity points like the defender can. All this would require a much longer playing time and probably a much larger map also, maybe multiple defending armies, I'm not sure. The card mechanic could still remain it might just have to be altered in some way, determines how well an army raids or how good a crop is, I'd assume they have slightly different 'difficulty' settings, ie the stability of tax and harvest means it is more reliable method of gaining activity points than raiding.
I hope that makes sense, just occurred to me. I hope that you might find any of it useful, even just to think about and then discard. Just trying to be helpful!