They started to go to the 3" because kamikaze tactics made a larger shell imperative.
This was the reason that the 20mm auto cannon started to fall out of favor in the USN. But I don't think this issue is related to the 40mm vs. 3-inch.
The size of the projectile can have several affects. First is the damage radius and lethal damage radius (how close you need to be for a damaging hit, or for a lethal hit). Second is the range (longer shells retain energy better, and so can fly farther.
You want both larger damage (and lethal) radius for your shells, and you want longer range. This is particularly true when you are facing Kamikazes (or today facing missiles). When it's a bomber, if you do sufficient damage to injure the pilot or damage the engine, etc., you can drive off the attack. This doesn't work if the plane (missile) is already diving in to strike you. Then momentum is on the side of the attacker, even if you kill the pilot or the engine. Unless you do it far away.
But smaller guns have the advantage of higher rates of fire. And this makes up for errors of estimation or aiming, as errors occur in a distribution, and if you put enough stuff in the air you are far more likely that one of those errors is just the error you need to get a hit.
All of those effects are dwarfed by the impact of proximity fuses. Guessing the distance to the target when you are prepping the shell to be loaded is inherently error-prone, and a 5% error in range setting for a shot at 2,000 yards is 100 yards, meaning the shell, even from a 6-inch (150mm) gun, will not be in lethal range. And good luck getting anything close to a 5% error rate on your settings for a plane that is not flying at a steady slow speed and a constant altitude and direction!
All of that goes away with proximity fuses. It take range estimation out of the fusing equation. It will detonate within the best radius your gun-laying can give it. If you aim well, it will detonate within a lethal radius. Smaller shells will still do damage, and still have the advantage of higher rates of fire. But that process stops when you get too small for a proximity warhead.
The USN AA doctrine by late war was really all about proximity fuses and rate of fire. Give me the fastest way to get the most proximity-fused projectiles up there.
At least that's what I have read.
-Mark
(aka: Mk 1)