I blame McArthur and Earle Harvey.
Let's start with black powder weapons. Black powder has only so much velocity so that you have to increase the bullet diameter to increase the effectiveness of the shot fired.
Once smokeless powder was invented people figured that you could now get far better results from smaller bullets. This is where people figured that you could shoot much further than was previously practical and this would confer the soldier a huge advantage in combat. After all the practice had been shown to be extremely effective with artillery.
The problem is that large, full power military rifles were never based on reality. They were based on several assumptions, the first being that you could now make rifles and ammo that could consistently nail the target at 500+ yards with great ease, the second was that you could shoot enemy troops at a range where they were still lined up in marching columns so that if you did miss the individual, you would probably hit somebody else.
A wonderful theory, except that in the immortal words of Captain Blackadder "It was "
The reality that with a few exceptions the average solider will never be as accurate as the rifle they carry. Some rifles came with near match-grade sights and the average infantryman couldn't be bothered because in reality they could rarely even see an enemy at 600 yards, let alone hit one. It worked with artillery because the guns had sophisticated sights and even if you missed by a dozen yards, the ammo was of such a nature that a miss was still likely to have some effect anyway.
Highly accurate long range rifles means that soldiers will change the way they act on the battlefield. If anybody still had illusions that you could march your troops in blocks up to the enemy and then devastate them with accurate firing by ranks was buried at the Marne. Dig in, then shoot became the rule, it was so effective that military manuals had to devote more time on how to fire and maneuver than on proper military things like marching, the proper jaunty angle for wearing your cap or shining your buttons !!!
Because these big powerful rifles didn't have much of a downside nobody bothered until they started to play with automatic weapons.
As long as your automatic weapon is a proper, water-cooled, tripod mounted piece of complex machinery, the ammo will perform as expected. Try it with an automatic rifle and it's not much fun to shoot. People either tried to work around the ammo and improve the rifle as much as possible or they did the other thing, use a lighter, less powerful bullet.
Take the BAR, one of the earliest automatic rifles. It was designed to fire semi-auto when -walking- towards the enemy, using a cup on the belt to absorb the recoil. Once you got in close, the gunner would switch to automatic and hose down the trench.
Never mind that the concept is , they try to work around the idea that the ammo is something inviolate. You couldn't rob a soldier from the opportunity of being able to shoot an enemy at 700 yards, would you ? Even if the soldier wasn't even trained to shoot that far and lacked the obvious equipment like a proper scope.
Those who did try to alter the ammo often encountered difficulties. Even using pistol-sized rounds made automatic weapons inaccurate when firing full auto. Semi-automatic rifles were seen as the most likely candidate and making the ammo a little bit lighter both in weight and recoil would be to the advantage of the soldier.
It was also grudgingly being accepted that a full size rifle was not as effective as previously thought. Many armies like France, Italy, Japan, USSR, Germany, Britain all looked into new cartridges and shorter rifles.
The US tried it with the .276 Pedersen and the Garand rifle. It would have been a 10-shot rifle using a reduced power cartridge. Except that McArthur didn't trust a rifle that didn't fire a high octane round because doing so would be an offense to already crumbling beliefs.
When it was established that in WWII the average maximum engagement range was 300 yards and the fact that several "intermediate cartridges" had performed very well over the years, especially with the German STG-44, which pointed many designers in a new direction, resulting in the AK-47 and the FN FAL in .280 British. Conversely the US started to look for a replacement of the M1 Garand.
Meanwhile Earle Harvey, a very talented designer, had entered his T25 rifle for the trials. His rifle was just as easy to make as the M1 and while the mechanism was heavily beefed up, he lightened the weapon in other areas making it light, handy and very tough, but it failed to pass the reliability tests and despite the fact that everybody seemed to favour Harvey's design, the design was dropped and another round of tests would be made, including this time the FN FAL and the British EM2. To make up for having lost, Harvey was added to the selection committee and he decided that if his T25 had failed at least he would make sure no foreign rifle concept would make it.
Harvey was apparently the kind of guy who would stand in a downpour, on the verge of catching pneumonia, drenched to the bone and could still convince you it wasn't raining. He twisted the reports on the test results in such a way that the new T65 cartridge (a scaled down full-powered version of the .30-06 rifle) which was designed to be lighter and used as semi-auto only with full-auto as a kind of "emergency option" He sold the T65 ammo and the T44 Rifle (which would become the M14 rifle) as the perfect rifle, far superior to anything the FAL and the EM2 could achieve whereas in reality both weapons performed exceptionally well. The US accepted the M14 rifle and the 7.62mm as the new standard.
When it came to selecting a standard NATO round, the US bean counters vetoed any deal that would imperil the change that had been implemented and in the end the British relented and accepted the 7.62mm for service.
It was during Vietnam that Harvey's mistake became clear. The M14 was a fine semi-automatic rifle with the ability to fire full auto if it was ever required. When faced by enemy troops carrying AK-47 rifles the M14's doctrine went over board as soldiers wanted to be able to retaliate in kind. This wasn't easy with the M14 in a manner highly reminiscent of the problems the Krag-Jorgensen had when facing Mausers, and less than a year after it was introduced, they were already discussing a replacement.
There is a very long list of attempts made by the US to come up with a suitable bullet, most died quietly, only to end up in the collections of the Aberdeen Proving Grounds.