Muzzle brakes are used to relieve recoil stresses on the mounting by directing gases to the rear of the gun. Those ports on the sides are actually angled rearwards. Without a muzzle brake you have to either beef up the mount, provide a longer recoil travel or simply wear the fact that the mounting will have a shorter life.
Now, you don't want to increase the recoil travel - the turret is already very cramped. If you beef up the mount, that again consumes internal turret space. So you need to alleviate the recoil, either by mounting more or larger recuperators. The British tended to mount more, rather than larger ones. If you ever get to Puckapunyal and the RAC museum, you'll see a sectioned Centurion turret used as a crew trainer. What is surprising is how short the recoil travel is on the 20 Pdr. There are four, short but prominent recuperators to absorb all that recoil, two above and two below.
So, yes, you could get rid of the muzzle brake if you wanted to but as in anything there are trade offs. APDS can be fired from guns with muzzle brakes, the 17 Pdr had one. It's just that there is a known problem with the petals of the sabot fouling muzzle brakes. So, perhaps a single baffle one instead of the double baffle?
The Germans were BTW by war's end experimenting with fixed guns - letting the armoured vehicle's mass and suspension absorb the recoil, however that was out of desperation to speed up manufacturing time by simplifying the vehicle and saving materials, particularly oil but were only starting to think about actually putting them in Panzerjaeger vehicles when the war ended. After the war no one except the Soviets really paid much attention to it as an idea and even they abandoned it after a few experiments.