Not sure if anyone has tried this yet, so figured I'd share what I did with the left side deflector since it makes perfect sense to remove it for the reasons you talk about.
Some people have small complaints about the felt heat (when stopped) with the deflector removed. So what I did was trim it down so there was still a bit of a lip sticking out, about 2-inches wide at the very front and narrows as it goes towards the back or bottom. My thinking was if I could deflect the hot air a little bit further out, there would be less heat coming straight up unto my thigh. This lip does help with that, but still doesn't restrict the hot air from blowing out.
I also thought it might help the left side look not so naked with the deflector removed. Not so much. The remaining piece of the deflector is so small there really isn't much difference in how it looks from the side. But it does seem to work for me.
Not the smoothest looking trim job, but I think with a Dremel tool it could be cleaned up.