Channel Dash - attacking Brest?

Since it was the Channel Dash recently it got me thinking. What was stopping the US Navy and Royal Navy from launching a large scale strike on the port at Brest to put the ships in question completely out of action? I know RAF Bomber command were doing raids against the port in 1941. I’m just wondering why a larger combined Air/Sea strike was never executed?