Simply use
-fnostrict-aliasing... that's what the kernel people do afaik, who are extremely interested in performance.