Me too. Co-written with Paul McCartney.
There was a story years ago that I read in some music mag where Costello was at a function with a bunch of other musical celebs. He was talking with or near Bonnie Bramlett (Delaney & Bonnie) and said something derogatory about Ray Charles, using the "N" word. Bonnie hauled off and decked him. A friend later told me that he'd heard that Costello said it because he didn't like Bramlett and was trying to piss her off. Either way, I stopped listening to his music for a long time after I heard it.
Apparently it was true, but I didn't hear the bit about James Brown...
http://ultimateclassicrock.com/elvis-costello-racist-remarks/