Etymologically, the suffix is hypothesized to have derived from a word in the Proto-Indo-European language ("hand" = a person). The connotation of a person as a pair of hands, in this case as in a participating part of a group, has some inherent negative connotations when not attached to a trade (such as fire-man or horse-man). "Chinaman" would be particularly offensive because it most likely originated as a term during the period of the Californian Gold Rush and the building of the transcontinental railroad, in which Chinese laborers were defined by their status as expendable hands for back breaking and ill-paying work. In that sense, it's a form of diminutive objectification and would have similar, but far less offensive in context, negative connotation when used to describe anybody of a particular nationality.
That being said it's not inherently offensive to refer in that way to people who would be direct descendants of users of that language group. The proper word in English for another English speaker would be Englishman, Scotsman or Irishman. It would not be proper for a non-English speaker like a Chinese man.