Because there are millions of people involved who have millions opinions on the topic and everyone wants their way. Thats why we have the bloody mess called HTML as a compromise.

"Why can't we assume UTF-8 unless told otherwise?"

And this I really do not understand either. I lost hours to restore data, that got lost in conversion from various text encoding formats, discovered too late.