In Western society, the onus for ensuring understanding falls to the writer or speaker, not the reader or listener. Malcolm Gladwell points this out in "Outliers," in which he discusses how foreign pilots would often crash because they communicated problems too subtly to air traffic control:
Western communication has what linguists call a “transmitter orientation”—that is, it is considered the responsibility of the speaker to communicate ideas clearly and unambiguously. … But Korea, like many Asian countries, is receiver orientated. It is up to the listener to make sense of what is being said.Sometimes, however, clarity in writing is sacrificed for other goals.
In academic journals, writers aim to impress. Writing that is overly technical, poorly organized, and difficult to follow can still build prestige for the writer, even if the underlying arguments are rubbish. Additionally, complicated math is often used for show, even if simpler math would be more appropriate for the topic. Making your paper easier to understand only makes it easier for others to criticize your arguments or data.
In the legal world, writers aim to complicate. Lawyers write contracts and other legal documents in language that can only be understood by other lawyers, keeping them all in business. Lengthy, convoluted documents are less likely to be read carefully by the people signing them, to the advantage of the parties creating such documents.
In academia, writers aim to meet page requirements. A high school student might make his point in 5 pages, but he is forced to include 5 more pages of tangents, needless repetition, and other fluff to reach a 10-page minimum. Similarly, the majority of doctoral students will submit dissertations that are hundreds of pages long, no matter what the topic.