Spirituality of J.S. Mill

I remember years ago in undergraduate philosophy, reading a small book called On Liberty by John Stewart Mill. I remarked to one of my friends how inspiring this book was to me, at which point my friend asked me what on earth I was thinking. Isn't Mill rather "utilitarian"? Why would Mill of all people be so inspiring?
   Looking back on it, I don't think it was Mill's political philosophy that inspired me so much as his implicit spiritual approach to life. Mill was writing at a time when a lot of theories of morality were deontological, or essentially "duty" based. Mill heralded a liberal approach to morality by suggesting that it's not duty that united human beings together in the same cause, but rather the wish to preserve and maximize the most pleasure for the most people. I realize that this is a crude rendering of a complex philosophy, and Mill offered a more nuanced rendition of what utilitarianism means, suggesting that there are even "scales" of moral goods that are based on the relative benefits of certain kinds of activities and pursuits. Nobody could ever accuse Mill of being a hedonist, since he did support the idea of delaying gratification in the pursuit of the highest goods in life. However, the radical departure of Mill from his philosophical contemporaries may very well have been his assertion that morals are not duties imposed on people from some heaven above, but are reflections of the deepest aspirations, hopes and wishes of people. Justice is a seeking of maximizing the collective gain of a diverse group of people. 
   Mill respected diversity of opinion, in part because he was aware that having more inputs can actually strengthen and broaden a person's initial argument. I suspect that what Mill was defending was a more inquiry based way to craft arguments: looking at the other side is a way of deepening one's own argument. But what I respect here is the spiritual approach of allowing others to have their own sense of meaning. Sometimes, if we are very determined to understand something, we are actually trying to assimilate the other's perspective into our own. What if those perspectives are not commensurate at all? If I look at it from a utilitarian perspective, the only way to balance these very divergent possibilities is through allowing everyone to pursue their own special good in life. I may not agree with the goods that you are pursuing (because they are not commensurate with mine), but what I can agree with is that your good doesn't have to be mine, and vice versa. This is a refreshing space which in fact allows people to simply pursue their projects mutually, without a pressure to completely "agree" on everything. There is a space where people honor each other including their disagreements.
   Today, I heard an interesting and funny story about a musical group, Beautiful South, who announced that they were breaking up due to "musical similarities". I laughed when I read this. It's ironic, but is it not what often happens? It's not that people in a group disband because they are too "similar" but rather because they assume that they are supposed to be similar (by default or by association), so there is a tendency to become frustrated when the other person doesn't match one's expectations (or projections). In those moments, it's possible to adopt a Mill perspective, which is to remember that people, no matter how similar, are often complexly different, and complicatedly so. In this way, one can also remember that this difference is what fills people with a sense of mystery, and that mystery is truly precious, not to be taken for granted. Mill's utilitarianism seems "utilitarian", but to what extent is his vision of incommensurate "pleasures" a veiled spiritual vision of alterity and seeing the other as not an appendage of the "self"?