The character was the movie for me. There was little else to it but the development of the character and Phoenix's performance. He was so good that it was worth seeing, as you say there was very little action though until near the end really. The next film should be even better.
Watched Joker last night and really liked it. Thought his performance was stunning. Didn't relate it to the Batman element rather than the sad story of a troubled human who gets kicked around once too often.