From a commenter at Volokh
The standard essentially is that the work, taken as a whole, appeal to the improper (prurient) interest in religion, and that it lack redeeming secular value. My advice to states wishing to erect 10 Commandments or similar monuments would be to hire pornography defense lawyers from the 1960s and 1970s for advice on how it's done, because their ouevre is now the state of the art in this area. [...] Go ahead and try it! Seriously! Read some of the old obscenity cases, cross out the word 'sex', write in the word 'religion' in crayon, and see if you don't get something astonishingly identical to the Supreme Court's current religious-monument jurisprudence!
For what it's worth, here's my take on the whole Establishment clause controversy: I don't see the clause as creating an individual right at all. Nobody has the right to the absence of an established religion. Suppose that Congress, in blatant defiance of the constitution, decided to establish Tibetan Buddhism as the state religion. Suppose it passed a law declaring the USA to be a Buddhist country, put the Dalai Lama's picture on all currency, put a mandala on the flag, adopted a mantra as the national anthem, and set up a voluntary tax from which priests would be paid. This would be absolutely unconstitutional, but I don't see how anyone's rights would be violated by it. The rest of the 1st amendment, including the Free Exercise clause, protect individual rights; the Establishment clause does not - it merely prohibits Congress from doing something that it would otherwise be allowed to do.
The reason why the Establishment clause is there in the first place is for the benefit of the states. At the time of federation, some of the states had established religions, and were quite happy with them, and saw no need to change them. Meanwhile, Virginia and the Carolinas had no established religion, and were against the whole idea. What all of these states did not want was the federal government establishing a religion over their heads, which would conflict with what they had decided to do at the state level.
The 14th amendment, for the first time, compelled the states to respect certain vaguely-defined individual rights, and eventually the Supreme Court decided that the Bill of Rights could serve as a guide to which rights the 14th amendment meant to impose on the states. And it decided that the whole of the 1st amendment should be read as if it were incorporated into the 14th, including the Establishment clause. Thus, suddenly the states were barred from establishing religion, and most of the cases we've seen since then stem from this decision. And that decision, I submit, was wrong.
Nobody has a right not to have an established religion, so long as he is free to ignore one if it is established. An establishment of religion neither compels anybody to do anything, nor forbids anyone from doing anything, so it makes no sense to regard the prohibition of establishment as a right. And if it's not a right of any person, then what phrase in the 14th amendment can possibly be read as forbidding it? Remember, the 14th amendment does not say "the 1st amendment is hereby extended to the states"; all it says is that the states have to give each person due process, that they can't infringe people's privileges and immunities, etc. The free exercise of religion is a right which, after the passage of the 14th amendment, the states could no longer infringe; if the absence of an established church is not a right then a state that establishes one can't be infringing it.