| is concerned with nodes, or is concerned with "truth", that is, Boolean values.
Explanation
The | or union operator
This operator returns the union of two sequences that, in this case, are interpreted as two sets of nodes. An interesting detail is that the union operator removes any duplicate nodes. Also, it only accepts operands of the type node()*, i.e. a sequence of nodes. The nodes are returned in document order.
The or operator
The technical term for this operator is Boolean Disjunction. It takes two arguments, both of which must evaluate to a Boolean value ("true" or "false") individually. Or, more precisely, the operands of or are jugded by their effective Boolean value by converting them to xs:boolean. All of this also applies to the and operator, by the way.
Examples
Use the union operator to enlarge a set of nodes, typically in a template match:
<xsl:template match="Root|Jon">
Why not use the or operator here? Because the match attribute expects a set of nodes as its value. union returns a set of nodes, whereas the or operator returns a Boolean value. You cannot define a template match for a Boolean.
Use the or operator to implement alternatives in XSLT code, mainly using xsl:if or xsl:choose:
<xsl:if test="$var lt 354 or $var gt 5647">
If any of the two operands of this or operation evaluates to true, then the content of xsl:if will be evaluated, too. But not only comparisons like "less than" (lt) have a Boolean value, the following is also perfectly legal:
<xsl:if test="$var1 or $var2">
The above expression only evaluates to true if at least one of the two variables is a non-empty sequence. This is because an empty sequence is defined to have the effective Boolean value of false.
Coercion
Note that because of the way XSLT coerces things to appropriate types, there are some contexts where either operator can be used. Consider these two conditionals:
<xsl:if test="Root | Jon"> ... <xsl:if>
<xsl:if test="Root or Jon"> ... <xsl:if>
The first conditional tests whether the union of the set of children named Root and the set of children named Jon is non-empty. The expression Root | Jon returns a sequence of nodes, and then that sequence is coerced to a boolean value because if requires a boolean value; if the sequence is non-empty, the effective boolean value is true.
The second conditional tests whether either of the two sets of children (children named Root and children named Jon) is non-empty. The expression Root or Jon returns a boolean value, and since the operator or requires boolean arguments, the two sets are each coerced to boolean, and then the or operator is applied.
The outcome is the same, but (as you can see) the two expressions reach that outcome in subtly different ways.
