Anyone know why this is and how to work around this?
Why this is
As you wrote in the comments, you meanwhile found this stack overflow answer on a related topic. To quote from it:
The rationale behind this is that for rotated pages iText attempts to lift the burden of adding the rotation and translation to page content required to draw upright text and have the coordinate system origin in the lower left of the page of the users' shoulders, so that the users don't have to deal with page rotation at all. Consequently, it also does so for annotations.
Unfortunately for your use case, though, iText only rotates the annotation rectangle when adding a new annotation; if you attempt to read the annotation rectangle later, iText returns the rotated rectangle, it does not undo its "correction"; if you attempt to set the annotation rectangle later, iText does not rotate it.
Even worse, iText's code to rotate the rectangle creates a Rect array in which the coordinates do not appear in the order lower-left, then upper right corner. While the order lower-left, then upper-right is not required by the specification, some programs have issues with other orders.
E.g. if one adds an ellipse annotation to a rotated page using iText without providing an appearance, Adobe Reader (both 9.5 and DC) draw the ellipse outside the annotation rectangle:
But if one provides an appearance, Adobe Reader draws that appearance inside the annotation rectangle:
(Screenshots of outputs of CreateEllipse.testCreateEllipseOnRotated() and CreateEllipse.testCreateCorrectEllipseAppearanceOnRotated() tests with the respective annotation selected)
How to work around this
Because of the limitations of the implementation of the rectangle annotation rotation correction in iText (both that it only is applied upon annotation creation and the untypical way it does so), I would suggest in your case, too, to switch off the iText rotation correction code entirely and do all the rotation yourself.
Unfortunately there is no simple property to switch that off. Instead you need to use a work-around as also explained in the already referenced stack overflow answer on a related topic: First remove the page Rotate entry, then add the annotation to the page, then restore the Rotate entry.