For the encrypt a function is required that is normally called Integer to Octet String Primitive or I2OSP. For decryption you need an OS2IP function to convert back to integers. Both are explained in my answer on the cryptography sister site. They are part of the RSA PKCS#1 specifications, for which version 2.2 is specified here.
The I2OSP and OS2IP functions are also used for other cryptographic primitives. For instance, they can be used for Elliptic Curve Cryptography to create flat ECDSA signatures or EC public key representations.
These functions are used to encode/decode to an octet string (byte array) of a given size. This size is normally directly related to the size of the modulus (P in your case) of RSA encryption.
The I2OSP function should be coded like this:
public static byte[] i2osp(final BigInteger i, final int size) {
if (size < 1) {
throw new IllegalArgumentException("Size of the octet string should be at least 1 but is " + size);
}
if (i == null || i.signum() == -1 || i.bitLength() > size * Byte.SIZE) {
throw new IllegalArgumentException("Integer should be a positive number or 0, no larger than the given size");
}
final byte[] signed = i.toByteArray();
if (signed.length == size) {
// (we are lucky, already the right size)
return signed;
}
final byte[] os = new byte[size];
if (signed.length < size) {
// (the dynamically sized array is too small, pad with 00 valued bytes at the left)
System.arraycopy(signed, 0, os, size - signed.length, signed.length);
return os;
}
// (signed representation too large, remove leading 00 valued byte)
System.arraycopy(signed, 1, os, 0, size);
return os;
}
Of course, to use this with the correct size in octets / bytes you should first know the key size in bytes first. For an RSA public- or private key this can be easily calculated from the modulus (in case it is not directly available, as in the Java JCA):
public static int keySizeInOctets(RSAKey key) {
int keySizeBits = key.getModulus().bitLength();
int keySizeBytes = (keySizeBits + Byte.SIZE - 1) / Byte.SIZE;
return keySizeBytes;
}
Note that RSAPublicKey
, RSAPrivateKey
and RSAPrivateCrtKey
all extend RSAKey
which provides access to the modulus. So you can use instances of these classes directly as argument for this method. Of course, the RSA providers in Java already contain I2OSP and OS2IP within the Cipher
and Signature
implementation classes, but the conversion from bit size to byte size (without floating point calculations) could come in handy.
Fortunately, the reverse function is not as complicated:
public static BigInteger os2ip(final byte[] data, final int size) {
if (data.length != size) {
throw new IllegalArgumentException("Size of the octet string should be precisely " + size);
}
return new BigInteger(1, data);
}
I've kept in the size validation so it can be called with the expected octet size, even though it is not required for the calculation itself.