MessageClass mask placement collides with alphabet

[ane]

Suppose I want to do the following:

I want to send an uncompressed Message Class 1 with Latin1 alphabet. This would mean, according to the SMPP specification, that I should have a data coding byte that looks like the following.

00010101
   ^^^^^ bits 0 and 1 are 01 = message class 1
   | |
   | ` bits 3 and 2 are 01 = 8bit 
   |
   ` the message has a message class (bit 4 is 1)

To do this, I do the following:

new GeneralDataCoding(Alphabet.ALPHA_LATIN1, MessageClass.CLASS1, false)

The byte value of this is translated to 0x13, which is 00010011. What is going on here? It looks like the message class and encoding are encoded at the same position. I should be getting 0x15, which is 00010101.

Let's try with an uncompressed UCS-2 class 0 message. According to the spec, I should be sending

00011000
   ^^^^^ bits 0 and 1 are 00 = message class 0
   | |
   | ` bits 3 and 2 are 10 = UCS2
   |
   ` the message has a message class (bit 4 is 1)

Which is hex 0x18. Doing this in Java

new GeneralDataCoding(Alphabet.ALPHA_UCS2, MessageClass.CLASS0, false);

This looks ok! Ok, to condense this a little bit, a test like this

    System.out.printf("0x%02X\n", new GeneralDataCoding(Alphabet.ALPHA_DEFAULT, null, false).toByte());
    System.out.printf("0x%02X\n", new GeneralDataCoding(Alphabet.ALPHA_LATIN1, null, false).toByte());
    System.out.printf("0x%02X\n", new GeneralDataCoding(Alphabet.ALPHA_UCS2, null, false).toByte());
    System.out.printf("0x%02X\n", new GeneralDataCoding(Alphabet.ALPHA_DEFAULT, MessageClass.CLASS0, false).toByte());
    System.out.printf("0x%02X\n", new GeneralDataCoding(Alphabet.ALPHA_DEFAULT, MessageClass.CLASS1, false).toByte());
    System.out.printf("0x%02X\n", new GeneralDataCoding(Alphabet.ALPHA_DEFAULT, MessageClass.CLASS2, false).toByte());
    System.out.printf("0x%02X\n", new GeneralDataCoding(Alphabet.ALPHA_LATIN1, MessageClass.CLASS0, false).toByte());
    System.out.printf("0x%02X\n", new GeneralDataCoding(Alphabet.ALPHA_LATIN1, MessageClass.CLASS1, false).toByte());
    System.out.printf("0x%02X\n", new GeneralDataCoding(Alphabet.ALPHA_LATIN1, MessageClass.CLASS2, false).toByte());
    System.out.printf("0x%02X\n", new GeneralDataCoding(Alphabet.ALPHA_UCS2, MessageClass.CLASS0, false).toByte());
    System.out.printf("0x%02X\n", new GeneralDataCoding(Alphabet.ALPHA_UCS2, MessageClass.CLASS1, false).toByte());
    System.out.printf("0x%02X\n", new GeneralDataCoding(Alphabet.ALPHA_UCS2, MessageClass.CLASS2, false).toByte());

prints

0x00    // ok (no class, default alphabet), 0b0000000
0x03    // ok (no class, latin1), 0b0000011
0x08    // ok (no class, ucs2), 0b0001000
0x10    // ok (class, class0, default), 0b0001000
0x11    // ok (class, class1, default), 0b0001001
0x12    // ok (class, class2, default), 0b0001011
0x13   // not ok, class 0 latin1 should be 0b00010100 (0x14) but this is 0b00010011
0x13   // not ok, class 1 latin1 should be 0b00010101 (0x15) but this is 0b00010011
0x13  // not ok, class 2 latin1 should be 0b00010110 (0x16) but this is 0b00010011
0x18  // ok, class 0 ucs-2 should be 0b00011000
0x19  // ok, class 0 ucs-2 should be 0b00011001
0x1A // ok, class 0 ucs-2 should be 0b00011010

So the default alphabet and ucs-2 work just fine, but something is off with latin1.

I think the reason is in the toByte method of GeneralDataCoding:

    public byte toByte() {
        byte value = compressed ? DataCodingFactory00xx.MASK_COMPRESSED : 0;
        value |= alphabet.value();
        if (messageClass != null) {
            value |= DataCodings.MASK_CONTAIN_MESSAGE_CLASS;
            value |= messageClass.value();
        }
        return value;
    }

For alphabets default and ucs-2, the masks are 0b00000000 and 0b00001000. If the message class is defined, this works correctly. But the mask for latin1 is 0x03, 0b00000011. This is the same as the message class mask! So essentially, due to luck, alphabets default and ucs-2 work. What should be done instead is something like

• If the mask contains no message class, proceed as usual.
• If the mask contains a message class, the alphabet should be set on bits 3-2 and the message class on bits 0-1.

The reason why I'm opening an issue here is to ask for confirmation. Am I thinking this correctly? According to the specification, IF a message class is defined, the alphabet should move to bits 3-2 and the message class should be on bits 0-1.

[pmoerenhout]

I think we have a problem here. The data coding scheme you mentioned, is the GSM 03.38 / 3GPP 23.040 / 3GPP 23.038 specification. This DCS is used in the mobile network as TP-DCS. The SMPP DCS is specified in the SMPP 3.4 / 5.0 specification.

For SMPP, the Latin1 DCS has value 0x03, which can get with GeneralDataCoding(Alphabet.ALPHA_LATIN1).

The GenericCoding class uses the MessageClass and compress bit, which are not in the SMPP spec but in the GSM spec (see also https://en.wikipedia.org/wiki/Data_Coding_Scheme). I will have to go through the GeneralCoding class and see how to correct this. It could be that some GSM only SMPP gateways will uses these value transparently.

Please also not from the SMPP spec:
The data_coding parameter will evolve to specify Character code settings only. Thus the recommended way to specify GSM message class control is by specifying the relevant setting in the TLV dest_addr_subunit.

I also see that in SMPP 3.3, the DCS is defined as GSM Data-Coding-Scheme ( See GSM 03.40 [2] 9.2.3.10), whereby in SMPP 3.4 and 5.0 it is the SMPP specific DCS.

[pmoerenhout]

The GeneralCoding combines the GSM 03.40 and SMPP DCS which is not very clear. The GSM 03.40 only uses the default (GSM), binary and UCS2. But the GeneralCoding doesn't check this. I will refactor this and make a new DataCoding, clearly separating the GSM and SMPP usages.

[pmoerenhout]

After some more thoughts, the Latin1 alphabet is not intended to be used with a message class. The valid calls are below.

I will first add a simple check to verify the alphabet when the message class is specified, like:

public GeneralDataCoding(Alphabet alphabet, MessageClass messageClass, boolean compressed) throws IllegalArgumentException {
    if (alphabet == null) {
      throw new IllegalArgumentException("alphabet is mandatory, can't be null");
    }
    if (messageClass != null && (alphabet != Alphabet.ALPHA_DEFAULT && alphabet != Alphabet.ALPHA_8_BIT && alphabet != Alphabet.ALPHA_UCS2 && alphabet != Alphabet.ALPHA_RESERVED_12)) {
      throw new IllegalArgumentException("alphabet is not supported, only default, 8bit or UCS-2 are allowed with messageClass");
    }
    this.alphabet = alphabet;
    this.messageClass = messageClass;
    this.compressed = compressed;
  }

@Test
  public void testGeneralCoding() {
    assertEquals(DataCodings.ZERO.toByte(), (byte) 0x00);
    assertEquals(GeneralDataCoding.DEFAULT.toByte(), (byte) 0x00);
    assertEquals(new GeneralDataCoding().toByte(), (byte) 0x00);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_DEFAULT).toByte(), (byte) 0x00);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_IA5).toByte(), (byte) 0x01);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_UNSPECIFIED_2).toByte(), (byte) 0x02);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_LATIN1).toByte(), (byte) 0x03);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_8_BIT).toByte(), (byte) 0x04);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_JIS).toByte(), (byte) 0x05);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_CYRILLIC).toByte(), (byte) 0x06);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_LATIN_HEBREW).toByte(), (byte) 0x07);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_UCS2).toByte(), (byte) 0x08);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_PICTOGRAM_ENCODING).toByte(), (byte) 0x09);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_ISO_2022_JP_MUSIC_CODES).toByte(), (byte) 0x0a);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_RESERVED_11).toByte(), (byte) 0x0b);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_RESERVED_12).toByte(), (byte) 0x0c);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_JIS_X_0212_1990).toByte(), (byte) 0x0d);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_RESERVED_15).toByte(), (byte) 0x0f);

    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_DEFAULT, MessageClass.CLASS0, false).toByte(), (byte) 0x10);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_DEFAULT, MessageClass.CLASS1, false).toByte(), (byte) 0x11);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_DEFAULT, MessageClass.CLASS2, false).toByte(), (byte) 0x12);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_DEFAULT, MessageClass.CLASS3, false).toByte(), (byte) 0x13);

    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_8_BIT, MessageClass.CLASS0, false).toByte(), (byte) 0x14);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_8_BIT, MessageClass.CLASS1, false).toByte(), (byte) 0x15);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_8_BIT, MessageClass.CLASS2, false).toByte(), (byte) 0x16);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_8_BIT, MessageClass.CLASS3, false).toByte(), (byte) 0x17);

    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_UCS2, MessageClass.CLASS0, false).toByte(), (byte) 0x18);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_UCS2, MessageClass.CLASS1, false).toByte(), (byte) 0x19);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_UCS2, MessageClass.CLASS2, false).toByte(), (byte) 0x1a);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_UCS2, MessageClass.CLASS3, false).toByte(), (byte) 0x1b);

    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_RESERVED_12, MessageClass.CLASS0, false).toByte(), (byte) 0x1c);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_RESERVED_12, MessageClass.CLASS1, false).toByte(), (byte) 0x1d);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_RESERVED_12, MessageClass.CLASS2, false).toByte(), (byte) 0x1e);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_RESERVED_12, MessageClass.CLASS3, false).toByte(), (byte) 0x1f);

    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_DEFAULT, null, true).toByte(), (byte) 0x20);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_8_BIT, null, true).toByte(), (byte) 0x24);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_UCS2, null, true).toByte(), (byte) 0x28);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_RESERVED_12, null, true).toByte(), (byte) 0x2c);

    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_DEFAULT, MessageClass.CLASS0, true).toByte(), (byte) 0x30);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_DEFAULT, MessageClass.CLASS1, true).toByte(), (byte) 0x31);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_DEFAULT, MessageClass.CLASS2, true).toByte(), (byte) 0x32);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_DEFAULT, MessageClass.CLASS3, true).toByte(), (byte) 0x33);

    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_8_BIT, MessageClass.CLASS0, true).toByte(), (byte) 0x34);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_8_BIT, MessageClass.CLASS1, true).toByte(), (byte) 0x35);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_8_BIT, MessageClass.CLASS2, true).toByte(), (byte) 0x36);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_8_BIT, MessageClass.CLASS3, true).toByte(), (byte) 0x37);

    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_UCS2, MessageClass.CLASS0, true).toByte(), (byte) 0x38);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_UCS2, MessageClass.CLASS1, true).toByte(), (byte) 0x39);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_UCS2, MessageClass.CLASS2, true).toByte(), (byte) 0x3a);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_UCS2, MessageClass.CLASS3, true).toByte(), (byte) 0x3b);

    assertEquals(new MessageWaitingDataCoding(IndicationSense.INACTIVE, IndicationType.VOICEMAIL_MESSAGE_WAITING).toByte(), (byte) 0xc0);
    assertEquals(new MessageWaitingDataCoding(IndicationSense.INACTIVE, IndicationType.FAX_MESSAGE_WAITING).toByte(), (byte) 0xc1);
    assertEquals(new MessageWaitingDataCoding(IndicationSense.INACTIVE, IndicationType.ELECTRONIC_MESSAGE_WAITING).toByte(), (byte) 0xc2);
    assertEquals(new MessageWaitingDataCoding(IndicationSense.INACTIVE, IndicationType.OTHER_MESSAGE_WAITING).toByte(), (byte) 0xc3);

    assertEquals(new MessageWaitingDataCoding(IndicationSense.ACTIVE, IndicationType.VOICEMAIL_MESSAGE_WAITING).toByte(), (byte) 0xc8);
    assertEquals(new MessageWaitingDataCoding(IndicationSense.ACTIVE, IndicationType.FAX_MESSAGE_WAITING).toByte(), (byte) 0xc9);
    assertEquals(new MessageWaitingDataCoding(IndicationSense.ACTIVE, IndicationType.ELECTRONIC_MESSAGE_WAITING).toByte(), (byte) 0xca);
    assertEquals(new MessageWaitingDataCoding(IndicationSense.ACTIVE, IndicationType.OTHER_MESSAGE_WAITING).toByte(), (byte) 0xcb);

    assertEquals(new MessageWaitingDataCoding(IndicationSense.INACTIVE, IndicationType.VOICEMAIL_MESSAGE_WAITING, Alphabet.ALPHA_DEFAULT).toByte(),
        (byte) 0xd0);
    assertEquals(new MessageWaitingDataCoding(IndicationSense.INACTIVE, IndicationType.FAX_MESSAGE_WAITING, Alphabet.ALPHA_DEFAULT).toByte(), (byte) 0xd1);
    assertEquals(new MessageWaitingDataCoding(IndicationSense.INACTIVE, IndicationType.ELECTRONIC_MESSAGE_WAITING, Alphabet.ALPHA_DEFAULT).toByte(),
        (byte) 0xd2);
    assertEquals(new MessageWaitingDataCoding(IndicationSense.INACTIVE, IndicationType.OTHER_MESSAGE_WAITING, Alphabet.ALPHA_DEFAULT).toByte(), (byte) 0xd3);

    assertEquals(new MessageWaitingDataCoding(IndicationSense.ACTIVE, IndicationType.VOICEMAIL_MESSAGE_WAITING, Alphabet.ALPHA_DEFAULT).toByte(),
        (byte) 0xd8);
    assertEquals(new MessageWaitingDataCoding(IndicationSense.ACTIVE, IndicationType.FAX_MESSAGE_WAITING, Alphabet.ALPHA_DEFAULT).toByte(), (byte) 0xd9);
    assertEquals(new MessageWaitingDataCoding(IndicationSense.ACTIVE, IndicationType.ELECTRONIC_MESSAGE_WAITING, Alphabet.ALPHA_DEFAULT).toByte(),
        (byte) 0xda);
    assertEquals(new MessageWaitingDataCoding(IndicationSense.ACTIVE, IndicationType.OTHER_MESSAGE_WAITING, Alphabet.ALPHA_DEFAULT).toByte(), (byte) 0xdb);

    assertEquals(new MessageWaitingDataCoding(IndicationSense.INACTIVE, IndicationType.VOICEMAIL_MESSAGE_WAITING, Alphabet.ALPHA_UCS2).toByte(),
        (byte) 0xe0);
    assertEquals(new MessageWaitingDataCoding(IndicationSense.INACTIVE, IndicationType.FAX_MESSAGE_WAITING, Alphabet.ALPHA_UCS2).toByte(), (byte) 0xe1);
    assertEquals(new MessageWaitingDataCoding(IndicationSense.INACTIVE, IndicationType.ELECTRONIC_MESSAGE_WAITING, Alphabet.ALPHA_UCS2).toByte(),
        (byte) 0xe2);
    assertEquals(new MessageWaitingDataCoding(IndicationSense.INACTIVE, IndicationType.OTHER_MESSAGE_WAITING, Alphabet.ALPHA_UCS2).toByte(), (byte) 0xe3);

    assertEquals(new MessageWaitingDataCoding(IndicationSense.ACTIVE, IndicationType.VOICEMAIL_MESSAGE_WAITING, Alphabet.ALPHA_UCS2).toByte(),
        (byte) 0xe8);
    assertEquals(new MessageWaitingDataCoding(IndicationSense.ACTIVE, IndicationType.FAX_MESSAGE_WAITING, Alphabet.ALPHA_UCS2).toByte(), (byte) 0xe9);
    assertEquals(new MessageWaitingDataCoding(IndicationSense.ACTIVE, IndicationType.ELECTRONIC_MESSAGE_WAITING, Alphabet.ALPHA_UCS2).toByte(),
        (byte) 0xea);
    assertEquals(new MessageWaitingDataCoding(IndicationSense.ACTIVE, IndicationType.OTHER_MESSAGE_WAITING, Alphabet.ALPHA_UCS2).toByte(), (byte) 0xeb);

    assertEquals(new SimpleDataCoding(Alphabet.ALPHA_DEFAULT, MessageClass.CLASS0).toByte(), (byte) 0xf0);
    assertEquals(new SimpleDataCoding(Alphabet.ALPHA_DEFAULT, MessageClass.CLASS1).toByte(), (byte) 0xf1);
    assertEquals(new SimpleDataCoding(Alphabet.ALPHA_DEFAULT, MessageClass.CLASS2).toByte(), (byte) 0xf2);
    assertEquals(new SimpleDataCoding(Alphabet.ALPHA_DEFAULT, MessageClass.CLASS3).toByte(), (byte) 0xf3);

    assertEquals(new SimpleDataCoding(Alphabet.ALPHA_8_BIT, MessageClass.CLASS0).toByte(), (byte) 0xf4);
    assertEquals(new SimpleDataCoding(Alphabet.ALPHA_8_BIT, MessageClass.CLASS1).toByte(), (byte) 0xf5);
    assertEquals(new SimpleDataCoding(Alphabet.ALPHA_8_BIT, MessageClass.CLASS2).toByte(), (byte) 0xf6);
    assertEquals(new SimpleDataCoding(Alphabet.ALPHA_8_BIT, MessageClass.CLASS3).toByte(), (byte) 0xf7);

    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_RESERVED_12, MessageClass.CLASS0, true).toByte(), (byte) 0x3c);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_RESERVED_12, MessageClass.CLASS1, true).toByte(), (byte) 0x3d);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_RESERVED_12, MessageClass.CLASS2, true).toByte(), (byte) 0x3e);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_RESERVED_12, MessageClass.CLASS3, true).toByte(), (byte) 0x3f);

    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_DEFAULT, MessageClass.CLASS0, true).toByte(), (byte) 0x30);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_DEFAULT, MessageClass.CLASS1, false).toByte(), (byte) 0x11);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_DEFAULT, MessageClass.CLASS1, true).toByte(), (byte) 0x31);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_DEFAULT, MessageClass.CLASS2, false).toByte(), (byte) 0x12);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_DEFAULT, MessageClass.CLASS2, true).toByte(), (byte) 0x32);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_DEFAULT, MessageClass.CLASS3, false).toByte(), (byte) 0x13);
    assertEquals(new GeneralDataCoding(Alphabet.ALPHA_DEFAULT, MessageClass.CLASS3, true).toByte(), (byte) 0x33);
  }