Reply To: Extracurricular challenges

@the_cryptographer_formerly_known_as_madness
Correct me if I am wrong but does the order of multiplying and adding the vector make a difference? It appears that both are reversible by multiplying by the inverse matrix and subtracting a vector, only that when reversing text encoded with the addition after encoding, this vector is not just the negative but takes into account all of the shifts and values in the matrix, however is still a value between 0 and MOD.

Example for 2×2:
Shifts S1 and S2
Addition before multiplication:
Encryption:

╭     ╮ ╭      ╮   ╭   ╮  
| A B | | x+S1 | = | m | 
| C D | | y+S2 |   | n | 
╰     ╯ ╰      ╯   ╰   ╯ 

Decryption:
Let a,b,c,d be the modular inverse of the matrix

╭     ╮ ╭   ╮   ╭    ╮   ╭   ╮   
| a b | | m | - | S1 | = | x | 
| c d | | n |   | S2 |   | y | 
╰     ╯ ╰   ╯   ╰    ╯   ╰   ╯ 

Addition after multiplication

╭     ╮ ╭   ╮   ╭    ╮   ╭   ╮   
| A B | | x | + | S1 | = | m | 
| C D | | y |   | S2 |   | n | 
╰     ╯ ╰   ╯   ╰    ╯   ╰   ╯ 

so

╭     ╮ ╭   ╮   ╭      ╮  
| A B | | x | = | m-S1 | 
| C D | | y |   | n-S2 | 
╰     ╯ ╰   ╯   ╰      ╯ 

Decryption:

╭     ╮ ╭      ╮   ╭   ╮  
| a b | | m-S1 | = | x | 
| c d | | n-S2 |   | y | 
╰     ╯ ╰      ╯   ╰   ╯ 

so

╭                   ╮   ╭   ╮  
| am+bn - (aS1+bS2) | = | x | 
| cm+dn - (cS1+dS2) |   | y | 
╰                   ╯   ╰   ╯ 

Giving

╭     ╮ ╭   ╮   ╭         ╮   ╭   ╮   
| a b | | m | - | aS1+bS2 | = | x | 
| c d | | n |   | cS1+dS2 |   | y | 
╰     ╯ ╰   ╯   ╰         ╯   ╰   ╯ 

Where the vector subtracted is equivalent to just a new shift vector.

When decoding both ciphers, I first multiplied by a matrix (which must have been the modular inverse of the key) then added a different shift to every row of the output vector.

I may be completely wrong in which case both ciphers had the addition before multiplication during encryption.

I hope this helps and good luck with the 5×5.

P.S. Use the hint button for 10B in my challenge, it will be near impossible to guess how the cipher works with no extra information (it took me over a year to design the cipher used).