Thursday, May 1, 2014

The Japanese J-19 FUJI code

In order to protect its diplomatic communications Japan’s Foreign Ministry used several cryptologic systems during WWII. In 1939 the PURPLE cipher machine was introduced for the most important embassies, however not all stations had this equipment so hand systems continued to play an important role in the prewar period and during the war.

One of the main hand systems was the J-19 code, enciphered either with bigram substitution tables or with transposition using a stencil.
Historical overview

The fist Japanese diplomatic system identified by US codebreakers was introduced during WWI and it was a simple bigram code called ‘JA’. There were two code tables, one of vowel-consonant combinations and the other of consonant vowel. Similar systems, some with 4-letter code tables were introduced in the 1920’s.
These unenciphered codes were easy to solve simply by taking advantage of the repetitions of the codegroups of the most commonly used words and phrases. US codebreakers solved these codes and thus learned details of Japan’s foreign policy. During the Washington Naval Conference the codebreakers of Herbert Yardley’s Black Chamber  were able to solve the Japanese code and their success allowed the US diplomats to pressure the Japanese representatives to agree to a battleship ratio of 5-5-3 for USA-UK-Japan. However this success became public knowledge when in 1931 Yardley published ‘The American Black Chamber’, a summary of the codebreaking achievements of his group. The book became an international best seller and especially in Japan it led to the introduction of new, more secure cryptosystems.

In the 1930’s the Japanese Foreign Ministry upgraded the security of its communications by introducing the RED and PURPLE cipher machines and by enciphering their codes mainly with transposition systems.

J-19 FUJI code
The J-19 code had bigram and 4- letter code tables similar to the ones used previously by the Japanese Foreign Ministry. According to the NSA study ‘West Wind Clear: Cryptology and the Winds Message Controversy A Documentary History’ it was used from 21 June 1941 till 15 August 1943.

In terms of security the J-19 FUJI and the similar codes J-16 MATSU to J-18 SAKURA, that preceded it in the period 1940-41, were much more sophisticated than the older Japanese diplomatic systems. They had roughly double the number of code groups at ~1.600 bigram entries and in addition there was a 4-letter table with 900 entries for ‘common foreign words, usually of a technical nature, proper names, geographic locations, months of the year, etc’.
Example of the code table from West Wind Clear’:

Once the messages were encoded using the J-19 code table then they had to be enciphered. There were two cipher procedures used with the J-19, substitution and transposition.

CIFOL VEVAZ substitution
J-19 messages with the indicators CIFOL or VEVAZ were enciphered using bigram substitution tables. A random letter sequence was coupled with the coded text and each pair of letters was substituted using the substitution table. According to the NSA study this cipher procedure was rarely used.

Columnar transposition using a stencil
The basic cipher system used with the J-19 code was columnar transposition based on a numerical key, with a stencil being used for additional security. The presence of ‘blank’ cages in the box created irregular lengths for each column of the text. Three different stencils were used each month with each being valid for 10 days. The numerical key changed daily. There were four different settings for the J-19 system: General, Europe, America and Asia.

Examples from ‘West Wind Clear’:

The use of a stencil and daily changing transposition key was a big improvement over past systems and offered considerable security in that time period. However both Allied and Axis codebreakers would eventually solve this complicated system during WWII.

Importance of the J-19 FUJI code
The J-19 code was important enough for both the Allies and the Axis to devote significant resources in solving it, even going so far as to build special cryptanalytic equipment. The reason they went through all this trouble is that during the war only a small number of Japanese embassies had the PURPLE cipher machine so the rest had to rely on hand systems and one of the main diplomatic codes was the J-19 FUJI.  Intercepted diplomatic traffic from around the world on this system carried economic, political, military and secret intelligence information. 

A special case was the Moscow embassy (moved to Kuibyshev during the war) and their use of the J-19 for communications with Tokyo. It seems that this embassy was either not given a PURPLE machine or perhaps they had to dismantle it in 1941, so they relied on the J-19 for their most important messages. During WWII Japan fought on the side of the Axis but was careful to avoid a confrontation with the Soviet Union. War between the SU and Japan finally broke out in August 1945 but during the period 1941-45 Japanese diplomats were free to collect and transmit important information from the SU on military and political developments as well as their discussions and negotiations with Soviet officials. These messages were a prime target for the Allied and German codebreakers.
Allied exploitation of the improved J series codes

When the new code J-16 MATSU was introduced in August 1940 it proved much harder to solve that the previous Japanese systems. A team of cryptanalysts of the Army’s Signal Intelligence Service, led by Frank Rowlett analyzed this traffic and came to the conclusion that it was a bigram code enciphered with a transposition key. Rowlett realized that this system was similar to the German ADFGVX cipher of WWI that they had already researched extensively as part of their training program. Their effort to solve this Japanese code had not progressed much when they were given copies of the code, the stencils and some of the numerical keys. These had been copied by US Naval Intelligence from a Japanese embassy or consulate in the US. Thanks to these specimens Rowlett’s work became much easier and messages could be read. Changes in the code could be followed by taking advantage of operator mistakes.
From the 1974 interview of Frank Rowlett, pages 25-28:

It is interesting to note that before they solved this code the head of the codebreaking department, William Friedman was very pessimistic about the prospects of success.
During the war traffic on this system increased significantly and the solution of the daily changing settings became a problem for the small group of personnel, so there was an effort to automate the process. The device built was an attachment for standard IBM punch card equipment called the ‘Electromechanagrammer’ or ‘Gee-Whizzer’.

According to the NSA study ‘It Wasn’t All Magic: The Early Struggle to Automate Cryptanalysis, 1930s – 1960s’, p50-51:
‘The Gee Whizzer had been the first to arrive. In its initial version it did not look impressive; it was just a box containing relays and telephone system type rotary switches. But when it was wired to one of the tabulating machines, it caused amazement and pride. Although primitive and ugly, it worked and saved hundreds of hours of dreadful labor needed to penetrate an important diplomatic target. It proved so useful that a series of larger and more sophisticated "Whizzers" was constructed during the war……………….When the Japanese made one of their diplomatic "transposition" systems much more difficult to solve through hand anagramming (reshuffling columns of code until they made "sense"), the American army did not have the manpower needed to apply the traditional hand tests.

Friedman's response was to try to find a way to further automate what had become a standard approach to mechanically testing for meaningful decipherments……………………………………..Rosen and the IBM consultants realized that not much could be done about the cards; there was no other viable memory medium. But it was thought that it might be possible to eliminate all but significant results from being printed. Rosen and his men, with the permission and help of IBM, turned the idea into the first and very simple Gee Whizzer. The Whizzer's two six-point, twenty-five-position rotary switches signalled the tabulator when the old log values that were not approaching a criterion value should be dropped from its counters. Then they instructed the tabulator to start building up a new plain-language indicator value.
Simple, inexpensive, and quickly implemented, the Gee Whizzer reinforced the belief among the cryptoengineers in Washington that practical and evolutionary changes were the ones that should be given support.’

Australian effort
The American codebreakers were not the only ones who were regularly reading this system. Their British allies were also exploiting the J-19 and in Australia a small group called Diplomatic Special Section solved several Japanese diplomatic codes. According to the book ‘Breaking Japanese Diplomatic Codes David Sissons and D Special Section during the Second World War’, p62 the J-19 was the main Japanese diplomatic code used in the period 1941-43.

The solution of the traffic on the Kuibyshev–Tokyo link was one of their main commitments and in page 38 it says:
The report alludes, very briefly, to the high intelligence value of the intercepts of the telegrams exchanged between the Japanese Foreign Ministry and its Ambassador in Russia, Sato Naotake.

It seems that in its coverage of the Kuibyshev–Tokyo–Kuibyshev circuit, the Section was able to provide strategic intelligence of value.
German exploitation of J-19 code

Foreign diplomatic codes were worked on by three different German agencies, the German High Command’s deciphering department – OKW/Chi, the Foreign Ministry’s deciphering deparment Pers Z and the Air Ministry’s Research Department - Reichsluftfahrtministerium Forschungsamt.
From postwar TICOM reports it is clear that these agencies worked on the J-19 with success but not many details are available on their operations.

At this time there is very limited information available on the work of the Forschungsamt. EASI vol7, p82 says that they worked on ‘a transposition with nulls over two and four letter code’. This was clearly either the J-19 or one of the similar J series systems.
At Pers Z it seems they solved it regularly in the period 1941-43. EASI vol6, p29 says:

A major diplomatic code (known to the Germans as JB-57) was solved at the end of 1941 and read currently for about two years. This was a two-four letter book, enciphered by a series or alphabets and/or stencil transposition with nulls.
The Germans were very interested in the messages from the embassy in the SU and according to David Kahn’s ‘The codebreakers’, p444:

the subsequent solutions provided the Germans with information about Russian war production and army activities
At OKW/Chi they not only solved this code but also built a specialized cryptanalytic device called the ‘Bigram search device’ (bigramm suchgerät) for recovering the daily settings. EASI vol3, p65 says:

FUJI, a transposition by means of a transposition square with nulls applied to a two and four letter code. This system was read until it ended in August, 1943. It was broken in a very short time by the use of special apparatus designed by the research section and operated by Weber. New traffic could be read in less than two hours with the aid of this machine.
The ‘Bigram search device’ is called ‘digraph weight recorder’ in the US report ‘European Axis Signal Intelligence in World War II’ volume 2. In pages 51-53 details are given on the operation of this device:

The digraph "weight" recorder consisted of: two teleprinter tape reading heads, a relay-bank interpreter circuit, a plugboard ‘’weight’’ assignor and a recording pen and drum.
Each head read its tape photoelectrically, at a speed of 75 positions per second.

The machine could find a solution in less than two hours and did the work of 20 people, thus saving manpower.

Bigram apparatus versus Gee-Whizzer

How did the German machine compare with the American ‘Gee-Whizzer’? European Axis Signal Intelligence in World War II’ volume 2 points out the differences in their operation and the pros and cons of each device:
If the sections of text chosen from any given message were well chosen from a cryptanalytic viewpoint, the American machine proved much faster than the German machine, because the print of totals was easier for the cryptanalyst to analyze than just the individual values listed by the German machine; but if the sections of text were not well chosen and no true columns were included in the choice, then the German machine ,had the advantage since it recorded all possible juxtapositions quickly, and all true matches were included in the data.

Sources: ‘European Axis Signal Intelligence in World War II’ volumes 2,3,6,7 , TICOM reports I-22, I-25, I-37, I-90, I-124, I-150,  DF-187B , ‘The Codebreakers’, ‘Breaking Japanese Diplomatic Codes David Sissons and D Special Section during the Second World War’, United States Cryptologic History Series IV: World War II Volume X: ‘West Wind Clear: Cryptology and the Winds Message Controversy A Documentary History’, United States Cryptologic History, Special Series, Volume 6, ‘It Wasn’t All Magic: The Early Struggle to Automate Cryptanalysis, 1930s – 1960s’, NSA interviews of Frank Rowlett 1974

1 comment: