Monday, June 1, 2015

The compromise of the State Department’s strip cipher – Things that don’t add up..

During WWII the US State Department used several cryptosystems in order to protect its radio communications from the Axis powers. The main systems used were the unenciphered Gray and Brown codebooks along with the enciphered codes A1, B1, C1, D1 and the new M-138 strip cipher. 

In the period 1940-1944 German, Japanese and Finnish codebreakers could solve State Department messages (both low and high level) from embassies around the world. The M-138-A strip cipher was the State Department’s high level system and it was used extensively during that period. Although we still don’t know the full story the information available points to a serious compromise both of the circular traffic (Washington to all embassies) and special traffic (Washington to specific embassy). In this area there was cooperation between Germany, Japan and Finland. The German success was made possible thanks to alphabet strips and key lists they received from the Japanese in 1941 and these were passed on by the Germans to their Finnish allies in 1942. The Finnish codebreakers solved several diplomatic links in that year and in 1943 started sharing their findings with the Japanese. German and Finnish codebreakers cooperated in the solution of the strips during the war, with visits of personnel to each country. The Axis codebreakers took advantage of mistakes in the use of the strip cipher by the State Department’s cipher unit.
Apart from purely diplomatic traffic the Axis powers were also able to read some of the messages of other organizations that were occasionally enciphered with State Department systems, such as the Office of Strategic Services, the Office of War Information and the Military Intelligence Service.


Postwar reports
Obviously the compromise of State Department codes and ciphers was a significant defeat for the Allies and from the available information it’s clear that both the US military authorities and the State Department leadership were interested in finding out the full extent of the damage. At the end of the war enemy codebreakers were interrogated and their surviving archives were examined for information on US codes.

Although most of the German and Japanese signal intelligence archives were lost at the end of WWII it was still possible to find important documents regarding their operations versus State Department cryptosystems. At the same time it was possible to locate and interrogate some of the people involved in the solution of US codes. The information obtained from these sources coupled with the information obtained during the war (solved Japanese telegrams, information from the Finnish codebreakers etc) means that by the end of 1945 the US authorities had a pretty good understanding of what systems had been solved by the Axis powers.
One would expect that this information (or at least a detailed summary) would be included in postwar reviews of Allied cipher security. Surprisingly this is not the case. The ‘European Axis Signal Intelligence in World War II’ volumes (dated May 1946) give an overview of German efforts against US diplomatic codes but the information on the M-138-A strip cipher is limited to the circular alphabet strips 0-1 and 0-2.

Volume 1 ‘Synopsis’, p6 says:
The U. S. Army Converter M-134A lSIGMYC) and the U. S. Navy Cipher Machine (HCM), furnished by the Navy to the State Department, were not read by the Germans. The State Department Strip systems 0-1 and 0-2 were solved, the former probably through a compromise and the latter through cryptanalysis. Several State Department codes including the Brown code (unenciphered) and Code A-1 (enciphered) were compromised and read, probably from 1938 and 1939, respectively.

……………………………………..
The value of the intelligence which the Germans got from State Department codes and strip ciphers is not accurately known. The strip systems were probably read too late to be of any great value.

Volumes 2, 3, 6, 7, 8 also have very limited information on the diplomatic strip cipher and some of the statements made contradict each other. Volume 2 ‘Notes on German High Level Cryptography and Cryptanalysis’, p82 says:

Cryptanalytic successes against American strip' ciphers were obtained by at least three German agencies. Dr. Rohrbach, cryptanalyst of the Foreign Office Cryptanalytic Section (Pers ZS), who claimed that his group of six cryptanalysts solved the United states State Department strip cipher (0-2) in 1943, without any previous knowledge concerning the general system, required over a year for solution
Volume 3, ‘The Signal Intelligence Agency of the Supreme Command, Armed Forces’, p59 says:

The leading German success in the American diplomatic field was the reading of the American strip systems. We know that the American strip system (0-2) was read by the Foreign Office Cryptanalytic Section (Pers ZS). Whether this is the same system mentioned by Huettenhain as having been broken at OKW/Chi is not known.
Volume 6 ‘The Foreign Office Cryptanalytic Section’, p24 says:

The Germans considered their main-successes with American systems to have included the solution off the Gray Code (called B3 by the Germans), the Brown Code (B8) and the State Department strip systems. The Grey Code had been in use since June 1918, and the Brown Code since 1938. Both systems were readable, the Brown Code having been compromised in 1941. The Strip System 0-1 was partially read in 1941, and the Strip system 0-2 was solved early in 1943. The strip systems mentioned were not read currently, but only after a delay of months.
Yet the same studies state that copies of the 0-1 strips and their keylist were found in the Pers Z archives. Why would the Germans have problems decoding messages when they had both the alphabet strips and the keylist?

Volume 7 ‘Goering's "Research" Bureau’, p74 says:
Paetzel stated that ‘we attempted a strip system and read  it here and there but not currently. We finally gave it up as it took too many personnel.’ He did not remember any of the originators. Traffic was America to Europe but whether Washington-London or Washington -Paris he did not recall. The system employed 30 out of a matrix of 50 strips in a setting.’

Volume 8 ‘Miscellaneous’, p24 says about the Finnish effort:
Other instances of Finnish successes were: Reading of the American strip system, which the Germans called AM 10

Another document, Special Research History SRH-366 'History of Army Strip Cipher devices' (dated 1948) repeats the same story in page 121:
There is also available now from TICOM studies information on German and Japanese cryptanalysis on Army and State Department strip systems. The most successful work was achieved by the Cryptanalytic Section (Pers ZS) of the German Foreign Office, which read our diplomatic strip traffic until sometime in 1944. During this period the State Department was using the ‘split generatrix’ procedure. After channel elimination was adopted, German cryptanalytic success appears to have ceased. From all available information, Japanese success on our diplomatic traffic appears to have been confined to physical compromise only.


Criticism of EASI volumes
From the information presented so far it is clear that despite having access to important Axis codebreakers and some of their archives the US authorities only had a very general idea of how the strip cipher was exploited during the war. The EASI volumes only mention circular strips 0-1 and 0-2 and they claim that ‘the strip systems were probably read too late to be of any great value’.

This is strange since they knew both from TICOM interrogations and ULTRA intelligence that several sets of strips had been solved during the war. Each US embassy had a set of ‘special’ strips used for direct communications with Washington and a set of ‘circular’ strips for decoding messages sent from Washington to all embassies and for intercommunication between embassies. The German agencies had an arrangement whereby OKW/Chi would attack the special strips and Pers Z the circulars (1).
Erich Huettenhain, chief cryptanalyst of OKW/Chi (Signal Intelligence Agency of the Supreme Command of the Armed Forces) said in report I-145:

In the course of time, as a result of compromises or partial compromises of the traffic on this key, or with the aid of other readable cypher traffic, other sets of strips were discovered by cryptanalysis. We can no longer state how many different sets of strips were reconstructed; probably 10 to 20’.
Considering that each strip set was used by more than one embassy and most used the same keylist (2) this seems to have been a serious compromise of US diplomatic traffic. Yet there is no indication from the TICOM reports that the US authorities tried to find out which specific strips were solved, which embassies used them and how much traffic was decoded. EASI volume 1 just says ‘The value of the intelligence which the Germans got from State Department codes and strip ciphers is not accurately known’.

Unfortunately important TICOM reports such as I-31Detailed interrogations of Dr. Hüttenhain, formerly head of research section of OKW/Chi, at Flensburg on 18-21 June 1945’, I-89Report by Prof Dr. H Rohrbach of Pers Z S on American strip cipher’ and DF-15Reports of group Athat were listed in the sources of the EASI volumes are still classified by the NSA. If these reports contain details about the German exploitation of the diplomatic strip cipher then this would raise questions as to why this information was not included in the EASI volumes.
Information that contradicts the official version of events

Even if the postwar TICOM interrogations did not have details on the strip cipher case this still doesn’t excuse the limited information found in the EASI volumes. During the war the US authorities solved Japanese military attaché messages that contained information on State Department codes and ciphers, including actual M-138-A strips and keylists. If that wasn’t enough they were also able to interrogate the Finnish codebreakers and learn of their work on US codes plus in 1945 they located the surviving archives of OKW/Chi, which included several boxes of decoded US diplomatic messages.
Let’s have a look at each case:

1). Throughout 1943 there was exchange of information on State Department codes and ciphers between the Finnish and Japanese signal intelligence agencies (3). The Finns had managed to solve several special strips in 1942 and in early 1943 they gave copies to the Japanese military attaché so he could transmit this information back to Tokyo.  These messages were in turn decoded by the Allied codebreakers and they clearly revealed the compromised M-138-A strips 10-3, 10-1, 18-1, 4-1, 7-1 (4).


More messages were exchanged regarding US codes and telegram No 101 of March 1943 contained the 33-1 strips while No 102 had solved messages on the 0-1 and 0-2 strips.



The exchange of information was not entirely one-sided since the Japanese shared the strips used by the US embassy in Vichy France (5).



2). In 1944 the exchange of information on State Department systems resumed but this time it was the Germans that shared their results with the Japanese. Germany and Japan had exchanged information on Allied codes and ciphers in 1941 when a Japanese mission headed by Colonel Tahei Hayashi, former head of the Army’s cryptologic agency visited Germany and exchanged US and British codes with systems solved by the Germans.  This promising start did not lead to closer cooperation as communications between Japan and Germany were problematic and the Germans did not trust the Japanese with their most recent codebreaking successes. Things changed in summer ’44, when under Hitler’s orders both M-138-A strips and decoded US messages were given to the Japanese representatives.


According to Wilhelm Fenner, head of the codebreaking department of OKW/Chi (Signal Intelligence Agency of the Supreme Command of the Armed Forces), despite receiving orders to give the Japanese everything they asked for he only shared with them material that would not damage German interests (6).


In July ’44 the Japanese were given M-138-A strips 0-5, 38-1 and 22-1.


In September ’44 the strips 0-2, 0-3, 0-4 were transmitted by the Japanese attaché in Budapest.


The Germans also gave the Japanese decoded State Department messages from Calcutta, Bombay, Moscow and Madras:




It seems that despite statements to the contrary some of them were enciphered with the strip cipher.


The US response downplays the compromise and says that ‘This is of course the old Brown code…’ however a report (7) examining the codes recovered from the Japanese messages shows that a message from Bombay, dated 9 August ’44 was enciphered on the keylist No13 and either the No 20-3 or 20-4 alphabet strips and also mentions a message from Calcutta of August 10, 1944 enciphered on keylist No 19 and alphabet strip 25-4. These seem to be the messages mentioned in telegram 190.
By decoding these messages sent in 1944 the US authorities had complete knowledge of the material sent from Germany to Japan (8).


3). In September 1944 Finland signed an armistice with the Soviet Union. The people in charge of the Finnish signal intelligence service anticipated this move and fearing a Soviet takeover of the country had taken measures to relocate the radio service to Sweden. This operation was called Stella Polaris (Polar Star). In late September roughly 700 people, comprising members of the intelligence services and their families were transported by ship to Sweden. The Finns had come to an agreement with the Swedish intelligence service that their people would be allowed to stay and in return the Swedes would get the Finnish crypto archives and their radio equipment. At the same time colonel Hallamaa, head of the signals intelligence service, gathered funds for the Stella Polaris group by selling the solved codes in the Finnish archives to the Americans, British and Japanese.
The Finns revealed to the American representatives that they had solved several State Department codes and could read the messages from a number of embassies including Bern, Switzerland. A summary of their work was included in the so-called Carlson-Goldsberry report of November 1944 (9). This report was prepared by cryptanalyst Paavo Carlson of the Army’s Signal Security Agency and Paul E. Goldsberry of the State Department’s cipher unit after interviewing Finnish codebreakers and receiving cipher material from them.


This report is still classified by the NSA, however a message sent from Washington in November 1944 says that 18 US cipher systems had been solved by the Finns (10). 


4). In 1945, after the war ended in Europe, the Anglo-Americans launched an operation called TICOM whose goal was to capture the archives of the German codebreaking agencies and interrogate their personnel. The surviving archives of OKW/Chi (Signal Intelligence Agency of the Supreme Command of the Armed Forces) were located at the bottom of lake Schliersee and divers were sent to retrieve the sealed boxes (11). According to a message from William Friedman to Frank Rowlett, dated 13 August 1945 (12), a cursory examination of the messages found in one of the boxes revealed several important US telegrams from 1943 and 1944. When these were matched with the State Department’s original messages it was found that some had been sent on the M-138-A strip system. Friedman’s comments on the discrepancies in this case are revealing:
It is largely a quote from a Note and maybe it was correct to send the message in Brown Code. But why mark it ‘Top Secret’?  If it was not in Brown Code then I am at a loss to account for this one and there would seem to be certainly something "very rotten in the State of Denmark". If, as I assume, the modified strip cipher was in use everywhere after 1 January 1944, and the statement of German cryptanalysts, to the effect that they could not handle the modified version of that cipher, are true then what shall we make of a case like this if the message was not in Brown but in strip? Or are these chaps lying? I am anxious to know as soon as possible what information SSA can dig up on the contents of the package





5). During WWII the US embassy in Bern, Switzerland served as the center of US intelligence activities in occupied Europe. The local station of the Office of Strategic Services was headed by Allen Dulles. In 1943 Dulles received word from Admiral Canaris and General Schellenberg that his communications had been compromised and in addition the German officials Hans Bernd Gisevius and Fritz Kolbe showed him actual decoded US messages. In 1944 he again received German reports containing decoded State Department messages (13).


In this case it’s not clear of which agencies (apart from the OSS) were given access to this material.

Conclusion
In the course of WWII the US authorities received information from various sources on the compromise of their diplomatic communications. Numerous problems with State Department crypto security were also identified in surveys conducted in 1941, 1943 and 1944 (14).

At the end of the war the Anglo-Americans initiated a program called TICOM (Target Intelligence Committee) whose goal was to capture the archives and personnel of the Axis signal intelligence agencies. This information would be then be used to ascertain the security of Allied codes and ciphers. The TICOM program proved to be a great success and tons of files were captured. In addition some of the most important enemy codebreakers were found and interrogated.
The information gathered from the TICOM program was summarized in postwar reports such as the ‘European Axis Signal Intelligence in World War II’ volumes. These reports dealt with all US Army and Navy cryptosystems but when it came to the State Department they just said:

The State Department Strip systems 0-1 and 0-2 were solved, the former probably through a compromise and the latter through cryptanalysis. Several State Department codes including the Brown code (unenciphered) and Code A-1 (enciphered) were compromised and read, probably from 1938 and 1939, respectively.
……………………………………..

The value of the intelligence which the Germans got from State Department codes and strip ciphers is not accurately known. The strip systems were probably read too late to be of any great value.
Leaving aside the question of what intelligence was leaked to the Axis powers as a result of the compromise of State Department systems it’s strange that the only M-138-A strips mentioned are 0-1 and 0-2. As we have seen the US authorities knew from the decoded Japanese military attaché messages that the Finns, Germans and Japanese had solved the circular strips 0-1, 0-2, 0-3, 0-4, 0-5 and the specials 10-3, 10-1, 18-1, 4-1, 7-1, 33-1, Vichy, 38-1, 22-1, 20-3 (or 20-4) and 25-4. These were just the strips mentioned in the Japanese traffic and not necessarily the only strips solved by the Axis (15). Yet the EASI volumes do not mention them. Nor do they mention which systems were solved by the Finnish codebreakers even though they had a detailed report on the subject.

There is also no mention of specific embassies such as Moscow and Bern, whose messages were known to have been read by the Germans through the material found in the OKW/Chi archives and the OSS reports.
The EASI volumes are dated May 1946, so it is understandable that they only had general information on Axis codebreaking activities. Processing all the captured material would have taken years. Yet most of the information on the strip cipher was available since early 1945 (16). With the cooperation of the State Department it should have been easy to identify which embassies used these strips and for how long.

It’s not clear why all the available information on the compromise of the State Department’s strip cipher was not included in the US reports. Hopefully we will learn more once the NSA declassifies the TICOM reports I-31, I-89, DF-15 and the Carlson-Goldsberry report.
Notes:

(1). TICOM D-60 ‘Miscellaneous Papers from a file of RR Dr. Huettenhain of OKW/Chi’, p6
(2). State Department’s strip cipher – reuse of alphabet strips and key lists, Statement of cryptographic systems now in use by Department of State - 1943

(3). British national archives HW 40/132 'Decrypts relating to enemy exploitation of US State Department cyphers, with related correspondence', NARA - RG 457 - Entry 9032 - box 1.018 - ‘JAT write up - selections from JMA traffic'
(4). Note that strips 9-1, 10-1, 18-1 had been copied from a US consulate in Japan in 1939 and given to the Germans in summer 1941. The Germans then gave these to their Finnish allies in 1942.

(5). NSA Friedman collection - telegram Tokyo-Helsinki No 719,  British national archives HW 40/132 'Decrypts relating to enemy exploitation of US State Department cyphers, with related correspondence'
(6). TICOM DF-187F, p29-30

(7). NARA - RG 457 - Entry 9032 - Box 214 - ‘M-138-A numerical keys/daily key table/alphabet strips
(8). NARA - RG 457 - Entry 9032 - box 1.018 - ‘JAT write up - selections from JMA traffic'

(9). Robert Louis Benson and Cecil J. Phillips, History of Venona, p52
(10). NARA - RG 226 - Entry 210 - box 348 - Director’s Office records relating to developments in Sweden, ca. May 1944 – January 1945

(12). NSA Friedman collection - Letter from William Friedman to Frank Rowlett: German decodes of five messages

(13). NARA-RG 226-entry 123-Bern-SI-INT-29 -Box 3-File 34 ‘German intelligence, Hungary
(14). NARA - RG 457- Entry 9032- box 1.384 - 'JCS Ad hoc committee report on cryptographic security of government communications'

(15). Dr Wolfgang Franz, who has in charge of OKW/Chi’s strip cipher program said in TICOM DF-176, p9 ‘All told, some 28 circuits were solved at the Bureau under my guidance, likewise six numerical keys-some of them only in part

(16). ‘JAT write up - selections from JMA traffic' is dated March 1945, the Carlson-Goldsberry report was written in November 1944, the OSS Bern reports were sent in late 1944, Friedman’s ‘German decodes of five messages’ is dated August 1945.

Update: More information on the strips mentioned in the Japanese messages can be found in New developments in the strip cipher case.

No comments:

Post a Comment