A System to Transcribe Documents in European Languages withHuman Help

Ravishankar Chityala1 and Sridevi Pudipeddi21 Minnesota Supercomputing Institute, University of Minnesota, Minneapolis, USA

2 Mathematics Department, Waldorf College, Forest City, Iowa, USA

Abstract— Existing text based CAPTCHAs can only tran-scribe words in an image that contains ASCII characters. Inthis paper, we are describing a system called UCAPTCHA,which is intended to transcribe words in an image thatcontains Unicode characters to text. Such a system willbe useful in transcribing scanned documents from manyEuropean languages like Spanish, German, French etc.

Keywords: CAPTCHA, OCR, Unicode, Image transcrip-tion.

1. IntroductionCompletely Automated Public Turing test to tell Comput-

ers and Humans Apart (CAPTCHA) present a problem thatonly humans can solve and computers may not solve. Thissystem can tell the difference between computer and human.In this age of world wide web, CAPTCHAs can be used toprevent spamming.

There are many different types of CAPTCHAs. One typeof CAPTCHA is the hand written CAPTCHA [2] (figure 1)that uses database formed of handwritten names of Americancities which were selected from postal letters. An image of arandomly selected city is shown to user to gain entry to theservice. But poor quality images can make it hard for theuser. EZ-Gimpy and Gimpy CAPTCHA [4] - selects wordsfrom its dictionary of 860 words and displays them corruptedand distorted in an image to gain access to the service.Due to the small size of the dictionary, this CAPTCHA wasbroken. Audio Based CAPTCHA [9] - make use of simplewords or numbers drawn from a random recordings and alterthem with some noise or disturbance. These CAPTCHAsproved to be useful for visually impaired people. In general,these CAPTCHAs are hard to solve due to accent andlanguage barriers. More CAPTCHAs can be found in thesepapers [6], [1], [5], [3]. Shah and Banday gave a nicesummary of some CAPTCHAS with a short description onhow each one works in their paper [7].

2. MotivationAlthough there are many CAPTCHAs, in this paper we

will be focusing on an image based CAPTCHA. The motiva-tion for this CAPTCHA was the image based reCAPTCHA.reCAPTCHA [9] (figure 2) is a free web based CAPTCHAservice that helps in digitizing books. In reCAPTCHA,

Figure 1: Hand Written CAPTCHA

the user is presented two words, obtained by scanning ofbooks, the “control“ word and ”unknown“ word. The user isexpected to solve the control word correctly and his responseto the unknown word is recorded. By presenting the sameunknown word to many different users, an accurate transcrip-tion of the unknown word can be obtained. reCAPTCHApresents words that are in ASCII character set and hence canhelp digitize and transcribe books, documents in English.

Figure 2: reCAPTCHA

Many of the European languages like Spanish, French,German, Italian etc. have many characters that are similar toASCII but differ by a few accented characters. In this paper,we describe a CAPTCHA system, called UCAPTCHA thatwill present words having Unicode characters. The users canrespond to most of the characters using ASCII charactersfrom their keyboard and complete the Unicode charactersusing on-screen keyboard. This system will help transcribebooks, documents etc. in European languages like Spanish,German, French etc. And hence increase the utility of

CAPTCHAs in transcription of valuable documents, maps,books etc.

In a typical setup, the user will be presented doublewords obtained from scanned documents either as a partof a HTML forms in some website or through our sitehttp://ucaptcha.msi.umn.edu. One of the words in the setuphas not been transcribed while the other word is transcribed.We will be referring to them as "unknown word" and "knownword" respectively. When the user solves the CAPTCHA, weevaluate their response to the known word. If the responsewas correct, we assume that the unknown word was re-sponded correctly and record both responses. If the responseto the known word was incorrect, we present an errormessage and request the user to complete the UCAPTCHAagain.

3. Methodology

In a CAPTCHA based system, the words shown to userswould be obtained by scanning the documents that need tobe transcribed. The scanned words that could be transcribedusing an Optical Character Recognition (OCR) programwill be converted to text. Some of the words may not betranscribed because they might be ill formed due to distortionand page curvature, artifact from the scanning process, poorquality of the original document etc. Such words that cannotbe transcribed by computers will be transcribed by a human.The first step therefore is to convert scanned documents in totext using OCR. During this process, some of the words maynot be transcribed. The details and the images of the wordsthat could not be transcribed are uploaded to a databaseand served using a web application to users. We used twodifferent OCR programs, GOCR [10] and Tesseract [11] toconvert scanned documents to text. The text files generatedby both the software were compared and the words thatdid not match were flagged as non-matching words and therest were identified as matching words. We also recordedthe location of the words. To obtain the image of the non-matching words and some of the matching words, we usedthe location obtained in the previous step. We then processedthe scanned document based on the following steps, thatwere coded in Matlab, [12] and Python, [13].

• Project the document after adaptive thresholding in thehorizontal direction and obtain a profile

• Segment the projected profile and separate individuallines based on their connectivity

• On the lines which have the non-matching word

– Perform vertical projection after performing dila-tion

– Segment the projected profile of the words– Separate individual words based on their connec-

tivity– Store the individual words as non-matching words

and a few matching words from that line. Even

though we are interested in the non-matchingwords we still require matching words and theimportance of these words will be explained in thesection, "Embedding mode".

Let us consider a document as a prototype to explain theabove-mentioned steps in detail. The document in figure 3 ispassed through GOCR and Tesseract. The figure 4 is a imageof a line which has one non-matching word and anothermatching word from figure 3.

Note: From now on we call matching words in theboth GOCR and Tesseract as known words and the non-matching words as unknown words.

Figure 3: A German document used as a prototype

Figure 4: The fifth line from the German document whichcontains one unknown word and one known word

4. The websiteOnce the words were obtained, we upload the information

like the name of the document, the location of the word

Figure 5: known word from figure 4

Figure 6: unknown word from figure 4

in the document and the location of the word in the filesystem to a MySQL database. During this process, we alsodistorted the word by a randomly distributed distortion. Awebsite available at http://ucaptcha.msi.umn.edu was codedin Django and serves two words. Users can then use anon screen keyboard to input Unicode characters, while theASCII characters can be entered using the regular keyboard.In the double words, one word is known whereas the otherword is unknown. As the double words are presented, theproperly transcribed unknown word will then be moved tothe known words list. The unknown word will be transferedto the known words list provided 95% of the transcriptionare the same.

The words are then presented in two modes: Demonstra-tion mode and Embedding mode.

4.1 Demonstration Mode

In the demonstration mode, we present the two words tothe users (Figure 7) in our website. By default the virtualkeyboard is not on. The user can turn it on using thekeyboard symbol at the bottom of the text box. Figure 8is the view with the virtual keyboard on the web page.Completing the CAPTCHA would take the user to a secondpage that indicates whether the user was successful or not.

Figure 7: View of the webpage in the demonstration modewithout the virtual keyboard on.

Figure 8: View of the webpage in the demonstration modewith the virtual keyboard on.

4.2 Embedding ModeIn the embedding mode, the mode that would be the most

useful way to disseminate the UCAPTCHA to more usersis shown in figure 9. The website that needs to embed theUCAPTCHA, can request the HTML code for embeddingby filling the "Embed in your site" link. After processingthe form, the web master of the embedding site would beprovided a HTML code that needs to be embedded in theirHTML POST form. When a user visiting the site fills thisform and the UCAPTCHA, the response to UCAPTCHAis sent to our site while the other information is passed tothe embedder website. If the user fails in solving the knownword correctly, a string of value zero is returned, otherwisea value of one is returned.

A web-master can also request words from a specific lan-guage when they request the embedding code. Thus, a Ger-man website will serve German words in its UCAPTCHA.This also has an advantage, that the German user solvingthe Unicode can do so using their physical keyboard insteadof the virtual on-screen keyboard. This allows the user tosolve the UCAPTCHA faster and with less hassle. In case,where the user does not have a physical German keyboard,they can use the virtual on-screen keyboard.

5. DiscussionSince there are only two words being presented, it is

possible that users can guess the location of the unknownword. Once located, they could spam the response to theunknown word while filling the known word correctly. Thiswould pass the test we perform to check if the response iscorrect. Such spamming can skew the transcription results. Inorder to prevent users from guessing the location of knownand unknown word, we locate them at random at run time.

Figure 9: View of the webpage in the Embedding mode.

In some cases, we also present two known words to the user.Also, to prevent the users from seeing the same word, wemaintain a large list of words.

6. ConclusionIn this paper, we present a CAPTCHA system useful for

transcribing words from text containing Unicode characters.Such a system will be useful in transcribing scanned docu-ments from many European languages like Spanish, German,French etc.

The CAPTCHA can be in demonstration mode, in whicha known and unknown word are presented in our site. It canalso be embedded in other sites. In both cases, when a userresponds to the CAPTCHA, as long as the response to knownword is correct, we assume that the unknown word was typedcorrectly. The same unknown word is presented to manydifferent users and if more than 95% responses are same,we assume that the unknown word was transcribed correctly.We then extract the unknown word from the unknown listand place it in the known word list.

7. Future workWe have assumed that the scanned document is free of

any distortions and artifacts but in a real life scenario, theseartifacts do exist. We will investigate various computationaland algorithmic approaches in solving this problem.

Also, we will work with European libraries to obtaindocuments that need to be transcribed. This will help thelibraries to transcribe digitize valuable documents.

8. AcknowledgmentWe would like to acknowledge The Supercomputing In-

stitute for Advanced Computational Research, University ofMinnesota for hosting the web application.

References[1] R. Ferzli, R. Bazzi, L. J. Karam, A captcha based on the human visual

system masking characterists, ACME, 2006.[2] A. Rusu, V. Govindaraju, Handwritten CAPTCHA: using the differ-

ence in the abilities of humans and machines in reading handwrittenwords, proc. of the 9th Int’l Workshop on Frontiers in HandwritingRecognition (IWFHR- 9 2004), 2004.

[3] D. Misra and K. Gaj, Face Recognition CAPTCHAs, Proceedingsof the Advanced International Conference on Telecommunicationsand International Conference on Internet and Web Applications andServices (AICT/ICIW 2006), 2006.

[4] L. von Ahn, M. Blum, and J. Langford, Telling Humans and ComputersApart Automatically, Communications of the ACM, vol. 47, no. 2, pp.57-60, February 2004.

[5] M.H. Shirali-Shahreza and M. Shirali-Shahreza, Persian/Arabic Baffle-text CAPTCHA, Journal of Universal Computer Science (J.UCS), vol.12, no. 12, pp. 1783-1796, December 2006.

[6] M.H. Shirali-Shahreza and M. Shirali-Shahreza, Question-BasedCAPTCHA, proc. of Int’l conference on computational Intelligenceand Multimedia Applications, 2007.

[7] Shah N.A and Banday M.T, Drag and Drop Image CAPTCHA,Sprouts: Working Papers on Information Systems, 8(46), 2008;http://sprouts.aisnet.org/8-46

[8] L. von Ahn, B. Maurer , C. McMillen, D. Abraham, and M. Blum,Science, Vol. 321, pp. 1465-1468, September 2008

[9] C. Nancy, Sound oriented captcha, in: Proceedings of the First Work-shop on Human Interactive Proofs, Xerox Palo Alto Research Center,CA, 2002.

[10] http://jocr.sourceforge.net/[11] http://code.google.com/p/tesseract-ocr/[12] http://www.mathworks.com/[13] http://www.python.org