This paper presents a novel Italian to Italian Sign Language Dictionary that displays word translation by means of a virtual character. The Dictionary is linked to MultiWordNet, a lexical and semantic database which includes several languages. The objective is to use it as a learning tool for Deaf people to enhance the learning of written languages.
1 Introduction
Sign language is a gestural visual language with sign as lexical units instead, as words are in spoken languages. Sign languages have their own grammars which are quite different from oral languages [1]. Sign Language is not universal language; each country has its own Sign Language like American Sign Language(ASL), British Sign Language (BSL) and Italian Sign Language (LIS). In some countries more than one Sign Languages are also used e.g. in Switzerland, Swiss-German, Swiss-French and Swiss-Italian Sign Language(s) are used [2] .
Users can not be restricted to a single platform in the sense of operating systems or hardware. As an equal opportunity to everyone, everywhere we have to facilitate deaf users on each platform by providing same facility with same interfacing. In this paper a Platform Independent LIS Dictionary (henceforth referred to as Dictionary) is proposed to support Deaf learning of both Sign Language and written language. It provides full set of lexemes and videos data set and an online interface with MultiWordNet synsets.
This paper is organized as follows: in section 2, related work is described. In section 3, the ATLAS lexicon dataset is explained. In section 4 abstract architecture of the Dictionary is presented. Section 5 concludes the paper.
2 Related Work
There is considerable work in progress on Sign Language dictionaries but many of them have limitations by providing limited functionality i.e. only for a word-to-sign search and only limited lexicon data set is available usually recorded with the help of some signer [3, 4]. A large lexicon project was started for American Sign Language which facilitated large lexicon data set but the purpose was to find similar signs in database by taking new video sign [5]. Although American Sign Language and British Sign language dictionaries are available online but they are just web based dictionaries showing prerecorded videos [6, 7].
3 ATLAS Lexicon Dataset
The Platform Independent LIS Dictionary is part of the Automatic Translation into Sign Languages (ATLAS) Project which aims at automatically translating sentences of the Italian language in the corresponding sequences of signs. The target domain is weather forecast news. Lexemes are basic unit of sentences; each individual lexeme comprises a dictionary. Our Sign Language dictionary has 3082 signs as set of Avatar animations which are rendered through a graphic engine. Each sign is linked to a corresponding Italian lemma. 2689 video animations were created using standard Lingua dei Segni Italiana dictionary. 393 Signs are custom created by a team consisting of expert signers and deaf. The existing corpus contains the translation of 55 Italian to LIS weather forecasts collected from bulletins of national broadcasting network RAI.
Concept mapping is the process to form meaningful relationship in words. It is argued that the concept mapping process promotes and assists meaningful learning by encouraging learners to identify key words, establish relationships with other words and relate new concepts to prior concepts [8]. In MultiWordNet concept mapping is achieved by interlinking nouns, verbs, adjectives, and adverbs through semantic relations of synonymy which are called synsets. Our dictionary is composed of a limited corpus but by linking it with MultiWordNet will enrich the possibility to find a sign by searching the sign for synonym lexeme [9].
4 The LIS Dictionary Architecture
The Sign Language Dictionary accepts Italian lemmas as input and provides as output their representation in the Italian Sign Language. The process for the generation of avatar based signs in the LIS Dictionary is defined as follows
(See Fig 1):
-
A LIS interpreter is recorded signing the lemma of each LIS sign.
-
A raw animation file is produced through the keyframe animation technique.
-
The raw animation file is played within the ATLAS player in order to produce virtual character based motion frames.
-
The frames are converted in movies by means of an automatic procedure.
Figure 1, shows the process of signing a lemma.
Input Italian words are linked to the correspondent movie in a database.
The signs are rendered through the ATLAS Player developed at Virtual Reality and Multi Media Park which is able to perform several functionalities such as the blending of signs in one sentence and the application of procedural trans30
formations on the sign to manage relocation [10]. Below a short description of the visualization module and of the procedures applied in order to produce LIS Dictionary lemmata.
[Fig. 1] The process for the generation of avatar based signs
4.1 The ATLAS Visualization system
The module used for the signs generation is system that generates real-time animations, display them in a player and export them as picture frames. It takes as input the symbolic form of LIS called ATLAS Written Extended LIS (AEWLIS) [11], a sequence of lemmata and a description of the meaning, syntactic number and other semantic, pragmatic and articulatory information.
The signing space in front of the interpreter is structured with locations where the items mentioned in the sentence are positioned and kept for further references in the sentence.
Signs are described by a default animation (stored in a repository, the “signary”), that has to be parametrized with respect to the context given by the specific sentence. When the player receives a sentence in AEWLIS form, it plays the signs in sequence adding manual and non-manual components with well defined priority strategies [12]. To some extent the LIS Dictionary involves the aforementioned Signary and links it with additional information provided by MultiWordNet.
The LIS Dictionary is implemented in such way that its functionality is enhanced according to the availability of resources. For offline users lexicon data set along video signs is stored locally while for online users dictionary is mapped with MultiWordNet synsets in client server fashion. Figure 2 shows an abstract architecture of the Platform Independent Italian Sign Language dictionary.
The dictionary provides same interface for mobile and web users in which user input or selects word from preloaded list of lemmas. LIS video animation is displayed against selected lemma and if the video of desired lemma doesn’t exist user can see any word from synset existing in dictionary to understand the meaning. Availability of signary database locally maximizes the performance as no streaming is required. Browsing through synonyms helps to enhance the usage of same sign for different lemmas.
[Fig. 2] Abstract Architecture of Platform
Independent Italian Sign Language dictionary
5 Conclusion and Future Work
In this paper we discussed the platform independent Italian Sign Language dictionary which is capable to run on several devices and is linked with MultiWordNet. It is available as a Web application and for Phone7, iPhone and Android mobile. Our dictionary contains a large video data set and its structure is flexible to include further signs in future. The Italian Sign Language dictionary is part of he ATLAS project which aims at automatically translating Italian sentences into Sign Language. We plan to develop a web applications that take effort of the LIS Dictionary. The objective is to propose such an interface as a tool to enhance Deaf learning of written Italian Language in schools. Since MultiWordNet has multiple links with other languages such as Spanish, German and Portuguese, it could be helpful for written multi-language learning.
References
[1] Liddell, S. K. Grammar, gesture and meaning in American Sign Language. Cambridge University Press, 2003.
[2] Timmermans, N. The status of sign languages in Europe. Council of Europe, 2005.
[3] Hawayek, A., Gratta, R. D., Cappelli, G. A bilingual dictionary Mexican Sign Language-Spanish / Spanish-Mexican Sign Language.
[4] Vettori, C., Streiter, O., Knapp, J. From Computer Assisted Language Learning (CALL) to sign language processing: the design of e-LIS, an electronic bilingual dictionary of italian sign language & Italian. 2004.
[5] Athitsos, V., Neidle, C., Sclaroff, S., Nash, J., Stefan, A., Thangali, A., Wang, H., Yuan, Q. Large Lexicon Project: American Sign Language Video Corpus and Sign Language Indexing/Retrieval Algorithms. 2010.
[6] Sign Language Online Dictionary and Lessons, ASL. Available at: http://handspeak.com
[7] British Language Online Dictionary, BSL. Available at: http://www.britishsignlanguage.com
[8] Novak, J. D., Cañas, A. J. The Theory Underlying Concept Maps and How to Construct Them 1. 2006. p. 1–33.
[9] Prinetto, P., Shoaib, U., Tiotto, G. The Italian Sign Language Sign Bank: Using WordNet for Sign Language corpus creation. Communications and Information Technology (ICCIT), 2011 International Conference on. 29–31 March 2011. p. 134–137.
[10] Lombardo, V., Nunnari, F., Damiano, R. A Virtual Interpreter for the Italian Sign Language. In Intelligent Virtual Agents: 10th International Conference, IVA 2010, Philadelphia, PA, USA. Proceedings Springer, 2010.
p. 201.
[11] Bertoldi, N., Tiotto, G., Prinetto, P., Piccolo, E., Nunnari, F., Lombardo, V., Damiano, R., Lesmo, L., Principe, A. D. On the creation and annotation of a large-scale italian-Lis parallel corpus. In 4th Workshop on the Representation and Processing of Sign Languages: Corpora and Sign Language Technologies, 2010. 2010.
[12] Lombardo, V., Battaglino, C., Damiano, R. An avatar–based interface for the Italian Sign Language. International Conference on Complex, Intelligent, and Software Intensive Systems, 2011.
