Anton Andreev's blog

GATE Corpus Quality Assurance

nospam@example.com (Anton Andreev) — Fri, 12 Mar 2010 15:45:25 +0200

The "Corpus Quality Assurance" tool and Annotation Transfer processing resource make it easy to compare the results of two pipelines. (when it works).

You won't find this tool in the "Tools" menu, but at bottom of the GATE interface when you click on a corpus.
In order to use Corpus Quality Assurance we need to two annotations sets so that we can compare them.

1. Create datastore

Create a corpus. Then right click it and use the "Save to datastore" option and point it to your datastore. Next populate the corpus and see how the documents do not appear as usually under the "Language resources" tab, they go to the datastore. Datastores are quite useful as they do not keep all the documents in memory. This way you can annotate a large number of documents before you encounter a problem with GATE or JAVA.

2. Create Annotation Set 1

Warning: Check which version of GATE are you using and what are the plug-ins loaded. GATE can load plug-ins from other GATE versions and instances. You might for example want to test against another version of GATE. But probably the second GATE will load the same list of plug-ins as the one you have just closed. Yes, the core of the GATE framework will be different, but because you load the same plug-ins, you will probably get the same results. So you might end up comparing the same results or wondering why the results are not one you expect. Now go to the File->Manage CREOLE Plugins and do the job of selecting your resources explicitly.

Add a new "Annotation Set Transfer" processing resource to the end of your pipe-line:

inputASName should be blank to use the default annotation set.
outoutASName will be our case "Annotation Set 1".
For more information see the documentation.

Now run the pipeline. You should find the new annotation set below the default one. If needed - collapse the default one to see the new one.

This new annotation set "Annotation Set 1" has been saved in the datastore.

2. Make changes
Replace plug-in versions or load another instance of GATE. Please check the plug-in versions of this new GATE instance.

3. Create Annotation Set 2

Open the datastore created in step 1(if it is not already available).
Open the corpus contained in that datastore (if it is not already available) by clicking the datastore, scrolling down until you find it, then right click and select "Load". It should now be available in its usual place under the "Language resources".
Check if "Annotation Set 1" is available.

Create "Annotation Set 2" by again adding "Annotation Set Transfer" processing resource and setting the correct parameters. Do not forget to remove "Document Reset PR" in the beginning of your pipeline or otherwise it will clear "Annotation Set 1".

Run the pipeline!

4.
Click on the corpus and then on the Corpus Quality Assurance at the bottom. Wait until it initializes, there is a progress bar on the right. this initiazlization is need so that the tool knows what are you annotations types and features so it can fill the lists on the right.

Select "Annotation Set 1" and "Annotation Set 2" as A and B in the first list.
Then select the annotations that you are interested in. The select the features by which you think two annotations are identical.
Select measures and then press the "Compare" button.

Click on "Document statistics". The click on a document where the "Only A", "Only B" or "Overlap" are greater then 0. This means we have a difference, click on the document and then click on the "Annotation Diff" button which is currently the second one on the right top of GATE.

SPARQL count

nospam@example.com (Anton Andreev) — Tue, 16 Feb 2010 11:36:35 +0200

You can test the count implementation of Ontotext in the OWLIM semantic database at the following address: ldsr.ontotext.com/sparql

In the form type the following query:

SELECT * FROM named <http://www.ontotext.com/count> WHERE {?s ?p ?o} LIMIT 100

Please note that this functionality is only available in Big OWLIM 3.2.7 and above.

Running GATE on OpenSolaris

nospam@example.com (Anton Andreev) — Fri, 12 Feb 2010 11:52:24 +0200

This post simply proves that GATE 5.1 NLP framework runs on OpenSolaris 2009.06 without any hassle. OpenSolaris uses bash as its default shell script language.

1. Download the GATE archive. Mine is "gate-5.1-build3431-BIN.zip".

2. Unzip the archive:
# unzip gate-5.1-build3431-BIN.zip

3. Install SUNWj6dev in order to have the Sun JDK 1.6 and not only the JRE.
# pkg install SUNWj6dev
You can learn more about the pkg - OpenSolaris Image Packaging System from here.

4. Issue the following commands in bash:
# JAVA_HOME=/usr/jdk/instances/jdk1.6.0
# echo $JAVA_HOME
Make sure that your JDK is really in /usr/jdk/instances/jdk1.6.0

5. Go to the bin folder of GATE and simply type:
# ./gate.sh

Re-locating/porting a GATE application

nospam@example.com (Anton Andreev) — Wed, 10 Feb 2010 19:21:32 +0200

The following article will give you basic knowledge on how to move a GATE application from one GATE instance to another.

Each GATE application is stored as a .gapp file.
For example KIM's default pipeline is stored in "IE.gapp" file. Indeed a gapp file is an XML file.

1. Gapp file considerations

1.1 Best place to save a gapp file:
Usually this should be somewhere in the tree of a GATE/KIM deployment. When you save your gate application(also called pipeline) the resulting gapp file is generated with relative paths to your GATE/KIM installation.

1.2 File-systems paths:
Usually a path in a gapp file looks like that:

$relpath$../../../C:/Program%20Files/GATE-5.0/plugins/Ontology_Tools

It is not recommended to use absolute paths, but if you do then you need to use the following syntax:

file:///G:/Work/gate-binary/gate-5.1-stable-3431/plugins/Ontology_Tools

1.3 Rename components:
The version of GATE on the destination machine might be different. Many plug-in names were changed from GATE 5.0 to 5.1, so you might find yourself porting an application from an older version of GATE to a newer. For example the plug-in "NP_Chunking" changed to "Tagger_NP_Chunking".

So you need to change this line from:
$relpath$../../../C:/Program%20Files/GATE-5.0/plugins/NP_Chunking
to:
$relpath$../../../C:/Program%20Files/GATE-5.0/plugins/Tagger_NP_Chunking

1.4 Remove components:
Sometimes you just need to remove components that you do not need. The following block describes a datastore:

<corpus class="gate.util.persistence.CorpusPersistence">
<dsData class="gate.util.persistence.DSPersistence">
<className>gate.persist.LuceneDataStoreImpl</className>
<storageUrlString>file:/D:/Ontologies/DataStore1/</storageUrlString>
</dsData>
<persistenceID class="string">Corpus for CP-HU___1255364292687___8444</persistenceID>
<resourceType>gate.corpora.SerialCorpusImpl</resourceType>
<resourceName>Corpus for CP-HU</resourceName>
<initParams class="gate.util.persistence.MapPersistence">
<mapType>gate.util.SimpleFeatureMapImpl</mapType>
<localMap/>
</initParams>
</corpus>

which can be safely removed all together.

2. Other resources:
Other resources include for example your Jape files. If you are using a standard GATE gazetteer then your gapp file refers to a def file which you also need to relocate. If you are using an OntoGazetteer then again your gapp file will refer to an ontology file and a map file that you need to package too.

RDF support for .NET

nospam@example.com (Anton Andreev) — Tue, 02 Feb 2010 16:37:19 +0200

I have found a new library that deals with RDF for .NET: www.dotnetrdf.org

Бионика

nospam@example.com (Anton Andreev) — Tue, 12 Jan 2010 11:01:51 +0200

Интересувам се от "бионика", да го наречем.

Филмът "Ghost in the Shell" и факта, че имаме ЕЕГ апарат(баща ми е невролог) ми дава основата да мечтая за тези технологии. Нека под "бионика" разбираме способността на човешката нервна система да комуникира с външни прикирепени устройства - протези. Става въпрос за контролиране на изкуствени крайници като ръце и крака, а също устроиства които възтановяват зрението. ЕЕГ апарата е (скъпо) устройство, което разчита електромагнитните импулси от мозъка. Тези импулси са много слаби и точно ЕЕГ апарата действа като усилвател. Електроди се поставят най-близо до черепа (без да се отваря - просто ти ровят в косата докато стигнат скалпа) и след това се използва и паста, която дава по-добра проводимост. Може да се прави ЕЕГ и като се отвори черепа - тогава сигнала ще е най-добър естествено.

Нека да обобщя - сигналите от мозъка могат да бъдат разчитани, може да е трудно, но компютрите допринасят за това. Възможно е дори компютъра да се адаптира към нервните импулси на даден човек. В информатиката автоматичната адапатация се прави чрез алгорими за невронни мрежи . Разбира се тези невронни мрежи са под формата на програма, която имитира функцията на човешкия неврон. Тук нещата не трябва да се смесват - тук не говоря за физически неврони, които могат примерно да се имплантират в човека, а за понятието неврон в информатиката. Компютърният неврон съществува само в рамките на софтуер.

Един вариант, който си мисля, е за създване на изкуствена ръка чрез хидравлика. Като интересното е, че за течност на хидравликата може да се използва кръвта. Хидравликата може да доведе до това, че да е нужен само един мотор, който да променя налягането и по този начин да се постигат различи състояния на ръката. Може да звучи малко кофти да се ползва кръв, но нека видим предимствата. Циркулиращата течност освен за хидравлика може да се използва и като източник на енергия - иначе ще трябва да зареждате винаги ръката си с ток, а батериите не са много леки (все още) и не държат дълго (все още). От една страна енергия може да се произвежда от самото движение на кръвта, но няма как да е много или ще затрудним сърцето. Другият вариант е да се използва топлината на кръвта. Има примерно стирлингови двигатели, които работят с човешка топлина и въртят перка само от топлината на дланта. Стирлинговите двигатели са типичният за света на механиката начин за преобразуване на топлина в енергия, когато има значение КПД-то. Примерно при колите няма толкова значение КПД-то и затова те не се ползват там. Проблемът обаче в случая е, че големината на двигателя е доста на фона на произведения ток.
Нанотехологиите ще променят всичко - ще бъде създаден такъв електропроводников елемент, който да конвертира топлина в електричестово - вероятно на първо време с ниско КПД. После кръвта ще преминава през нещо с много голяма повърхност (заемащо малко обем) покрито с този електропроводников елемент и тази комбинация от голяма площ и малки размери ще дава нужната енергия в крайна сметка. Може електричестовото да се увеличи като се пусне някаква мрежа в тялото на човека, която да играе същата роля на "орган", който преобразува топлината на тялото в ток, ако кръвта преминаваща през ръката не дава нужното електрозахранване. А може и ръката да е покрита с боя, която допълнително оползотворява падащата светлина (има такова нещо - click here).

Темата просто е интересна.

По-долу може да видите low temperature Stirling engine:

Кисело зеле

nospam@example.com (Anton Andreev) — Tue, 05 Jan 2010 18:13:43 +0200

Преди да запчнете с приготовленята за зелето, трябва да сте изпили бутилка Becks, 30 грама ракия, да сте замезили с 100 грама трушия и . . . 100 грама варени кестени. Това е тествания вариант - зелето става.

Абе, можше и без тая част - but it feels good, doesn't it?

Зелките се слагат цели та да има за сърми. Кочана се изрязва - прави се конусовиден разрез. Набучват се няколко пъти (с нож) да влезе солената вода, но не прекалено. Сложихме и няколко половинки.

Слага се още:
- 1 царевица (преварява се за по-сигурно)
- 2 дюли (нарязани)
- 1 цвекло за цвят(нарязано на няколко парчета)
- 2 ябълки (нарязани)
- 2 корена хрян(настърган)
- 1 кг морска сол (разтворихме я преди това колкото можеше)
- 1 речен камък (наследствен от Търново)

Нашият бидон е към 70 литра. Сложихме му чучурче за преточване, но се наложи Митко да го смени. После Митко преточи зелето доста пъти.
Държахме го два месеца на балкона и го почнахме. Сложихме му натриев бензоат да спре процеса на ферментация по 1мг на 1л. Камъка може би също има значение за втасването на зелето (мих го преди да го сложим).

Много добро стана. Хапваме го с олио, червен пипер и ракия.

Извадихме листа(добре оформени) и ги напълнихме с кайма, суджук и ориз. Овкусихме сместа преди това с ароматни подправки и вегета. Поставихме сърмите свити и притиснати една до друга в тефлонова тенджера. Покрихме сърмите с вода. Всико това къкри 20 минути и накрая добавихме половин буркан с домашни домати (от село Емен). Варихме още 10 минути и започна плющенето. Друго е когато сам си направиш зелето. Снимки нямаме ... трябваше да ядем. Сърмите бяха разкошни и преядохме ...

Life

nospam@example.com (Anton Andreev) — Fri, 18 Dec 2009 13:01:27 +0200

Life is expensive, but at least you get a free trip round the sun.

Das Leben ist teuer, aber zumindest erhalten Sie eine kostenlose Reise um die Sonne.

Compiling GATE

nospam@example.com (Anton Andreev) — Thu, 17 Dec 2009 17:05:57 +0200

This post explains some steps of how to download and set up the GATE source code. This is needed when you would like to improve something.

SVN checkout: https://gate.svn.sourceforge.net/svnroot/gate

Open Eclipse in a new workspace (recommended). Use File->Import->Existing Projects into workspace->select your gate source dir->Finish.
Use the "Java Element Filters" to hide all "Non-Java elements" to make your project more compact.

Update:
My problems were related to an error that prevented me from downloading all GATE files while svn checking out the source. The problem is due to the fact that a filename allowed on Linux is not allowed on Windows. It was about ".cow:no-iframe" and ":" is not allowed on Windows. This halts the entire svn checkout and made me do all sort of tweaks and patches. The GATE source is over 500 MB and 13 000 files, so make sure you have everything before trying to fix it like me. If you are having problems like me, you should try a checkout on Linux to see if it is an OS dependency. If you copy source from Linux to Windows then you need to check in Eclipse that Properties->Resource->Text File Encoding->other is set to "UTF8".

I am interested in refactoring the code of the Othomatcher. This is the processing resource that matches all annotations of the same entity that they are indeed referring to the same thing. This is needed when a person or an organization is mentioned in different forms in the same document.

Радост

nospam@example.com (Anton Andreev) — Thu, 26 Nov 2009 16:08:16 +0200

След като публикуваха линк на сайта на GATE към блога ми:

[15:02:53] Крум: ми, Тончо, да вземеш да почерпиш ли...
[15:03:14] Филип: eeee
[15:03:25] Филип: да не се надуеш сега
[15:03:26] Филип:
[15:03:36] Филип: браво
[15:08:34] аз: ами ... направо се пръскам честно казано
[15:08:50] аз: тука овиках света, питай другите
[15:08:52] аз:
[15:09:15] Ясен: verno verno, dopredi malko be6e na prozoreca i vika6e

5 Favorite Songs

nospam@example.com (Anton Andreev) — Fri, 20 Nov 2009 14:28:38 +0200

This song and image are from Ghost in the Shell - "Lithium Flower". I love this song as I find it both romantic and energetic.

This song and image are from Ghost in the Shell - "Origa - Rise".

This clip is from the new game "Borderlands", song is "Cage The Elephant - Ain't No Rest For The Wicked". You can watch the original clip here with better sound.

Using OWLIM with Aduna OpenRDF Sesame

nospam@example.com (Anton Andreev) — Thu, 05 Nov 2009 15:48:59 +0200

Summary:
This article covers some of the basics of Sesame and how to deploy Ontotext OWLIM as a semantic repository.
You also need a copy of OWLIM(http://www.ontotext.com/owlim) semantic database from Ontotext. OWLIM comes in two favors: Swift OWLIM and Big OWLIM. Here the Big-OWLIM version 3.2.5 has been used. Big-OWLIM is the more robust and scalable version of OWLIM.
Currently this is not a guide to optimize your OWLIM performance, it just tells you how to install/use OWLIM. In order to configure OWLIM for best performance you need to read the documentation and/or consult Ontotext.

Without installing Sesame
This means to use only the content that comes with the OWLIM archive.

Actually when I first downloaded OWLIM I started searching for something like "start-server", but (from the OWLIM archive) there is no such thing. You can only use Sesame and OWLIM as libraries, not as a server waiting on a specific port. This is because OWLIM comes with some of the Sesame libaries, but not all. This on the other hand keeps the the OWLIM package size small - only 4MB. Also you may want a different version of Sesame.

You can use the "example.cmd" in the "getting-started" folder. This way by using various configuration files you can load ontologies, set queries to execute and do this without writing a single line of Java code. This use of OWLIM as a library is helpful when you would like to complete some task and release the memory.

Installing OWLIM in Sesame
The benefit of this option is that by having Sesame running as a "server" you actually use Big OWLIM as a "server" in the sense that we are all used to when using a relational regular database. Sesame allows support for external storage and inference engines. This allows you to use the same API (the Sesame API) and use different back-ends(like Big OWLIM for example).

First step is to download Aduna OpenRDF Sesame 2.2.4. There are also newer versions of Sesame, but in this article version 2.2.4 is recommended.
Let's refer to your Sesame installation folder as "SESAME_INSTALL"

Note that you may need to add the Lucene indexing jar file to the SESAME_INSTALL\lib folder. For Big OWLIM 3.2.6 for example version Lucene version 2.9.1 is required to be in your classpath.

What we do next is to enable enable Sesame to work through Tomcat as a server.
Next we deploy the Workbench which is the web UI that we would like to use.

1. Copy the following wars:
openrdf-sesame.war
openrdf-workbench.war
from
(Windows): %SESAME_INSTALL%\war
(Linux): $SESAME_INSTALL/war
to your Tomcat webapps folder.

1.1 Start the Tomcat server
(Linux):NOTE: the user which starts the Tomcat server, must be the same as the user which you logged-in to perform the deployment of Big OWLIM.

2. Sesame uses an application folder to store its files:
(Windows): %APPDATA%\Aduna\
(Linux): $HOME/.aduna

From the command prompt execute:
(Windows): echo %APPDATA%\Aduna\
(Linux): echo $HOME/.aduna
to see the full path.
If you have never used Sesame (just downloaded) maybe these directories are still not created!

To configure the Sesame Workbench go to:
(Windows): "%APPDATA%\Aduna\OpenRDF Sesame"
(Linux): $HOME/.aduna/openrdf-sesame
and then check for "templates" folder. You need to create this directory if it does not exist. You must put in it a repository descriptor file for your new repository. You have to use the "bigowlim.ttl" from Big OWLIM installation folder as a template.

3. Then you need to add the owlim-big-3.2.5.jar from Big OWLIM installation folder to:
(Windows): "%TOMCAT_HOME%\webapps\openrdf-sesame\WEB-INF\lib"
(Linux): $TOMCAT_HOME/webapps/openrdf-sesame/WEB-INF/lib

4. Open http://localhost:8080/openrdf-workbench to get to know the Sesame UI workbench.

And here comes the bad surprise - OWLIM won't appear in the “new repository” form in the “Sesame Workbench” UI as available Semantic Repository type!

How to enable OWLIM in Aduna workbench
Summary:
Here will learn more details about how Sesame works and the basic configuration files you would need.
The key moment here is that we are going to use Sesame from another client. This is the console client. We are going to create an OWLIM repository through the console client and then this repository will be usable through the Sesame Web Workbench.

First you need to do the above procedure, but this time you need to copy the bigowlim.ttl in:
(Windows): "%APPDATA%\Aduna\OpenRDF Sesame console\templates"
(Linux): $HOME/.aduna/openrdf-sesame-console/templates

and you need to put owlim-big-3.2.5.jar file in SESAME_INSTALL\ib folder.
We are doing this to make the console client aware of OWLIM.

Now there are two ways to continue to create an OWLIM repository and make it available in the Sesame UI workbench:

1.1 Start the OpenRDF Sesame console with:
(Windows): %SESAME_INSTALL%\bin\console.bat
(Linux): $SESAME_INSTALL/bin/console.sh
This way you connect to the default repository in
(Windows): "%APPDATA%\Aduna\OpenRDF Sesame console\"
(Linux): $HOME/.aduna/openrdf-sesame-console

1.2 Connect to the openrdf-sesame.war that you have loaded in Tomcat by entering this command in the console:
connect http://localhost:8080/openrdf-sesame.

1.3 Now you can create a repository which will use
(Windows): "%APPDATA%\Aduna\OpenRDF Sesame\repositories\"
(Linux): $HOME/.aduna/openrdf-sesame

as this is the repository configured for the openrdf-sesame.war with the following command:
create bigowlim.
Do not add the "ttl" extension.

Alternative path to enable Big OWLIM in Sesame workbench.

2.1 Start the OpenRDF Sesame console with:
(Windows): %SESAME_INSTALL%\bin\console.bat
(Linux): $SESAME_INSTALL/bin/console.sh

This way you connect to the default repository in:
(Windows): "%APPDATA%\Aduna\OpenRDF Sesame console\"
(Linux): $HOME/.aduna/openrdf-sesame-console

2.2 Issue the command:
create bigowlim.

to create the new repository. Do not add the "ttl" extension. So now your new repository is in the console config/storage folder
(Windows): "%APPDATA%\Aduna\OpenRDF Sesame console\repositories\"
(Linux): $HOME/.aduna/openrdf-sesame-console/repositories

,but you need it in the config/storage file

(Windows): "%APPDATA%\Aduna\OpenRDF Sesame\repositories\"
(Linux): $HOME/.aduna/openrdf-sesame/repositories

because this is the one used by the Sesame Workbench.

2.4 Copy your newly created repository from:
(Windows): "%APPDATA%\Aduna\OpenRDF Sesame console\repositories\"
(Linux): $HOME/.aduna/openrdf-sesame-console/repositories

to

(Windows): "%APPDATA%\Aduna\OpenRDF Sesame\repositories\"
(Linux): $HOME/.aduna/openrdf-sesame/repositories

Next start again the workbench http://localhost:8080/openrdf-workbench and your new OWLIM repository should be ready to use (you have already created it in the previous steps).

So the bottom line is that you use the Sesame console with config/storage
(Windows): "%APPDATA%\Aduna\OpenRDF Sesame console\"
(Linux): $HOME/.aduna/openrdf-sesame-console

to create an OWLIM repository and you use

(Windows): "%APPDATA%\Aduna\OpenRDF Sesame\"
(Linux): $HOME/.aduna/openrdf-sesame

config/storage with the Sesame workbench.

Special thanks goes to:
Zdravko Tashev
Danail Kozhuharov

Playstation 3 , Linux, Java 6 , Ontotext KIM

nospam@example.com (Anton Andreev) — Tue, 03 Nov 2009 21:16:53 +0200

I successfully installed Yellow Dog Linux 6.2 on Sony PS3. Then I installed the IBM Java VM: ibm-java-sdk-6.0-5.0-linux-ppc.tgz. I couldn't start the IBM JVM 6 the first time. There was some error, so I had to install libstdc++5.0-3.2.2-24565cl.ppc.rpm. I found the needed rpm by using a rpmsearch web-site.

Then I unzipped Ontotext KIM and set the path to the IBM JVM in the /kim/bin/config_machine.sh and it worked (because I had already made some fixes to KIM).

I had to set Yellow Dog Linux to runlevel 3 to free memory. Also YDL uses the unused video RAM as swap space automatically. This theoretically means that the PS3 has 256MB + 256MB Video RAM resulting in total 512 RAM. I have also set 1Gb disk swap space during YDL installation and I think the Video RAM swap file is used with higher priority than the disk based one. I also set KIM to use 440Mb of RAM in /kim/bin/config_kim.sh: export KIM_MAX_JAVA_HEAP="440m". The default is "1g". When connecting to KIM from another machine I also had the usual problems with JAVA RMI. I also decided to turn off the YDL firewall.

KIM works quite slow. It's like 10 documents for 30 minutes. This is due to the supervisor from Sony under which Linux is running. Then comes the really low amount of memory.

Anyway it was fun and proved that I am a geek

Gazetteers - KIM/GATE part 7

nospam@example.com (Anton Andreev) — Mon, 02 Nov 2009 15:35:19 +0200

Gazetteers are called these processing resources in GATE that use dictionary(any data source) to annotate text. This is the simplest and most intuitive way to use predefined knowledge to annotate documents.

The GATE user interface for gazetteers (which is also used in KIM) is not that easy for beginners (at least for me).
Demo 1

Demo 1 shows how to create and use a gazetteer from the user interface. "def" files are these files that contain a list of "lst" files. On the other hand every "lst" file contains items - one per row. It is a 3 level hierarchy. This organization is shown in the demo by creating a new "def" file called "MyGazetter.def". Then we create two "lst" files. After creating the first one (people.lst) we add this "lst" file to the "def" file by using the "insert menu" on the left.

Remarks:
1. Do not forget to save your "lst" and "def" files! That is exactly what happened the first time I pressed "Run Application", so I went back and made sure everything is saved and then I reran the application.
2. While creating the gazetteer in the beginning we can not create a new "def" file by setting the "listsURL" to a non-existent "def" file as this will trigger an error. We do that later on by clicking the "New" button in the "Linear Definition" panel.
3. You can set in the gazetteer options that if several annotations overlap only the longest matters!
4. When we created the "animals.lst" file and started entering values, note that there was no indication anywhere in the interface on which file we were now working on.
5. Also keep in mind that by replacing the ANNIE gazetteer with our own instance, we have to put the new one at the same position as the old one in the pipeline. It was third place. The gazetteer uses some results from previous steps in the pipeline and other GATE processing resources expect results from the gazetteer, so position really matters!

Demo 2

In this demo:
- we see the use of Morphological Analyzer to get the root of a word
- and the use of Flexible Gazetteer to annotate all forms of a word

When we run the standard ANNIE gazetteer we only match "city" and not "cities" in the lookup annotations. We need to add the CREOLE plug-in directory "Tools" to enable both the Morphological Analyzer and Flexible Gazetteer. Next we add the Morphological Analyzer and now we get an additional feature to every token: "Token.root" which contains the root of a word which in our case is "city". The FG(Flexible Gazetteer) does not work the usual way. It does not process the annotations itself, but it works on a selected feature of a selected annotation. So by using the FG we need to select a gazetteer (as this a required parameter to create a FG) so we choose the ANNIE gazetteer to process the "Token.root" annotation.feature. By using the FG we make a standard ANNIE gazetteer see not "cities", but "city" instead. And because "city" is being recognized by default by the ANNIE gazetteer as location, so becomes and "cities".

Demo 2 shows what is the result only with the ANNIE gazetteer and the result from the joint work of the Morphological Analyzer and Flexible Gazetteer.
You may want to see the result produced only by the FG. To do that you need to give a name to the "outputAnnotationSetName" option in the FG, which is available when you click on the ANNIE application and then click on the FG processing resource on the right.

Quick links:
The Semantic Annotation Workflow - KIM part 10
KIM Multi-threaded Clustered Client Application - KIM part 9
Gazetteers - KIM/GATE part 7
Strict Rules vs Machine Learning - KIM part 6
Tips and Tricks - KIM part 5
Using a Gate application - KIM part 4
Gate tutorial - KIM part 3
Using KIM from .NET - KIM part 2
Getting Started - KIM part 1
Installation - KIM part 0

Volvo S40 2,4i Automatic

nospam@example.com (Anton Andreev) — Wed, 28 Oct 2009 15:15:06 +0200

This Volvo costs approximately 9000 euro and seems like the perfect car to me except for the gas expenditures.
But in Bulgaria you can make it run as bi-fuel and save a considerable amount of cash.

The Semantic Annotation Workflow - KIM part 10

nospam@example.com (Anton Andreev) — Mon, 26 Oct 2009 15:45:13 +0200

Click to enlarge

Summary
This short article is more about of process of semantic annotation and not what exactly it is. If you want to learn more about semantic annotation check here. The example here is just one way of doing things and it is used in the Ontotext KIM platform. The article started as of to show the importance of the "Instance URI".

Let's define entity as something that has some value which is worth identifying it. In the case of "named entities" an entity is for example a person or a location presented in a text in the form of: "John Brown" or "Europe".

When we have an entity in the text, that we want to use, we usually get a URI to it that uniquely identifies it. We are going to call this URI the "Instance URI". The "Instance URI" helps us identify when we are talking about the same thing while processing thousands of documents.

Another useful clarification is that "Semantic Database" is a software for storing RDF, it may have or may not have a reasoner enabled and provides an API for SeRQL or/and SPARQL queries.

1. First a Gazetteer is used to locate previously known named entities. A Gazetteer is a GATE processing resource which locates named entities using some database. Usually the database is represented by well organized text files or RDF. If our entity is found in the Gazetteer then we are done, because the Gazetteer has enough information for the entity including its "Instance URI".

2. It's obvious that we can not match everything from some kind of a database. So next we explore the idea of using some rules (in this case GATE JAPE rules) to identify named entities. When tuned, rules work fine in many cases. Using machine learning is better, but it also has shortcomings.

3. Next is the Orthomatcher which is also a processing resource in GATE. The job of the Orthomatcher is complex, it tries to resolve when "Mr. Brown" and "Brown" are the same thing, actually the same person. The Orthomatcher finds the different representations of the same entity and puts the most informative as main. If we have "Sun", "Sun Microsystems", "Sun Microsystems Inc." then we can say that the last one is the most informative. Often the most informative is the longest one as it is suppose to carry more information. So we now have the "most informative representation" of the entity and we know where in the text the author meant the same thing (same entity). We could name it "MIR"(Most Representative Entity).

4. The Instance Generator is the GATE processing resources that generates the URI we are looking for.
- There are default algorithms that uses MIR and generate a unique URI which is the one we call the "Instance URI".
- Then RDF is generated which describes the new entity using the "Instance URI".
- Then the RDF is being send to a semantic database. The semantic database does the job of adding or updating the RDF for this entity.
- In the end the Instance Generator returns the "Instance URI" that was generated based on MIR to the user(you).

Conclusion
It is important to note that every time we have the same MIR then the same URI is generated. This is the default and logical behavior for named entities.

Actually generating URIs is an interesting topic. For example the URI may include the current time-stamp in it. If the time-stamp is precise to the second and because of the fact that we are always going towards the future, then every time we encounter this entity we are going to have a different time-stamp and so a different URI. So we won't be able to have entity co-occurrence because different URIs mean different entities in RDF. In some cases having different instances for the same thing is really what we need.

Remarks:
The "OWLIM" that is classified as a semantic database is a product developed by Ontotext.

Quick links:
The Semantic Annotation Workflow - KIM part 10
KIM Multi-threaded Clustered Client Application - KIM part 9
Gazetteers - KIM/GATE part 7
Strict Rules vs Machine Learning - KIM part 6
Tips and Tricks - KIM part 5
Using a Gate application - KIM part 4
Gate tutorial - KIM part 3
Using KIM from .NET - KIM part 2
Getting Started - KIM part 1
Installation - KIM part 0

Chevrolet Cruze Test Drive

nospam@example.com (Anton Andreev) — Sun, 18 Oct 2009 14:01:54 +0300

Хареса ни Chevrolet Cruze. Имаше оферта за 6 спенна автоматична скоростна кутия за 28 400 лв. Нова кола си викам, на изплащане що да не си я вземем. Двигателят на тая оферта обаче е 1,6 бензин. Има и 2% пълно каско.

Отидох ме до Bulvaria за TestDrive. Там се държаха доста любезно.
Колата на външен вид е наистина страхотна. Проблемът е, че следните неща ми направиха малко кофти впечатление:
- вътре тапицерията е на места с текстил, доста пластмаса и не може да разбереш дали ще издържи дълго или нещо ще се счупи
- колата вози малко твърдо, може да се дължи на факта, че не е много тежка и окачването е твърдо (ние щото сме с френска кола)
- аз разбирам от автоматици и мисля, че има едно закъснение (ама само понякога) от една секунда когато натиснеш газта рязко и очакваш колата да подскочи - да се включи kickdown-a и да се шмугнеш нещо. Една секунда просто нищо не става, но това трябва сами да го тествате, може да е субетктивно.

Иначе колата вървеше добре - на едната права отсечка си ускори добре. Всъщност наистина не е лоша идея колата.

Друг проблем обаче е: "It was in 2005 that all the mainstream models from GM Daewoo were rebranded as Chevrolet in Europe". Това въобще не е същата компания като в Щатитие и това почва да буди подозрения за надежността на колата. Все пак си мисля, че американците правят по-добри атоматични скоростни кутии. Трябва наистина да се прочете някой блог пост на човек, който си е купил и ползва Chevrolet Cruze.

Дразни ме и факта, че отдолу не се слага защита на колите в днешно време. А в България е съвсем "нормално" да си шибнеш картера и да ти изтече маслото следващите 30 метра или да си удариш скоростната кутия. При автоматична скоростна има здрав корпус (точно поради тая причина), но все пак фалове стават - може да ти се счупят точно тия неща дето държат кутията.

KIM/GATE Getting Started - part 1

nospam@example.com (Anton Andreev) — Wed, 30 Sep 2009 14:56:57 +0300

When you install KIM(click here for install post), you have 5 options in general:

Hint: Always stop KIM with \bin\stopKIM.bat to prevent possible data corruption (in the OWLIM semantic database).
Hint: On Linux use the ".sh" extension, not ".bat" to run the scripts.

1. Use KIM-GATE UI

This will start both Sheffield GATE(gate.ac.uk) developer and the Ontotext KIM server. GATE developer is a Java desktop application that communicates with the KIM server automatically.
Use \bin\startKIMGate.bat(sh) to start it and it will load the KIM's default information extraction pipeline which actually has no name, you will see it as "Conditional Corpus Pipeline".

With the KIM/GATE developer you can start annotating documents. Simply said by annotating documents I mean that you will be able to see your documents marked with different colors. Every color will be an annotation, where an annotation can be: personal name, location, time, job position ... etc.

You need to:

- read the GATE documentation, watch the following flash movies.

- know that "Corpus" is a funky name for a bunch of documents. You populate a corpus with documents and then you annotate the corpus - actually the documents in it.

- explore the GATE architecture for things like: "processing resource"/"gate plug-in", "gate application" ... not that many . Applications can be used from both the UI and the Java API.

- know that the GATE developer(from the KIM package) starts with a default application, but when you download GATE from the GATE website, you need to load GATE's default application(pipeline) from the menus with "Load ANNIE with defaults".

2. Import documents through the KIM populator tool

You need to start the KIM server first with \bin\startKIM.bat(sh).(not start startKIMGate.bat!)

Then start \bin\tools\toolPopulate.cmd which on Windows starts the populator with a classic Windows UI.
Point to a folder that contains text or html documents. Press "Start".

This tool will call the "AddDocument" method from the KIM Document Repository API. Usually this method can be configured what operations to be performed when called. It's default behavior is to:
- create a full-text search index over the document
- extract entities and add them to the OWLIM semantic database

Technical remarks:
Both the standalone KIM server(startKIM.bat) and the GATE developer(from the KIM installation) can work in the same time(startKIMGate.bat), but the populator tool needs a running KIM instance - it does not start one for you. You can not start the GATE developer(from the KIM installation) only and expect the populator tool to work.

The see your results you should whether use the KIM WEB UI (see point 4) or the Sesame WEB UI(see part 5) to make queries. “results” means entities, triples, etc.

3. Write a JAVA program that connects to KIM

You can perform semantic annotation make, queries(SeRQL, SPARQL only in 3.0) through a JAVA API. You can also search through KIM Document Repository full text search index.
Please check:
Gate tutorial - KIM part 3
Using a Gate application - KIM part 4

The see your results you can do that programmatically again or use:KIM WEB UI (see point 4) or the Sesame WEB UI(see part 5) to make queries.

4. Explore KIM standard front-end UI

You need Tomcat to start the default WEB UI. I have to admit that many efforts have been put in making this interface really good.
Copy KIM.war from KIM Clients to Tomcat's \webapps.
Start KIM with \bin\startKIM.bat(sh).
Start Tomcat.
With you browser open: http://localhost:8080/KIM/

To try the latest version of the user-interface - visit: latest_news.semanticannotation.com
Many of the features will be disable because KIM 2.4 requires CORE(CORE = Co-Occuring and Ranking of Entities) over a relational database. So a version of Oracle is required. Please check the documentation on How to enable CORE DB with Oracle. Future versions of KIM will be less dependent on Oracle.

5. Use the Aduna Sesame Web UI to write semantic queries

Start Sesame: \bin\startSesame.bat. You do not need to have the KIM server running - see below.
Copy sesame-web-ui.war from \KIM Clients\ to your Tomcat's \webapps folder.
Start Tomcat.
You may experience a problem with Tomcat. This if you have KIM.war deployed. When you start it Tomcat will try to run the KIM.war and if you do not have a KIM instance running this will result in some error messages. This is fixed by either starting KIM(startKIM.bat) or removing kim.war and the KIM folder from webapps in Tomcat. If you remove them then do not forget to add them again next time you try to use the KIM UI.
Start your browser and point it to http://localhost:8080/sesame-web-ui.
Now you probably need to press "Go >>" from the UI to confirm the KIM seasame server.
Next enter the default username/password: admin/admin.

Click on the top on "SeRQL-S" to enter a new select query. Keep in mind that this is a SeRQL query, not SPARQL.

SELECT company, name FROM
{company} <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> {<http://proton.semanticweb.org/2006/05/protonu#Company>},
{company} <http://proton.semanticweb.org/2006/05/protons#hasMainAlias> {alias},
{alias} <http://www.w3.org/2000/01/rdf-schema#label> {name}

This query will first display the company URI in the semantic database that uniquely identifies each company and the second column is the main name that this company is known by. More that 7000 comapnies should be dispalyed from KIM's default knowledge base.

Note: When you use startSesame.bat you start a new OWLIM instance. When you have started a KIM server, you also have OWLIM server running inside KIM. So you might end up with two OWLIM servers running, but that's OK as they are configured on different ports.

Conclusion:
Next step is the KIM documentation itself.
If you have any problems please use the Google Ontotext search engine or/and subscribe to the kim-discussion mailing list.

Quick links:
The Semantic Annotation Workflow - KIM part 10
KIM Multi-threaded Clustered Client Application - KIM part 9
Gazetteers - KIM/GATE part 7
Strict Rules vs Machine Learning - KIM part 6
Tips and Tricks - KIM part 5
Using a Gate application - KIM part 4
Gate tutorial - KIM part 3
Using KIM from .NET - KIM part 2
Getting Started - KIM part 1
Installation - KIM part 0

Debian kFreebBSD installer

nospam@example.com (Anton Andreev) — Sun, 13 Sep 2009 11:16:37 +0300

I was able to install Debian kFreeBSD using the ported Debian installer.

I used the installer from:
[1] http://d-i.debian.org/daily-images/kfreebsd-amd64/
[2] http://d-i.debian.org/daily-images/kfreebsd-i386/

I did that under VMWare. Also the ISO is only 12MB, so it is a net-install. In the end I had a working Gnome installation out of the box.
I am only missing now the Java 1.6. Only Java 1.5 is now available.

Debian kFreeBSD will make Debian more independent OS.
It will increase the package quality as it will force some package maintainers and developers to remove kernel dependent code, and we all agree that kernel dependent code is definitely not a good practice.
In some cases the multi-core support in Freebsd might suit your needs better and you will still be using Debian.

Ontotext KIM Search

nospam@example.com (Anton Andreev) — Fri, 11 Sep 2009 13:11:36 +0300

I have decided to create an experimental Ontotext KIM custom search powered by Google.

This custom search engine includes: KIM documentation, KIM discussion mailing list, GATE users guide and some posts from this blog. Other resources should also be considered.

Update: Posts from the KIM-discussion mailing list are now indexed by Google(so far).