November 17, 2014

Weblog der Fachschaft Informatik

Hitchhikerarbeitstreffen am Samstag

Weblog der Fachschaft Informatik

Am Samstag, den 22.11., ab 10 Uhr findet in der Fachschaft ein Hitchhikerarbeitstreffen statt. Dabei soll der gute alte Hitchhiker auf 2014 gebracht werden, um die letzten Änderungen an Studiengang und Rechtschreibung eingebaut werden.

Interessiert, mit zu machen? Dann komm vorbei.

Weblog der Fachschaft Informatik

Wir suchen dich!

Weblog der Fachschaft Informatik

Die Fachschaft braucht euch als

  1. Kassenprüfer (m, w)
  2. Studentisches FBR-Mitglied (m, w)

Das Fachschaftswiki beschreibt den Job der Kassenprüfer:

Die Kassenprüfung findet vor der Vollversammlung, auf welcher der Fachschaftsrat neu gewählt wird, durch zwei gewählte Kassenprüfer statt. Die Kassenprüfer werden auf der vorherigen Vollversammlung gewählt und dürfen keine Mitglieder im Fachschaftsrat sein [...]. Geprüft werden sollte im speziellen:

  • Mittelverwendung: Waren die Ausgaben sinnvoll, bzw. entsprechen sie der Finanzordnung
  • Ausgaben: Entsprechen die (größeren) Ausgaben (stichprobenhaft) den angenommenen Finanzanträgen im Fachschaftsrat (oder sind diese unter dem Freibetrag).
  • Buchführung: Ist das Kassenbuch, die Belegsammlung, die Kontoauszüge, … sauber geführt, geordnet und auch ohne Anwesenheit des Finanzers wieder auffindbar. Wurde das Kassenbuch in einer nicht veränderbaren Weise (durch Unterschriften/handschriftlicht) fixiert und ist vor nachträglichen Änderungen geschützt.

Zum Abschluss der Kassenprüfung muss die aktuelle Finanzlage aller Kassen im Kassenbuch vermerkt, durch Unterschrift der Kassenprüfer (mit Datum) fixiert und inkl. einer Übersicht über die obigen Fragen auf der Vollversammlung berichtet werden.

Der Fachbereichsrat (FBR) setzt sich neben den drei wissenschaftlichen Mitarbeitern, einem nichtwissenschaftlichen Mitarbeiter und neun Professoren auch aus vier studentischen Mitgliedern zusammen. Die letzteren werden am Ende Januar 2015 wieder neu gewählt, und dafür suchen wir Kandidaten.

Der FBR berät und entscheidet in Angelegenheiten des Fachbereichs, so die Fachbereichswebsite. Als studentisches Mitglied hat man insbesondere die Aufgabe, ein Auge auf die Studienangelegenheiten zu legen: Modulhandbuch, Studiengänge, Langzeitplanung, etc.  Wie das konkret aussieht, ist schwer in Texte zu packen, deshalb sprecht einfach mal die aktuellen FBR-Vertreter an oder kommt in die nächste FBR-Sitzung mit. Die Sitzungen sind fachbereichsöffentlich, bis auf den letzten Teil in dem es um Personalangelegenheiten geht.

Wenn ihr das machen wollt, dann meldet euch bis zum 7.12.14 mit einer Mail an kasse.

November 16, 2014

Python-like generator functions, implemented as a library.

This is possible through two simple tricks:

  1. The language represents stack frames as objects on the heap.
  2. There's a native function for grabbing the current stack frame.

Turns out replacing the calling stack frame is really the only thing you need in order to implement coroutines and generics. This is

The implementation is only 23 lines long. On my screen, including the screenshot, this weblog article is already longer up until here.

November 13, 2014

I’ve been using powerline-shell for quite a while and like it a lot. I get aware of that every time I use a terminal which does not tell me which branch I’m on. Some days ago I stumbled upon promptline.vim and as I’m also using vim-airline I gave it a try. Promptline.vim exports a shell […]

November 10, 2014

So you’re the guy who is allowed to setup a local DBpedia mirror or more generally a local Linked Data mirror for your work group? OK, today is your lucky day and you’re in the right place. I hope you’ll be able to benefit from my many hours of trials and errors. If anything goes wrong (or everything works fine), feel free to leave a comment below.

Versions of this guide

There are three older versions of this guide:

  • Oct. 2010: The first version focusing on DBpedia 3.5 – 3.6 and Virtuoso 6.1
  • May 2012: A bigger update to DBpedia 3.7 (new local language versions) and Virtuoso 6.1.5+ (with a lot of updates making pre-processing of the dumps easier)
  • Apr. 2014: Update to DBpedia 3.9 and Virtuoso 7

In this step by step guide I’ll tell you how to install a local Linked Data mirror of the DBpedia 2014, hosting a combination of the regular English and (exemplary) the i18n German datasets adding up to over half a billion triples. If this isn’t enough you can also follow the links to the Freebase, DBLP, Yago, Umbel and datasets / vocabularies adding up to over 3.5 billion triples.

Let’s jump in.

Used Versions

  • DBpedia 2014
  • Virtuoso OpenSource 7.1.0
  • Ubuntu 14.04 LTS


A strong machine with root access and enough RAM: We used a VM with 4 Cores and 32 GBs of RAM for DBpedia only. If you intend to also load Freebase and other datasets i recommend at least 64 GBs of RAM (we actually ended up using a 16 Core, 256 GB RAM Server). For installing i recommend more than 128 GB free HD space for DBpedia alone, 256 GB if you want to load Freebase as well, especially for downloading and repacking the datasets, as well as the growing database file when importing (mine grew to 50 GBs for DBpedia and 180 GB with Freebase).

Let’s go

Download and install virtuoso

Go and download virtuoso opensource: either from (make sure you get v7.1.0 as in this guide or newer version).

Put the file in your home dir on the server, then extract it and switch to the directory:

cd ~
tar -xvzf virtuoso-7.1.0.tar.gz
cd virtuoso-opensource-7.1.0 # or newer, depending what you got

Now do the following to install the prerequisites and then build virtuoso:

sudo aptitude install libxml2-dev libssl-dev autoconf libgraphviz-dev \
     libmagickcore-dev libmagickwand-dev dnsutils gawk bison flex gperf

# NOTICE: the following will _not_ install into /usr/local but into /usr
# (so might clash with packages by your distribution if you install
# "the" virtuoso package)
# You'll find the db in /var/lib/virtuoso/db !
# check output for errors and FIX THEM! (e.g., install missing packages)
export CFLAGS="-O2 -m64"
./configure --with-layout=debian --enable-dbpedia-vad --enable-rdfmappers-vad

# the following will build with 5 processes in parallel
# choose something like your server's #CPUs + 1
make -j5

This will take about 5 min

sudo make install

Now change the following values in /var/lib/virtuoso/db/virtuoso.ini, the performance tuning stuff is according to

# note: virtuoso ignores lines starting with whitespace and stuff after a ;
# you need to include the directory where your datasets will be downloaded
# to, in our case /usr/local/data/datasets:
DirsAllowed = ., /usr/share/virtuoso/vad, /usr/local/data/datasets
# IMPORTANT: for performance also do this
# the following two are as suggested by comments in the original .ini
# file in order to use the RAM on your server:
NumberOfBuffers = 2720000
MaxDirtyBuffers = 2000000
# each buffer caches a 8K page of data and occupies approx. 8700 bytes of
# memory. It's suggested to set this value to 65 % of ram for a db only server
# so if you have 32 GB of ram: 32*1000^3*0.65/8700 = 2390804
# default is 2000 which will use 16 MB ram ;)
# Make sure to remove whitespace if you uncomment existing lines!
MaxCheckpointRemap = 625000
# set this to 1/4th of NumberOfBuffers
# I like to increase the ResultSetMaxrows, MaxQueryCostEstimationTime
# and MaxQueryExecutionTime drastically as it's a local store where we
# do quite complex queries... up to you (don't do this if a lot of people
# use it).
# In any case for the importer to be more robust add the following setting
# to this section:
ShortenLongURIs = 1

The next step installs an init-script (autostart) and starts the virtuoso server. (If you’ve changed directories to edit /var/lib/virtuoso/db/virtuoso.ini, go back to the virtuoso source dir!):

sudo cp debian/init.d /etc/init.d/virtuoso-opensource &&
sudo chmod a+x /etc/init.d/virtuoso-opensource &&
sudo bash debian/virtuoso-opensource.postinst.debhelper

You should now have a running virtuoso server.

DBpedia URIs (en) vs. DBpedia IRIs (i18n)

The DBpedia 2014 consists of several datasets: one “standard” English version and several localized versions for other languages (i18n). The standard version mints URIs by going through all English Wikipedia articles. For all of these the Wikipedia cross-language links are used to extract corresponding labels in other languages for the en URIs (e.g., de/labels_en_uris_de.nt.bz2). This is problematic as for example articles which are only in the German Wikipedia won’t be extracted. To solve this problem the i18n versions exists and create IRIs in the form of for every article in the German Wikipedia (e.g., de/labels_de.nt.bz2).

This approach has several implications. For backwards compatibility reasons the standard DBpedia makes statements about URIs such as while the local chapters, like the German one, make statements about IRIs such asöder (note the ö). In other words and as written above: the standard DBpedia uses URIs to identify things, while the localized versions use IRIs. This also means thatöder shouldn’t work. That said, clicking the link will actually work as there is magic going on in your browser to give you what you probably meant. Using curl curl -i -L -H "Accept: application/rdf+xml"öder or SPARQLing the endpoint will nevertheless not be so nice/sloppy and can cause quite some headache: select * where { dbpedia:Gerhard_Schröder ?p ?o. } vs. select * where { <> ?p ?o. }. In order to mitigate this historic problem a bit DBpedia actually offers owl:sameAs links from IRIs to URIs: en/iri_same_as_uri_en which you should load, so you at least have a link to what you want if someone tries to get info about an IRI.

As the standard DBpedia provides labels, abstracts and a couple other things in several languages, there are two types of files in the localized DBpedia folders: There are triples directly associating the English URIs with for example the German labels (de/labels_en_uris_de) and there are the localized triple files which associate for example the DE IRIs with the German labels (de/labels_de).

Downloading the DBpedia dump files & Repacking

For our group we decided that we wanted a reasonably complete mirror of the standard DBpedia (EN) (have a look at datasets loaded into the public DBpedia SPARQL Endpoint), but also the i18n versions for the German DBpedia loaded in separate graphs, as well as each of their pagelink datasets in another separate graph. For this we download the corresponding files in (NT) format as follows. If you need something different do so (and maybe report back if there were problems and how you solved them).

Another hint: Virtuoso can only import plain (uncompressed) or gzipped files, the DBpedia dumps are bzipped, so you either repack them into gzip format or extract them. On our server the importing procedure was reasonably slower from extracted files than from gzipped ones (ignoring the vast amount of wasted disk space for the extracted files). File access becomes a bottleneck if you have a couple of cores idling. This is why I decided on repacking all the files from bz2 to gz. As you can see I do the repacking per folder in parallel, if that’s not suitable for you, feel free to change it. You might also want to change this if you want to do it in parallel to downloading. The repackaging process below took about 1 hour but was worth it in the end. The more CPUs you have, the more you can parallelize this process.

# see comment above, you could also get the all_language.tar or another DBpedia version...
mkdir -p /usr/local/data/datasets/dbpedia/2014
cd /usr/local/data/datasets/dbpedia/2014
wget -r -nc -nH --cut-dirs=1 -np -l1 -A '*.nt.bz2' -A '*.owl' -R '*unredirected*'{en/,de/,links/,dbpedia_2014.owl}

# if you want to save space do this:
for d in */ ; do for i in "${d%/}"/*.bz2 ; do bzcat "$i" | gzip > "${i%.bz2}.gz" && rm "$i" ; done & done
# else do:
#bunzip2 */*.bz2 &

# notice that the extraction (and repacking) of *.bz2 takes quite a while (about 1 hour)
# gzipped data is reasonably packed, but still very fast to access (in contrast to bz2), so maybe this is the best choice.

Data Cleaning and The bulk loader scripts

In contrast to the previous versions of this article the virtuoso import will take care of shortening too long IRIs itself. Also it seems the bulk loader script is included in the more recent Virtuoso versions, so as a reference only: see the old version for the cleaning script and VirtBulkRDFLoaderExampleDbpedia and
for info about the bulk loader scripts.

Importing DBpedia dumps into virtuoso

Now AFTER the re-/unpacking of the DBpedia dumps we will register all files in the dbpedia dir (recursively ld_dir_all) to be added to the dbpedia graph. If you use this method make sure that only files reside in the given subtree that you really want to import.
Also don’t forget to import the dbpedia_2014.owl file (first step in the script below)!
If you only want one directory’s files to be added (non recursive) use ld_dir('dir', '*.*', 'graph');.
If you manually want to add some files, use ld_add('file', 'graph');.
See the VirtBulkRDFLoaderScript file for details.

Be warned that it might be a bad idea to import the normal and i18n dataset into one graph if you didn’t select specific languages, as it might introduce a lot of duplicates.

In order to keep track (and easily reproduce) what was selected and imported into which graph, I actually link (ln -s) the repacked files into a directory structure beneath /usr/local/data/datasets/dbpedia/2014/importedGraphs/ and import from there instead. To make sure you think about this, I use that path below, so it won’t work if you didn’t pay attention. If you really want to import all downloaded files, just import /usr/local/data/datasets/dbpedia/2014/.

Also be aware of the fact that if you load certain parts of dumps in different graphs (such as I did with the pagelinks, as well as the i18n versions of the DE and FR datasets) that only triples from the graph will be shown when you visit the local pages with your browser (SPARQL is unaffected by this)!

So if you want to load the same datasets as loaded on the official endpoint (but restricted to the EN and DE ones ) the following should do the trick to link them up for the next steps:

cd /usr/local/data/datasets/dbpedia/2014/
mkdir importedGraphs
cd importedGraphs

# ln -s ../../dbpedia_2014.owl ./ # see below!
ln -s ../../links/* ./

ln -s ../../en/article_categories_en.nt.gz ./
ln -s ../../en/category_labels_en.nt.gz ./
ln -s ../../en/disambiguations_en.nt.gz ./
ln -s ../../en/external_links_en.nt.gz ./
ln -s ../../en/freebase_links_en.nt.gz ./
ln -s ../../en/geo_coordinates_en.nt.gz ./
ln -s ../../en/geonames_links_en_en.nt.gz ./
ln -s ../../en/homepages_en.nt.gz ./
ln -s ../../en/images_en.nt.gz ./
ln -s ../../en/infobox_properties_en.nt.gz ./
ln -s ../../en/infobox_property_definitions_en.nt.gz ./
ln -s ../../en/instance_types_en.nt.gz ./
ln -s ../../en/instance_types_heuristic_en.nt.gz ./
ln -s ../../en/interlanguage_links_chapters_en.nt.gz ./
ln -s ../../en/iri_same_as_uri_en.nt.gz ./
ln -s ../../en/labels_en.nt.gz ./
ln -s ../../en/long_abstracts_en.nt.gz ./
ln -s ../../en/mappingbased_properties_cleaned_en.nt.gz ./
ln -s ../../en/page_ids_en.nt.gz ./
ln -s ../../en/persondata_en.nt.gz ./
ln -s ../../en/redirects_transitive_en.nt.gz ./
ln -s ../../en/revision_ids_en.nt.gz ./
ln -s ../../en/revision_uris_en.nt.gz ./
ln -s ../../en/short_abstracts_en.nt.gz ./
ln -s ../../en/skos_categories_en.nt.gz ./
ln -s ../../en/specific_mappingbased_properties_en.nt.gz ./
ln -s ../../en/wikipedia_links_en.nt.gz ./

ln -s ../../de/labels_en_uris_de.nt.gz ./
ln -s ../../de/long_abstracts_en_uris_de.nt.gz ./
ln -s ../../de/short_abstracts_en_uris_de.nt.gz ./

ln -s ../../fr/labels_en_uris_fr.nt.gz ./
ln -s ../../fr/long_abstracts_en_uris_fr.nt.gz ./
ln -s ../../fr/short_abstracts_en_uris_fr.nt.gz ./
cd ..

ln -s ../../en/genders_en.nt.gz ./
ln -s ../../en/out_degree_en.nt.gz ./
ln -s ../../en/page_length_en.nt.gz ./
cd ..

ln -s ../../en/page_links_en.nt.gz ./
cd ..

ln -s ../../en/topical_concepts_en.nt.gz ./
cd ..

ln -s ../../de/article_categories_de.nt.gz ./
ln -s ../../de/category_labels_de.nt.gz ./
ln -s ../../de/disambiguations_de.nt.gz ./
ln -s ../../de/external_links_de.nt.gz ./
ln -s ../../de/freebase_links_de.nt.gz ./
ln -s ../../de/geo_coordinates_de.nt.gz ./
ln -s ../../de/homepages_de.nt.gz ./
ln -s ../../de/images_de.nt.gz ./
ln -s ../../de/infobox_properties_de.nt.gz ./
ln -s ../../de/infobox_property_definitions_de.nt.gz ./
ln -s ../../de/instance_types_de.nt.gz ./
ln -s ../../de/interlanguage_links_chapters_de.nt.gz ./
ln -s ../../de/iri_same_as_uri_de.nt.gz ./
ln -s ../../de/labels_de.nt.gz ./
ln -s ../../de/long_abstracts_de.nt.gz ./
ln -s ../../de/mappingbased_properties_de.nt.gz ./
ln -s ../../de/out_degree_de.nt.gz ./
ln -s ../../de/page_ids_de.nt.gz ./
ln -s ../../de/page_length_de.nt.gz ./
ln -s ../../de/persondata_de.nt.gz ./
ln -s ../../de/pnd_de.nt.gz ./
ln -s ../../de/redirects_transitive_de.nt.gz ./
ln -s ../../de/revision_ids_de.nt.gz ./
ln -s ../../de/revision_uris_de.nt.gz ./
ln -s ../../de/short_abstracts_de.nt.gz ./
ln -s ../../de/skos_categories_de.nt.gz ./
ln -s ../../de/specific_mappingbased_properties_de.nt.gz ./
ln -s ../../de/wikipedia_links_de.nt.gz ./
cd ..

ln -s ../../de/page_links_de.nt.gz ./
cd ..

This should have prepared your importedGraphs directory. From this directory you can run the following command which print out the necessary isql commands to register your graphs for importing:

for g in * ; do echo "ld_dir_all('$(pwd)/$g', '*.*', 'http://$g');" ; done

One more thing (thanks to Romain): In order for the DBpedia.vad package (which is installed at the end) to work correctly, the dbpedia_2014.owl file needs to be imported into graph

Note: In the following i will assume that your virtuoso isql command is called isql. If you’re in lack of such a command it might be called isql-vt, but this usually means you installed it using some other method than described in here

isql # enter virtuoso sql mode
-- we are in sql mode now
ld_add('/usr/local/data/datasets/remote/dbpedia/2014/dbpedia_2014.owl', '');
ld_dir_all('/usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/', '*.*', '');
ld_dir_all('/usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/', '*.*', '');
ld_dir_all('/usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/', '*.*', '');
ld_dir_all('/usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/', '*.*', '');
ld_dir_all('/usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/', '*.*', '');
ld_dir_all('/usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/', '*.*', '');

-- do the following to see which files were registered to be added:
-- if unsatisfied use:
-- delete from DB.DBA.LOAD_LIST;

You can now also register other datasets like Freebase, DBLP, Yago, Umbel and … that you want to be loaded. Our full DB.DBA.LOAD_LIST currently looks like this:

SELECT ll_graph, ll_file FROM DB.DBA.LOAD_LIST;
ll_graph                             ll_file
VARCHAR                              VARCHAR NOT NULL
____________________________________                   /usr/local/data/datasets/remote/dblp/l3s/2014-11-08/dblp.nt.gz /usr/local/data/datasets/remote/dbpedia/2014/dbpedia_2014.owl                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/                /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/               /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/               /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/               /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/         /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/      /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/   /usr/local/data/datasets/remote/dbpedia/2014/importedGraphs/              /usr/local/data/datasets/remote/freebase/2014-11-02/freebase-rdf-2014-11-02-00-00.gz                    /usr/local/data/datasets/remote/           /usr/local/data/datasets/remote/umbel/External Ontologies/dbpediaOntology.n3           /usr/local/data/datasets/remote/umbel/External Ontologies/               /usr/local/data/datasets/remote/umbel/Ontology/umbel.n3           /usr/local/data/datasets/remote/umbel/Reference Structure/umbel_reference_concepts.n3  /usr/local/data/datasets/remote/yago/yago2/2012-12/yagoLabels.ttl.gz

114 Rows. -- 5 msec.

OK, now comes the fun (and long part: about 1.5 hours (new virtuoso 7 is cool ;) for DBpedia alone, +~3 hours for Freebase)… After we registered the files to be added, now let’s finally start the process. Fire up screen if you didn’t already. (For more detailed metering than below see VirtTipsAndTricksGuideLDMeterUtility.)

sudo aptitude install screen
screen isql
-- depending on the amount of CPUs and your IO performance you can run
-- more rdf_loader_run(); commands in other isql sessions which will
-- speed up the import process.
-- you can watch the progress from another isql session with:
-- select * from DB.DBA.LOAD_LIST;
-- if you need to stop the loading for any reason: rdf_load_stop ();
-- if you want to force stopping: rdf_load_stop(1);
commit WORK;

After this:
Take a look into var/lib/virtuoso/db/virtuoso.log file. Should you find any errors in there… FIX THEM! You might use the dump, but it’s incomplete then. Any error quits out of the loading of the corresponding file and continues with the next one, so you’re only using the part of that file up to the place where the error occurred. (Should you find errors you can’t fix please leave a comment.)

Final polishing

You can & should now install the DBpedia and RDF Mappers packages from the Virtuoso Conductor.

login: dba
pw: dba

Go to System Admin / Packages. Install the dbpedia (v. 1.4.28) and rdf_mappers (v. 1.34.74) packages (takes about 5 minutes).

Testing your local mirror

Go to the sparql-endpoint of your server http://your-server:8890/sparql (or in isql prefix with: SPARQL)

sparql SELECT COUNT(*) WHERE { ?s ?p ?o } ;

This shouldn’t take long in Virtuoso 7 anymore and for me now returns 695,553,624 for DBpedia (en+de), 3,543,872,243 with DBpedia (en+de), Freebase, DBLP, Yago, Umbel and

I also like this query showing all the graphs and how many triples are in them:

sparql SELECT ?g COUNT(*) { GRAPH ?g {?s ?p ?o.} } GROUP BY ?g ORDER BY DESC 2;
g                                                           callret-1
LONG VARCHAR                                                LONG VARCHAR
____________________________________________________________                                     2760013365                                          375176108                                149707899                                       92508750                                          72519345                             55804533                                      21900162                          15372307                                   403452  256065                          149638                         27063                                           8727
http://localhost:8890/DAV/                                  6187                  2639                                      1702                                 1480                              1226                            937                                   857                 804                  741                696               691                        661
virtrdf-label                                               638                                  557                                    553                      482                444                386                            332                    311                         252                      225                     183                   172                              160                               160                    144                    143                            139                         117                   103            102                     90                       87                 85                      79               68                       41             32                   26                     23                     21        21
http://localhost:8890/sparql                                14                   12
dbprdf-label                                                6

59 ROWS. -- 61717 msec.

Congratulations, you just imported over half a billion triples (or over 3.5 G triples).

Backing up this initial state

Now is a good moment to backup the whole db (takes about half an hour):

sudo -i
cd /
/etc/init.d/virtuoso-opensource stop &&
tar -cvf - /var/lib/virtuoso | lzop > virtuoso-7.1.0-DBDUMP-$(date '+%F')-dbpedia-2014-en_de.tar.lzop &&
/etc/init.d/virtuoso-opensource start

Afterwards you might want to repack this with xz (lzma) like this:

# aptitude install xz
for f in virtuoso-7.1.0-DBDUMP-*.tar.lzop ; do lzop -d -c "$f" | xz > "${f%lzop}.xz" ; done

Yay, done ;)
As always, feel free to leave comments if i made a mistake or to tell us about your problems or how happy you are :D.

Our database dump file

In case you really want exactly the same state of the public datasets that we have loaded (as described above) you can download our database dump (57 GB, md5sum, including: DBpedia 2014 en,de,links,dbpedia_2014.owl, Freebase, DBLP, Yago, Umbel and


Many thanks to the DBpedia team for their endless efforts of providing us all with a great dataset. Also many thanks to the Virtuoso crew for releasing an opensource version of their DB.


  • 2014-11-11: Added link to our Dump-File
  • 2014-11-24: Thanks to Romain: Load dbpedia_2014.owl into graph for DBpedia.vad to find it when resolving http://your-server:8890/ontology/author for example.

November 08, 2014

Weblog der Fachschaft Informatik

Informatik Praxistag 2014

Weblog der Fachschaft Informatik

Kleine Erinnerung:
Am 13.November 2014 findet der diesjährige Informatik-Praxistag statt.

Es finden 3 Podiumsdiskussionen statt:

  •  11:00 bis 11:45 Uhr: Softwareentwicklung in Deutschland – Wird in 20 Jahren am Standort Deutschland noch Software produziert?
  • 13:30 bis 14:15 Uhr: Mittelstand versus Großkonzern – Welcher Arbeitgeber passt zu mir?
  • 14:15 bis 15:00 Uhr: Bachelor, Master, Promotion – Wie starte ich eine erfolgreiche Karriere?

Außerdem stellen sich diverse Unternehmen vor.

Alle weiteren Informationen findet ihr hier beim FIT.

October 30, 2014

Weblog der Fachschaft Informatik

Diskussionsrunde zum Thema “Bargeldlose Zukunft”

Weblog der Fachschaft Informatik


Am Mittwoch, den 5.11.14, findet eine Fishbowl-Diskusion rund um das Thema “Scheinlos glücklich? Über Bitcoins, digitale Tauschgeschäfte und der bargeldlosen Zukunft” statt. Dort hab ihr die Möglichkeit euch über das Thema zu informieren und mit Experten ins Gespräch zu kommen. Weiter werden Studierende ihre Seminararbeiten zu diesem Thema als Poster präsentieren und für Fragen zur Verfügung stehen. Starten wird die Veranstaltung um 18 Uhr im BIC, die Diskussion findet dann um 19 Uhr statt.

October 29, 2014

On this page you’ll find a list of Content-types. If you want to know how to determine it programmatically, read the article Get the mimetype for a file type. File type MIME [...]

When receiving data from a server or sending data to it, the receiver should always get a hint of what type the data is. @media only [...]

October 28, 2014

Weblog der Fachschaft Informatik

2. FSR-Sitzung im Wintersemester 14

Weblog der Fachschaft Informatik

Auch hier die Erinnerung: Am Mittwoch, den 29.10. um 14:15 ist wieder FSR-Sitzung. Die Vorläufige Tagesordnung ist wie folgt:

  1. Festlegung der TO
  2. Protokolle
  3. Mitteilungen
  4. Crêpesstand am Informatik Praxistag
  5. Weihnachtsfeier und VV
  6. Handtücher GMF
  7. Verschiedenes

October 24, 2014

Weblog der Fachschaft Informatik

Die EWoche endet…

Weblog der Fachschaft Informatik

Die aktuelle EWoche geht langsam aber sicher zu Ende, am Montag beginnen wieder die Vorlesungen. Wir danken den zahlreichen Helfern, ohne die Events nicht möglich gewesen wären und wünschen allen Erstsemestern einen guten Start ins “echte” Studium.

Heute fand zum krönenden Abschluss unseres Info-Vorkurses ein Robocode-Turnier statt in dem sich unsere Erstsemester in 19 Teams in ihren Programmierkünsten messen konnten. Gratulation an die Sieger:

  1. Team “Fucking BUBBLES!!!” mit dem Roboter “Der Augenschmelzer”
  2. Team “The Destroyers of the Uni” mit dem Roboter “OLAF!!”
  3. Team “Schnitzel” mit dem Roboter “Penetrator”

Robocode Turnier Ewoche WS 2014

October 22, 2014

Romain Schmitz

Parse JSON in your shell

Romain Schmitz

jsawk curl -s\?Content-Type\=text/plain\;%20charset\=UTF-8\&Server\=httpbin | \ jsawk 'return this["Content-Length"]' jq jshon

October 19, 2014

Weblog der Fachschaft Informatik


Weblog der Fachschaft Informatik

Eines der Highlights jeder Ewoche seit meiner Immatrikulation ist die Glühweinwanderung, gerne abgekürzt GwwzHt. Der Suffix zHt steht hierbei für “zum Humbergturm”, ein Ziel, das laut Google-Maps nur 3,6 Kilometern oder 40 Minuten von der Uni entfernt ist.

Dieses Mal verlief der Hinweg auch fast perfekt. Von der Fachschaft ((1) im Bild) über den Bremerhof zu jener Weggabelung ((2) im Bild) ging nichts schief. Doch mit dem Weg forkten auch die Wandergruppen und der Glühweinwagen folgte dem blauen Weg und besuchte den Aschbacher Hof – zum ersten Mal seit mindestens fünf Jahren, wenn nicht seit immer. gluehwein-weg

Der andere Teil der Gruppe wählte den roten Weg zum Turm, erfreute sich an diesem und wanderte wieder zurück zur Uni oder direkt in die Innenstadt (der rote Weg weiter). Um das zeitlich einzuordnen, als dieser Teil sich schon wieder auf den Heimweg begab, kam der Glühweinwagen erst wieder an der Universität an. Für diejenigen, die mit dem Glühweinwagen gezogen sind und am Freitag morgen frisch sein mussten, war das auch das Ende der GwwzAH.

Aber für den Rest ging es nochmal los, den Turm zu erreichen. In Summa machte das dann eine sechsstündige Wanderung, die nochmal auf Yannicks GPS-Track anschaubar ist.

October 15, 2014

Jenkins is an extendable open source continuous integration (CI) server, developed by Sun Microsystems and also known as the former Hudson project. A CI toolchain helps you automate repetitive tasks [...]

October 13, 2014

Weblog der Fachschaft Informatik

Bilder von Scotland Yard 3.3

Weblog der Fachschaft Informatik

RedBull Station WS14/15

Die Bilder von den Scotland Yard 3.3 Teams sind da!
Klick mich! (Login erforderlich)

October 02, 2014

Weblog der Fachschaft Informatik

Einladung zur ersten FSR-Sitzung im Wintersemester

Weblog der Fachschaft Informatik

Die nächste FSR-Sitzung findet am 8.10. um 12:00 Uhr im Couchraum (48-467) statt. Die Vorläufige Tagesordnung ist:

  1. Festlegen der TO
  2. Protokolle
  3. Mitteilungen
  4. Anschaffungen
  5. KIF
  6. Vollversammlung
  7. Crêpstand zum ASTA-Sommerfest
  8. AJF & VLU
  9. Verschiedenes

Wie üblich sind alle herzlich eingeladen und dieses mal insbesondere jene, die zur KIF fahren wollen.

October 01, 2014

You cannot restore the original firmware via the web user interface. [see screenshot] Click System > Administration Enable SSH on the LAN ports [see screenshot] Set a password for the root user (or [...]

September 30, 2014

.note {display: none;} TODO: This article explains how to set up your PPPoE-based internet connection with a TP-LINK 1043ND. Replace the firmware Before flashing OpenWRT I upgraded the stock firmware. In this case [...]

September 22, 2014

Weblog der Fachschaft Informatik

KIF 42,5 in Karlsruhe

Weblog der Fachschaft Informatik

Die Konferenz der Informatikfachschaften (kurz KIF) lädt zur 42,5. KIF vom 12.-16.11. ein, dieses mal in Karlsruhe. Die Teilnahmekosten belaufen sich auf 30 Euro und dazu noch ungefähr 5 Euro Fahrtkosten (mit dem Studi-Ticket kann man den größten Weg schon zurück legen)
Der FSR und die FSK zusammen übernehmen diese Kosten und laden jeden Studierenden dazu ein an der KIF in Karlsruhe teilzunehmen. Wer das möchte darf sich gerne bei melden.

September 17, 2014

Last week we created a docker image with a Debian base installation which will serve us as the starting point in today’s article. In the next sections you will learn [...]

Wer macht mit?