Changes between Version 1 and Version 2 of Running imports


Ignore:
Timestamp:
Sep 4, 2013, 3:30:23 PM (6 years ago)
Author:
david.vanenckevort@…
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Running imports

    v1 v2  
    44The ETL scripts are based on the AbstractConceptImporter in the imports-common package.
    55
    6 == Updating the index ==
     6== Post-processing tasks ==
     7There are a few post-processing tasks defined in the [source:trunk/code/conceptwiki/util/fix-ups] project. There is a generic UtilRunner class that allows to select the tasks on the command-line. The complete list of options can be printed by:
     8{{{
     9 cd util/fix-ups
     10 mvn --quiet exec:java -Dexec.mainClass="nl.nbic.conceptwiki.fixups.UtilRunner" -Dexec.args="-h"
     11}}}
    712
    8 Currently we run the imports with the SOLR database disabled to improve the ETL performance. To update the index you should run the UtilRunner in the fix-up package with the -r command.
     13The UtilRunner takes configuration from fix-ups.properties in your home directory. This file is required to run the tasks.
     14
     15The following properties must be defined in this file.
    916
    1017{{{
    11  cd util/fix-ups
    12  mvn exec:java -Dexec.mainClass="nl.nbic.conceptwiki.fixups.UtilRunner" -Dexec.args="-r"
     18
    1319}}}
    1420
    15 This will use the settings as defined in service.properties in the service-impl package. Make sure that it is using the correct Neo4j datastore and SOLR instance. If you change the properties file you need to mvn install the service-impl again.
     21The UtilRunner supports the following operations:
     22
     23reindex:: Rebuild the SOLR index
     24linksets:: Generate linksets. Takes a target directory as a required argument
     25preflabels:: Generate the mapping between preferred terms and concepts. Takes a target filename as required argument
     26getconcept:: Query the graph for a specific concept
     27
     28After an import the reindex, linksets and preflabels tasks need to be run, but not in a particular order.
     29
     30The output of the linksets and preflabels commands needs to be zipped and uploaded to [http://downloads.nbiceng.net/linksets/]
     31