wiki:Installation Guide

Installation Guide

Architecture

ConceptWiki has been implemented as a set of modules. Together the modules of the ConceptWiki project are compiled into a Java Web Archive that exposes the APIs for the external developers. ConceptWiki is faceless, a web application to give a interface to explore and curate ConceptWiki is available in the https://trac.nbic.nl/conceptrewrite ConceptRewrite project.

  • Storage service: The storage layer has the responsibility to create, retrieve, update and delete the domain objects (concepts, labels, definitions, etc). The storage layer runs consistency checks to determine if the UUID is already present. It contains the edit history of each domain object in a relational database. Currently it uses the following external components:
    • Neo4j: Neo4j graph database, version 1.7, was downloaded from www.neo4j.org available under license GPL v3.0. Neo4j is a NOSQL graph database that interacts with the storage layer via Java.
    • MySQL: The MySQL database stores the edit history.
    • SOLR: The SOLR DB contains the index of all concepts.
  • Concept Service: The service layer is the primary interface for the web service and for the import scripts. It provides the validation of the incoming requests and executes checks on the format of the input data.
  • Web services: RMI exports.The Web services and RMI exports provide the interfaces for external access to the system. The RMI exports provide a more low level access for batch processes (e.g. data imports), while the web services encapsulate the service implementation to provide a higher level API suitable for easy programmatic read access to the concept store.

To build ConceptWiki from source Maven 3 is required. Maven will download all the managed dependencies and build the ConceptWiki war file to be deployed.

Getting and building the code

To build ConceptWiki you can check out the code from subversion:

  svn co https://trac.nbic.nl/svn/conceptwiki/trunk/code/conceptwiki/ conceptwiki

Building ConceptWiki is done with Maven:

  cd conceptwiki
  mvn clean package

If everything builds without errors the Java Web Archive ./web/web-ws/target/web-ws-0.0.1-SNAPSHOT.war (where 0.0.1-SNAPSHOT is the version as defined in the pom.xml).

Deployment

Prerequisites

ConceptWiki is developed with the following external non-managed dependencies:

  • Java 1.7
  • SOLR 4.0
  • MySQL 5.5
  • Tomcat 7

SOLR configuration

We advise to run SOLR in its own non-priviledged system account. To deploy SOLR in Tomcat we advise to configure solr/home in tomcat/conf/Catalina/localhost/solr.xml, the configuration could look like the following:

<?xml version="1.0" encoding="UTF-8"?>
<Context path="/solr" docBase="/opt/solr/war/apache-solr-4.0.0.war">
  <Environment name="solr/home" type="java.lang.String" value="/opt/solr/home" override="true"/>
</Context>

Tomcat should be configured to have at least 2GB heap space, you can set the JAVA_OPTS environment variable by creating a tomcat/bin/setenv.sh. This file could look like the following:

#!/bin/sh
JAVA_OPTS="-server -Xmx2048m"

The ConceptWiki index should be created in the directory specified by solr/home, e.g. /opt/solr/home.

The location of SOLR is specified in the ConceptWiki configuration with the property solr.url. This property should point to the fully qualified URL of the SOLR instance.

MySQL configuration

ConceptWiki requires a relational database to store the ancillary data (users, branches, languages) and the history. For MySQL schema files are included to create the database schema. To allow ConceptWiki to manage the schema a user with sufficient privileges is required. For MySQL the following should suffice to set up the database.

CREATE DATABASE conceptwiki;
GRANT ALL ON conceptwiki.* TO 'conceptwiki'@'localhost' IDENTIFIED BY 'password';
FLUSH PRIVILEGES;

In the GRANT line the hostname after the @ should be the fully qualified hostname of the server running the ConceptWiki web services. If multiple instances of the web services are running a pattern can be used for the hostname.

The database configuration is specified in the ConceptWiki configuration with the following properties:

# MySQL - history service
jdbc.driver=com.mysql.jdbc.Driver
jdbc.host=jdbc:mysql://localhost/conceptwiki
jdbc.user=conceptwiki
jdbc.pass=password
jdbc.initscript=classpath:/nl/nbic/conceptwiki/service/schema-mysql.sql
# MySQL - user service
jdbc.user.driver=com.mysql.jdbc.Driver
jdbc.user.host=jdbc:mysql://localhost/conceptwiki
jdbc.user.user=conceptwiki
jdbc.user.pass=password
jdbc.user.initscript=classpath:/schema-mysql.sql

Neo4j Graph configuration

The Neo4j Graph database is embedded in the web services. The data directory for the database should be on reliable, high performance storage. The location is specified in the ConceptWiki configuration with the following property:

# setup neo4j properties
neo4j.storeDir=workshop/neo4j_workshopdata

Configuring and deploying the web services

We advise to run the ConceptWiki web-services in its own non-priviledged system account. To deploy in Tomcat the WAR file should be placed in the webapps directory.

Tomcat should be configured to have at least 2GB heap space and 256MB PermGen space. It is also strongly advised to disable the Neo4j Usage Data Collector. You can set the JAVA_OPTS environment variable by creating a tomcat/bin/setenv.sh. This file could look like the following:

#!/bin/sh
JAVA_OPTS="-server -Xmx2048m -XX:MaxPermSize=256m -Dneo4j.ext.udc.disable=true -DCONCEPTWIKI_CONFIG_PATH=/opt/conceptwiki/config/conceptwiki.properties"
LANG=en_US.UTF-8

ConceptWiki is configured using Java properties. The location of the properties file can be specified in the environment variable CONCEPTWIKI_CONFIG_PATH. The following properties are available. Where possible sensible defaults are provided.

Property name Default value Description
jdbc.driver com.mysql.jdbc.Driver The class name of the JDBC driver to connect to the database
jdbc.host dbc:mysql://localhost/conceptwiki The JDBC connection string to connect to the database
jdbc.user conceptwiki The name of the database user used to access the database
jdbc.pass The password used to access the database
jdbc.initscript classpath:/nl/nbic/conceptwiki/service/schema-mysql.sql The script used to create the database schema
jdbc.user.driver com.mysql.jdbc.Driver The class name of the JDBC driver to connect to the database
jdbc.user.host dbc:mysql://localhost/conceptwiki The JDBC connection string to connect to the database
jdbc.user.user conceptwiki The name of the database user used to access the database
jdbc.user.pass The password used to access the database
jdbc.user.initscript classpath:/schema-mysql.sql The script used to create the database schema for the user table
neo4j.storeDir workshop/neo4j_workshopdata Path to the Graph database persistent storage directory
solr.url http://localhost:10080/solr The URL to connect to the SOLR index
void.location /opt/conceptwiki/void Location of the VOID descriptor files describing that data in ConceptWiki

Configure logging

A default installation will log at the debugging level and use a synchronous logger. This has a major impact on performance and for an acceptance or production server the logging should be configured at a coarser level (WARN or ERROR). ConceptWiki? web services use the logback system for logging. There are two different options for configuring the logging.

  • Change the tomcat log configuration
  • Add a logback.xml to the class path (e.g. put it in tomcat's lib directory)

logback.xml could look like the following:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <!-- daily rollover -->
            <fileNamePattern>${user.home}/tomcat/logs/conceptwiki.%d{yyyy-MM-dd}.log</fileNamePattern>

            <!-- keep 30 days' worth of history -->
            <maxHistory>30</maxHistory>
        </rollingPolicy>
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>
        
    <logger name="nl.nbic.conceptwiki" level="WARN"/>
    <root level="ERROR">
        <appender-ref ref="FILE"/>
    </root>
</configuration>

Stable URL schema for ConceptWiki in production

ConceptWiki in production must provide stable URLs. The conventions are detailed in the versioning policy. For the deployment this imposes a few requirements. The following URL patterns must be supported.

URL pattern Description
/3/ Version 3 named URL namespace.
/ Stable version URL namespace.

In the stable namespace a few rewrites must be implemented to point the stable version URL to the specific version that represents the stable release at the moment.

URL pattern Rewrite Proxy / Redirect
/concept/ /concept/index/ Redirect
/concept/index/ /3/concept/index Proxy
/web-ws/ /3/web-ws/ Proxy
Last modified 5 years ago Last modified on Jan 21, 2014, 5:04:43 PM