trafodion-development team mailing list archive

["Birdsall, Dave" <dave.birdsall@xxxxxx>]

Hi Matt,

This sounds promising. But what I see in the examples given seem to be environmental settings for a particular installation?

Which of these files would be the appropriate place to put something like "this code needs the TLOG turned on by default"? That is, a property of the code itself, apart from the environment where it is being installed.

Thanks,

Dave

From: Fredriksen, Lars
Sent: Friday, August 15, 2014 2:35 PM
To: Brown, Matt (Trafodion); Birdsall, Dave; Trafodion-Development@xxxxxxxxxxxxxxxxxxx
Cc: trafodion-general@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Trafodion-development] A scheme to let installer fixes work for older core builds

Hi Matt,
I would not be a opposed to adopting this...

Lars
On 8/15/2014 4:14 PM, Brown, Matt (Trafodion) wrote:
Just my $.02

Rather than invent something I'd really like to see Traf adopt the Apache Hadoop/HBase model for configuration. DCS uses this already. For example it has a /conf directory that contains a dcs-env.sh and dcs-site.xml. the env file exports the necessary env variables while the site.xml file can be used to override the default properties. It also allows for alternate configurations i.e., you can specify which config directory to use.

Contents for both of these are shown below. As Traf SQL dev has deep knowledge of JNI I don't think it would be difficult to invoke the Java methods that read/write these files. Moreover Hadoop/HBase users are used to seeing these files and understand how to modify them. No reason the installer couldn't use these too.

/conf/dcs-site.xml
<!--
/**
* Copyright 2010 The Apache Software Foundation
*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements.  See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership.  The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License.  You may obtain a copy of the License at
*
*     http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-->
<configuration>
      <property>
    <name>dcs.master.port</name>
    <value>24400</value>
    <description>
        Default DCS port.
    </description>
  </property>
  <property>
    <name>dcs.master.info.port</name>
    <value>40040</value>
    <description>The port for the Dcs Master web UI.
    Set to -1 if you do not want a UI instance run.
    </description>
  </property>
  <property>
    <name>dcs.server.info.port</name>
    <value>50030</value>
    <description>The port for the DcsServer web UI
    Set to -1 if you do not want the server UI to run.
    </description>
  </property>
  <property>
    <name>dcs.server.info.port.auto</name>
    <value>true</value>
    <description>Whether or not the DcsServer
    UI should search for a port to bind to. Enables automatic port
    search if dcs.server.info.port.auto is already in use.
    Useful for testing, turned off by default.
    </description>
  </property>
  <property>
    <name>dcs.zookeeper.property.clientPort</name>
    <value>2182</value>
    <description>Property from ZooKeeper's config zoo.cfg.
    The port at which the clients will connect.
    </description>
  </property>

/conf/dcs-env.sh
#/**
# *(C) Copyright 2013 Hewlett-Packard Development Company, L.P.
# *
# * Licensed under the Apache License, Version 2.0 (the "License");
# * you may not use this file except in compliance with the License.
# * You may obtain a copy of the License at
# *
# *     http://www.apache.org/licenses/LICENSE-2.0
# *
# * Unless required by applicable law or agreed to in writing, software
# * distributed under the License is distributed on an "AS IS" BASIS,
# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# * See the License for the specific language governing permissions and
# * limitations under the License.
# */
#
#/**
# * Copyright 2007 The Apache Software Foundation
# *
# * Licensed to the Apache Software Foundation (ASF) under one
# * or more contributor license agreements.  See the NOTICE file
# * distributed with this work for additional information
# * regarding copyright ownership.  The ASF licenses this file
# * to you under the Apache License, Version 2.0 (the
# * "License"); you may not use this file except in compliance
# * with the License.  You may obtain a copy of the License at
# *
# *     http://www.apache.org/licenses/LICENSE-2.0
# *
# * Unless required by applicable law or agreed to in writing, software
# * distributed under the License is distributed on an "AS IS" BASIS,
# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# * See the License for the specific language governing permissions and
# * limitations under the License.
# */

# Set environment variables here.

# This script sets variables multiple times over the course of starting an dcs process,
# so try to keep things idempotent unless you want to take an even deeper look
# into the startup scripts (bin/dcs, etc.)

# The java implementation to use.  Java 1.6 required.
# export JAVA_HOME=/usr/java/jdk1.6.0/

# Extra Java CLASSPATH elements.  Optional.
# export DCS_CLASSPATH=

# The maximum amount of heap to use, in MB. Default is 128.
# export DCS_HEAPSIZE=128

# Extra Java runtime options.
# Below are what we set by default.  May only work with SUN JVM.
# For more on why as well as other possible settings,
# see http://wiki.apache.org/hadoop/PerformanceTuning
export DCS_OPTS="-XX:+UseConcMarkSweepGC"

# Uncomment below to enable java garbage collection logging for the server-side processes
# this enables basic gc logging for the server processes to the .out file
# export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps $DCS_GC_OPTS"

# this enables gc logging using automatic GC log rolling. Only applies to jdk 1.6.0_34+ and 1.7.0_2+. Either use this set of options or the one above
# export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=512M $DCS_GC_OPTS"

# Uncomment below to enable java garbage collection logging for the client processes in the .out file.
# export CLIENT_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps $DCS_GC_OPTS"

# Uncomment below (along with above GC logging) to put GC information in its own logfile (will set DCS_GC_OPTS).
# This applies to both the server and client GC options above
# export DCS_USE_GC_LOGFILE=true

# Uncomment below if you intend to use the EXPERIMENTAL off heap cache.
# export DCS_OPTS="$DCS_OPTS -XX:MaxDirectMemorySize="
# Set dcs.offheapcache.percentage in dcs-site.xml to a nonzero value.

# Uncomment and adjust to enable JMX exporting
# See jmxremote.password and jmxremote.access in $JRE_HOME/lib/management to configure remote password access.
# More details at: http://java.sun.com/javase/6/docs/technotes/guides/management/agent.html
# export DCS_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"
# export DCS_MASTER_OPTS="$DCS_MASTER_OPTS $DCS_JMX_BASE -Dcom.sun.management.jmxremote.port=10101"
# export DCS_SERVER_OPTS="$DCS_SERVER_OPTS $DCS_JMX_BASE -Dcom.sun.management.jmxremote.port=10102"
# export DCS_REST_OPTS="$DCS_REST_OPTS $DCS_JMX_BASE -Dcom.sun.management.jmxremote.port=10103"
# export DCS_ZOOKEEPER_OPTS="$DCS_ZOOKEEPER_OPTS $DCS_JMX_BASE -Dcom.sun.management.jmxremote.port=10104"

# File naming hosts on which DCS Servers will run.  $DCS_HOME/conf/servers by default.
# export DCS_SERVERS=${DCS_HOME}/conf/servers

# Extra ssh options.  Empty by default.
# export DCS_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=DCS_CONF_DIR"

# Where log files are stored.  $DCS_HOME/logs by default.
# export DCS_LOG_DIR=${DCS_HOME}/logs

# Enable remote JDWP debugging of major dcs processes. Meant for Core Developers
# export DCS_MASTER_OPTS="$DCS_MASTER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070"
# export DCS_SERVER_OPTS="$DCS_SERVER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071"
# export DCS_RESET_OPTS="$DCS_RESET_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8072"
# export DCS_ZOOKEEPER_OPTS="$DCS_ZOOKEEPER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073"

# A string representing this instance of dcs. $USER by default.
# export DCS_IDENT_STRING=$USER

# The scheduling priority for daemon processes.  See 'man nice'.
# export DCS_NICENESS=10

# The directory where pid files are stored. $DCS_HOME/tmp by default.
# export DCS_PID_DIR=/var/dcs/pids

# Tell DCS whether it should manage it's own instance of Zookeeper or not.
# export DCS_MANAGES_ZK=true

# Tell DCS where the user program environment lives.
export DCS_USER_PROGRAM_HOME=$MY_SQROOT

From: Trafodion-development [mailto:trafodion-development-bounces+mattbrown=hp.com@xxxxxxxxxxxxxxxxxxx]On Behalf Of Fredriksen, Lars
Sent: Friday, August 15, 2014 1:33 PM
To: Birdsall, Dave; Trafodion-Development@xxxxxxxxxxxxxxxxxxx<mailto:Trafodion-Development@xxxxxxxxxxxxxxxxxxx>
Cc: trafodion-general@xxxxxxxxxxxxxxxxxxx<mailto:trafodion-general@xxxxxxxxxxxxxxxxxxx>
Subject: Re: [Trafodion-development] A scheme to let installer fixes work for older core builds

Hi Dave,
The only thing I would add, is that the install team has talked a little bit about externalizing some of the "knowledge" that is currently hard coded into some of the scripts. Things like ports, rpm dependencies etc, and it may make sense to use a common repository file for all of this. If we do so, we may have to go to an yaml or xml format so that we might be able to convey more information in a standard way - anyway, I think this proposal is a great start, and we can just expand as needed..

Lars
On 8/15/2014 12:11 PM, Birdsall, Dave wrote:

Hi Trafodion folk,



Recently we've encountered the following situation:



Some folks have downloaded the 0.8.3 versions of Trafodion together with the 0.8.3 versions of the installer and have run into certain installer bugs. The bugs have been fixed in newer versions of the installer but those versions today can't always run with the 0.8.3 versions of Trafodion core. So, users who try 0.8.3 still run into the bugs, and installer developers answer the same questions as before.



We could back-fit fixes into the 0.8.3 installer, but we've come up with a better idea. We'll describe that here:



We will add a new file, "installInfo.txt" somewhere within the Trafodion core directory tree (the precise location is TBD; the install team will make a proposal). The purpose of this file is to tell install what features to expect in this version of core. For example, recently a change to core was checked in to turn on TLOG by default. The installer required a corresponding change. By noting this feature in this file, the installer can figure out whether TLOG should be turned on by default or not, and so will work with both newer and older versions of core.



A new entry will be required in this file whenever a change goes into core that requires a corresponding change in the installer. So, in the future, when a core developer asks an install developer to make a corresponding change, the install developer will ask what the identifier is for this change in the InstallInfo.txt file.



The format of "installInfo.txt" will be simple.



A line beginning with '#' is a comment and will be ignored.



A line consisting of a single identifier (that is, some string of letters and numbers, starting with a letter) will be considered a feature name.



The installer will raise a warning when it finds a feature name that it doesn't know about. That will clue you in that you are using an old installer that doesn't understand a new core.



If the file is missing, this is interpreted by the installer as the lack of any new features. That will allow new installers to work with 0.8.3 core, for example.



We are aware of only two changes so far in 0.8.4 that required corresponding installer changes. So, the initial version of the InstallInfo.txt file will contain just two identifiers, one for each.



Please add new identifiers to the end of the file. We imagine that if the file gets big, we'll "age out" the older identifiers from the beginning of the file.



You'll be seeing this change show up in core shortly; one of us will follow up with another e-mail when that happens.



By the way, if you have a better idea of how to solve this problem, feel free to respond.



Thanks,



Dave Birdsall, Marvin Anderson, Steve Varnau, Yves Cadieux













--
Lars Fredriksen
Hewlett-Packard Company
lars.fredriksen@xxxxxx<mailto:lars.fredriksen@xxxxxx>
Office +1 630-968-9904
Cell +1 630-215-7672
PC +1 512-319-3265
HP-BIS<http://h71028.www7.hp.com/enterprise/w1/en/software/business-intelligence-overview.html?jumpid=ex_r2858_w1/en/large/tsg/go_bi>

--
Lars Fredriksen
Hewlett-Packard Company
lars.fredriksen@xxxxxx<mailto:lars.fredriksen@xxxxxx>
Office +1 630-968-9904
Cell +1 630-215-7672
PC +1 512-319-3265
HP-BIS<http://h71028.www7.hp.com/enterprise/w1/en/software/business-intelligence-overview.html?jumpid=ex_r2858_w1/en/large/tsg/go_bi>