Wednesday, August 14, 2013

Building Your Microsoft Solution with Mono

Background:

The agent for the CloudStack Hyper-V plugin was written in C# using the Microsoft Visual Studio tool chain.

However, Apache CloudStack source should be able to be built using open source tools.

Problem:

How do you compile an ASP.NET MVC4 web app written in C# using an open source tool chain?

Solution:

The latest version of Mono include a tools called Xbuild, which can consume the .sln and .csproj files that Visual Studio generates for your solution.

However, you will have to make some updates to the way your project fetches NuGet dependencies, and the projects that get built.

Let's cover all the steps involved one at a time.

Install Mono

First, install the latest release of Mono 3.x.  This version introduces the support for C# 5.0 and ships with the ASP.NET WebStack (Release Notes)

Although a package for this version is not advertised on the Mono Downloads page, the
mono archives include a 3.0.10 Windows .msi

Alternatively, there is a 3.x Debian package available.  For this package, update your apt sources and use apt-get to install the mono-complete package, which contains the tool chain.  E.g.
sed -e "\$adeb http://debian.meebey.net/experimental/mono /" -i /etc/apt/sources.list
apt-get update
apt-get install mono-complete

Mono ships with an empty certificate store.  The store needs to be populated with common certificates in order for HTTPS to work.  Use the mozroots tool to do this.  E.g.
root@mgmtserver:~/github/cshv3/plugins/hypervisors/hyperv/DotNet# mozroots --import --sync --machine
Mozilla Roots Importer - version 3.0.6.0
Download and import trusted root certificates from Mozilla's MXR.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Downloading from 'http://mxr.mozilla.org/seamonkey/source/security/nss/lib/ckfw/builtins/certdata.txt?raw=1'...
Importing certificates into user store...
Import process completed.

NB: whether you add the certs to your user ( "mozroots --import --sync) or the machine (mozroots --import --sync --machine) depends on what user is used to run web requests.  On Debian 7.0, I found that the machine certificate store had to be updated.

Understand NuGet Packages

NuGet is a package manager for the .NET platform.  These packages consist of assemblies used at compile time and runtime.  The packages are stored in perpetuity on the NuGet website, and fetched by a similarly named command line tool.

Each VisualStudio project lists its NuGet dependencies in the packages.config file, which is in same folder as the project's .csproj file.  E.g.
Administrator@cc-svr10 ~/github/cshv3/plugins/hypervisors/hyperv/DotNet/ServerResource
$ cat ./HypervResource/packages.config
<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="AWSSDK" version="1.5.23.0" targetFramework="net45" />
  <package id="DotNetZip" version="1.9.1.8" targetFramework="net45" />
  <package id="log4net" version="2.0.0" targetFramework="net45" />
  <package id="Newtonsoft.Json" version="4.5.11" targetFramework="net45" />
</packages>

By default, your Visual Studio project will search for these assemblies in the packages directory in the folder containing the .sln file.  E.g.
$ cat ./HypervResource/HypervResource.csproj
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
...
  <ItemGroup>
    <Reference Include="AWSSDK">
      <HintPath>..\packages\AWSSDK.1.5.23.0\lib\AWSSDK.dll</HintPath>
    </Reference>
...
</Project>

Create a NuGet target

A NuGet target is a build script that downloads packages to the packages folder in advance of compile step of your build.  After this step, the compiler will be able to resolve the assembly dependencies.

Using a NuGet target to automate downloading is a better solution than adding assemblies to your source code assembly or creating a custom script for download (the .sln and .csproj files).  The NuGet target keeps package dependencies at arms length. NuGet carries the legal risk of redistributing other organisations' binaries.  Also, the NuGet target requires much less maintenance work.
Once setup, the NuGet target the list of files to download list the projects being compiled.

To create the target, update your .sln and .csproj files with the NuGet-related tags using Visual
Studio.  Simply open the .sln, and select Project -> Enable NuGet Package Restore.  This creates a .nuget folder with an msbuild task in the NuGet.targets file, its configuration in NuGet.Config, and the command line tool NuGet.exe.  E.g.
Administrator@cc-svr10 ~/github/cshv3/plugins/hypervisors/hyperv/DotNet/ServerResource
$ find ./ | grep NuGet
./.nuget/NuGet.Config
./.nuget/NuGet.exe
./.nuget/NuGet.targets

You will want to add the build script (NuGet.targets) and its configuration (NuGet.Config) to your source code repository.  NuGet.exe should be downloaded separately, as we will explain next.

Update the NuGet target to run with XBuild

Although Mono's Xbuild toll will execute NuGet.targets automatically, Xbuild does not support enough MSBuild tasks to allow it to executed the default NuGet.targets script.

First, you will have to create a script to download NuGet.exe, because Xbuild does not yet support the embedded Task tag (Source).  E.g. if wget is installed, you could use the following instructions:
wget http://nuget.org/nuget.exe
cp nuget.exe ./.nuget/NuGet.exe
chmod a+x ./.nuget/NuGet.exe
Note the change of case.  The code generated NuGet.target expects NuGet.exe

Secondly, if you plan to build on a Windows OS, you will have to update NuGet.targets
to bypass unsupported tags.  Normally, NuGet.targets uses the OS property to bypass unsupported tags automatically.  However, this only works when Mono is used on Linux.  E.g. in the example below an unsupported property function is avoided when the OS is not "Window_NT".
<PropertyGroup Condition=" '$(OS)' == 'Windows_NT'">
    <!-- Windows specific commands -->
    <NuGetToolsPath>$([System.IO.Path]::Combine($(SolutionDir), ".nuget"))</NuGetToolsPath>
    <PackagesConfig>$([System.IO.Path]::Combine($(ProjectDir), "packages.config"))</PackagesConfig>
</PropertyGroup>

<PropertyGroup Condition=" '$(OS)' != 'Windows_NT'">
    <!-- We need to launch nuget.exe with the mono command if we're not on windows -->
    <NuGetToolsPath>$(SolutionDir).nuget</NuGetToolsPath>
    <PackagesConfig>packages.config</PackagesConfig>
</PropertyGroup>

To be able to build on Windows, you can either delete the Windows version or revise the conditionals to use a custom property to determine if xbuild is being used.  E.g. in the example below, we
bypass the unsupported function property using a test for the property BuildWithMono.
<PropertyGroup Condition=" '$(OS)' == 'Windows_NT'">
    <!-- Windows specific commands -->
    <!-- <NuGetToolsPath>$([System.IO.Path]::Combine($(SolutionDir), ".nuget"))</NuGetToolsPath> -->
    <NuGetToolsPath>$(SolutionDir).nuget</NuGetToolsPath>
    <!--<PackagesConfig>$([System.IO.Path]::Combine($(ProjectDir), "packages.config"))</PackagesConfig> -->
    <PackagesConfig>packages.config</PackagesConfig>
</PropertyGroup>

<PropertyGroup Condition=" '$(OS)' != 'Windows_NT'">
    <!-- We need to launch nuget.exe with the mono command if we're not on windows -->
    <NuGetToolsPath>$(SolutionDir).nuget</NuGetToolsPath>
    <PackagesConfig>packages.config</PackagesConfig>
</PropertyGroup>

To trigger the bypass, we set buildWithMono when calling Xbuild.  E.g.
xbuild /p:BuildWithMono="true" ServerResource.sln

What code needs bypassing?

  1. Mono Xbuild cannot interpret property functions.  These are properties whose values are results of executing an inline function call. E.g.
  2. <!-- Windows specific commands -->
    <NuGetToolsPath>$([System.IO.Path]::Combine($(SolutionDir), ".nuget"))</NuGetToolsPath>
    <PackagesConfig>$([System.IO.Path]::Combine($(ProjectDir), "packages.config"))</PackagesConfig>
  3. Mono does not implement all tasks precisely.  E.g. the Exec tag is missing the LogStandardErrorAsError property available with .NET.  Thus,
    <Exec Command="$(RestoreCommand)"
          LogStandardErrorAsError="true"
          Condition="'$(OS)' == 'Windows_NT' And Exists('$(PackagesConfig)')" />
    causes
    Error executing task Exec: Task does not have property "LogStandardErrorAsError" defined
  4. Xbuild must call NuGet.exe through mono.  E.g.
  5. <NuGetCommand Condition=" '$(OS)' == 'Windows_NT'">"$(NuGetExePath)"</NuGetCommand>
    <NuGetCommand Condition=" '$(OS)' != 'Windows_NT' ">mono --runtime=v4.0.30319 $(NuGetExePath)</NuGetCommand>

Skip unsupported projects

Xbuild cannot compile projects that require additional proprietary assemblies not available through NuGet or the Mono implementation.  For example, unit tests created using Visual Studio Test Tools make use of Visual Studio-specific assemblies.   E.g.
xbuild ServerResource.sln
...
HypervResourceControllerTest.cs(18,17): error CS0234: The type or namespace name `VisualStudio' does not exist in the namespace `Microsoft'. Are you missing an assembly reference?
In this case, HypervResourceControllerTest.cs(18,17) is a reference to Visual Studio test tools:
using Microsoft.VisualStudio.TestTools.UnitTesting;
To create new configuration in you solution that skips compiling these projects follow these steps:
  1. In Visual Studio, create a new configuration that excludes the projects you're not interested in. -(Build -> Configuration Manager..., select on the Active solution platform: drop down list)
  2. Using the Configuration Manager, remove unwanted solutions from the configuration.
  3. Next, close VisualStudio, which will save changes to the .sln and .csproj The .sln will record which projects are associated with the configuration. The .csproj will record the settings of the configuration, such as whether TRACE or DEBUG is defined.
  4. Finally, when calling xbuild assign your configuration's name to the Configuration property.

E.g.
xbuild /p:Configuration="NoUnitTests" /p:BuildWithMono="true" ServerResource.sln
The above will build the projects associated with the NoUnitTests configuration. Source

Final Remarks:

Mono's support keeps getting better with every release.  The quirks discussed in this post may have been addressed by the time you read it.


Thursday, August 08, 2013

Using CloudStack's Log Files: XenServer Integration Troubleshooting

Background


CloudStack uses the XenServer plugin model to make extensions to XenAPI aka XAPI.  These extensions are written in a combination of Python and Bash scripts.  Python is the programming language for XenServer plugin model; however, calls to command line tools are more natural to write in a bash script.

These XenServer extensions and the management server plugins that use them generate logging information to assist developers and admins to diagnose problems.


Problem


What are the log files are useful, where are they, how do I use them?


Solution


The management server uses the log4j logging library, to generate three logs: 
  • the general log, aka FILE
  • the CloudStack API call log, aka APISERVER
  • the AWS API log, aka AWSAPI.  
The files each log writes to are identified in the <appender> section of the log4j config file. Specifically, the file name is in the ActiveFileName parameter.
XenServer logs sit on the Dom0 operating system in /var/log (source).  CloudStack's XAPI extensions write to /var/log/SMlog.  Although all XAPI events are summarised in /var/log/xensource.log, this log is not recommended.  The information available is limited and difficult to find due to the large number of events being logged.

In the following sub sections, we 
  • explain how to find the management server logs for a development CloudStack
  • explain how to find the management server logs for a production CloudStack
  • explain how the XenServer logs are generated
  • walk through an example of using the logs to diagnose a problem

Developer CloudStack Managment Server Logs


To find the exact log file locations, first find the log4j configuration file using grep.  

Developers that launch CloudPlatform in Jetty using should look under the ./client folder for the log4j configuration.  The ./client folder contains the files that execute when mvn -pl :cloud-client-ui jetty:run is executed.  E.g.
 root@mgmtserver:~/github/cshv3# find ./client/ | grep "log4j.*\.xml"  
 ./client/tomcatconf/log4j-cloud.xml.in  
 ./client/target/generated-webapp/WEB-INF/classes/log4j-cloud.xml  
 ./client/target/cloud-client-ui-4.2.0-SNAPSHOT/WEB-INF/classes/log4j-cloud.xml  
 ./client/target/conf/log4j-cloud.xml  
Next, examine the <appender> elements.in the files.  E.g.
 root@mgmtserver:~/github/cshv3# more ./client/target/generated-webapp/WEB-INF/classes/log4j-cloud.xml   
  ...   
   <!-- ================================= -->   
   <!-- Preserve messages in a local file -->   
   <!-- ================================= -->   
   <!-- A regular appender FIXME implement code that will close/reopen logs on SIGHUP by logrotate FIXME make the paths configurable using the build system -->   
   <appender name="FILE" class="org.apache.log4j.rolling.RollingFileAppender">   
   <param name="Append" value="true"/>   
   <param name="Threshold" value="TRACE"/>   
   <rollingPolicy class="org.apache.log4j.rolling.TimeBasedRollingPolicy">   
    <param name="FileNamePattern" value="vmops.log.%d{yyyy-MM-dd}.gz"/>   
    <param name="ActiveFileName" value="vmops.log"/>   
   </rollingPolicy>   
   <layout class="org.apache.log4j.EnhancedPatternLayout">   
    <param name="ConversionPattern" value="%d{ISO8601} %-5p [%c{1.}] (%t:%x) %m%n"/>   
   </layout>   
   </appender>   
In this case, vmops.log is the general log file.

Production CloudStack Management Server Logs


With production servers, the process is the same:  look for the log4j config.  

In the case of Citrix CloudPlatform 3.0.x, we would see the following:
  [root@cs1-mgr management]# find / | grep "log4j.*\.xml"   
  /etc/cloud/management/log4j-cloud.xml   
  /etc/cloud/server/log4j-cloud.xml   
Next, examine the <appender> elements.in the files.  E.g. 
 [root@cs1-mgr management]# more /etc/cloud/management/log4j-cloud.xml  
 <?xml version="1.0" encoding="UTF-8"?>  
 <!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">  
 <log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/" debug="false">  
   <throwableRenderer class="com.cloud.utils.log.CglibThrowableRenderer"/>  
   <!-- ================================= -->  
   <!-- Preserve messages in a local file -->  
   <!-- ================================= -->  
   <!-- A regular appender FIXME implement code that will close/reopen logs on SIGHUP by logrotate FIXME make the paths configurable using the build system -->  
   <appender name="FILE" class="org.apache.log4j.rolling.RollingFileAppender">  
    <param name="Append" value="true"/>  
    <param name="Threshold" value="TRACE"/>  
    <rollingPolicy class="org.apache.log4j.rolling.TimeBasedRollingPolicy">  
     <param name="FileNamePattern" value="/var/log/cloud/management/management-server.log.%d{yyyy-MM-dd}.gz"/>  
     <param name="ActiveFileName" value="/var/log/cloud/management/management-server.log"/>  
    </rollingPolicy>  
    <layout class="org.apache.log4j.EnhancedPatternLayout">  
      <param name="ConversionPattern" value="%d{ISO8601} %-5p [%c{3}] (%t:%x) %m%n"/>  
    </layout>  
   </appender>  
Although we there are two log4j configuration files, they point to the same files.  In both cases the general log is /var/log/cloud/management/management-server.log

CloudStack Logging on XenServer


CloudStack logs are generated by the XAPI extensions that CloudStack adds to the XenServer host.  These extensions are deployed in /etc/xapi.d/plugins folder.  E.g.
 [root@camctxlabs ~]# find /etc/xapi.d/plugins | grep vmops  
 /etc/xapi.d/plugins/vmopsSnapshot  
 /etc/xapi.d/plugins/vmops  
 /etc/xapi.d/plugins/vmopspremium  
The XAPI extensions are written in Python, but they make use of bash shell scripts in the /opt/xensource/bin folder.  E.g.
 [root@camctxlabs ~]# find /opt/xensource/bin | grep secondarystorage  
 /opt/xensource/bin/copy_vhd_to_secondarystorage.sh  
 /opt/xensource/bin/copy_vhd_from_secondarystorage.sh  
The XAPI extensions use functions from the sm.util module.  The logging functions in this module write to the SMlog log file.  E.g. in the code below, echo(fn) uses util.SMlog to update the log file /var/log/SMlog.  The @echo declaration in front of copy_vhd_to_secondarystorage causes echo(fn) to be executed in its place. echo is passed the copy_vhd_to_secondarystorage function as a parameter.
 [root@camctxlabs ~]# more /etc/xapi.d/plugins/vmopspremium  
 #!/usr/bin/python  
 # Licensed to the Apache Software Foundation (ASF) under one  
 ...  
 def echo(fn):  
   def wrapped(*v, **k):  
     name = fn.__name__  
     util.SMlog("#### VMOPS enter %s ####" % name )  
     res = fn(*v, **k)  
     util.SMlog("#### VMOPS exit %s ####" % name )  
     return res  
   return wrapped  
 ...  
 @echo  
 def copy_vhd_to_secondarystorage(session, args):  
   mountpoint = args['mountpoint']  
   vdiuuid = args['vdiuuid']  
   sruuid = args['sruuid']  
   try:  
     cmd = ["bash", "/opt/xensource/bin/copy_vhd_to_secondarystorage.sh", mou  
     ntpoint, vdiuuid, sruuid]  
     txt = util.pread2(cmd)  
   except:  
     txt = '10#failed'  
     return txt  
 ...  
Logging messages are also generated by exception handling code in the sm.utils modules.  E.g. in the code above, util.pread2 is used to execute bash scripts.  If an error occurs, it will be reported to the SMlog file.

Example of Using the CloudStack Logs


In this example, I find out that I accidentally downloaded an out of date vhd-utils to my XenServer.

To start, I followed the instructions on how to build from master.  These told me how to build and run the latest management server.  After configuring a zone with S3 secondary storage, NFS staging storage, and a XenServer cluster, I waited for the system VM template to download and for the management server to start its secondary storage VM.  After a while, I noticed that the secondary storage system VM did not start.  Also, I saw that exceptions were appearing in console I used to launch the management server.  Therefore, I knew I had a problem.

The first step was to look for the exceptions that occurred in the management server log.  This is the vmops.log file, because I'm using a developer environment.

NB:  Look for the first exception and the logs just before it.  

Here's what I found:
 2013-08-05 10:16:01,853 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:null) Completed acquiring hosts for clusters not owned by any management server  
 2013-08-05 10:16:02,403 WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-11:null) destoryVDIbyNameLabel failed due to there are 0 VDIs with name cloud-8e789d62-3062-4a13-8235-35ca49b7b924  
 2013-08-05 10:16:02,404 WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-11:null) can not create vdi in sr 8544bcca-f54d-cbc5-151d-b91c822addba  
 WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-11:null) Catch Exception com.cloud.utils.exception.CloudRuntimeException for template + due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr 8544bcca-f54d-cbc5-151d-b91c822addba  
 com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr 8544bcca-f54d-cbc5-151d-b91c822addba  
     at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)  
     at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:864)  
     at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:70)  
     at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)  
     at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:625)  
     at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73)  
     at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)  
     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)  
     at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)  
     at java.util.concurrent.FutureTask.run(FutureTask.java:166)  
     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)  
     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)  
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
     at java.lang.Thread.run(Thread.java:724)  
 INFO [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-8124c385) Unable to contact resource.  
 com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1] is unreachable: Unable to create Vol[1|vm=1|ROOT]:Catch Exception com.cloud.utils.exception.CloudRuntimeException for template + due to com.cloud.utils.exception.CloudRuntimeException: can not create vdi in sr 8544bcca-f54d-cbc5-151d-b91c822addba  
Next, look at all the management server code that was executing at the time.  This code ends at the first line named in the stack dump.  E.g. 
 com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:793)
It starts around the last log message.  E.g.
 root@mgmtserver:~/github/cshv3# grep -n -R "destoryVDIbyNameLabel" * --include *.java  
 plugins/hypervisors/xen/src/com/cloud/hypervisor/xen/resource/CitrixResourceBase.java:3069:        s_logger.warn("destoryVDIbyNameLabel failed due to there are " + vdis.size() + " VDIs with name " + nameLabel);  
 plugins/hypervisors/xen/src/com/cloud/hypervisor/xen/resource/XenServerStorageProcessor.java:800:        s_logger.warn("destoryVDIbyNameLabel failed due to there are " + vdis.size() + " VDIs with name " + nameLabel);  
Have a look at the source code in this area.  E.g.
   private String copy_vhd_from_secondarystorage(Connection conn, String mountpoint, String sruuid, int wait) {  
     String nameLabel = "cloud-" + UUID.randomUUID().toString();  
     String results = hypervisorResource.callHostPluginAsync(conn, "vmopspremium", "copy_vhd_from_secondarystorage",  
         wait, "mountpoint", mountpoint, "sruuid", sruuid, "namelabel", nameLabel);  
     String errMsg = null;  
     if (results == null || results.isEmpty()) {  
       errMsg = "copy_vhd_from_secondarystorage return null";  
     } else {  
       String[] tmp = results.split("#");  
       String status = tmp[0];  
       if (status.equals("0")) {  
         return tmp[1];  
       } else {  
         errMsg = tmp[1];  
       }  
     }  
     String source = mountpoint.substring(mountpoint.lastIndexOf('/') + 1);  
     if( hypervisorResource.killCopyProcess(conn, source) ) {  
       destroyVDIbyNameLabel(conn, nameLabel);  
     }  
     s_logger.warn(errMsg);  
     throw new CloudRuntimeException(errMsg);  
   }  
In the source above, we see that a call was being made to the XenServer extension, which suggests we should look at the SMlog on the XenServer itself for that period of time.  

NB:  The SMlog files is regularly archived.  So make a copy of the file.  If it does not fit the time period, look at the dates of the archived versions to find the useful one.  Make a copy and use gunzip to uncompress the file.


In my case, the SMlog file shows that a call to the copy_vhd_from_secondarystorage extension threw an exception.  E.g.

 [2106] 2013-08-05 10:07:34.249054    #### VMOPS enter copy_vhd_from_secondarystorage ####  
 [2106] 2013-08-05 10:07:34.249197    ['bash', '/opt/xensource/bin/copy_vhd_from_secondarystorage.sh', '10.70.176.36:/mnt/cshv3/secondarystorage/template/tmpl/1/1/e35bed31-3d25-465a-9e56-02d817491550.vhd', '8544bcca-f54d-cbc5-151d-b91c822addba', 'cloud-8e789d62-3062-4a13-8235-35ca49b7b924']  
 ...  
 [2127] 2013-08-05 10:07:34.824402    ['/usr/sbin/vhd-util', 'create', '--debug', '-n', '/dev/VG_XenStorage-8544bcca-f54d-cbc5-151d-b91c822addba/VHD-9a7e2a96-511b-4d94-a745-2619abb99919', '-s', '2000', '-S', '2097152']  
 [2127] 2013-08-05 10:07:34.835447    FAILED: (rc 22) stdout: 'options: <-n name> <-s size (MB)> [-r reserve] [-h help]  
 ', stderr: 'create: invalid option -- S  
 '  
 [2127] 2013-08-05 10:07:34.835849    lock: released /var/lock/sm/8544bcca-f54d-cbc5-151d-b91c822addba/sr  
 [2127] 2013-08-05 10:07:34.843383    ***** vdi_create: EXCEPTION util.CommandException, 22  
  File "/opt/xensource/sm/SRCommand.py", line 94, in run  
   return self._run_locked(sr)  
 ...  
 [2106] 2013-08-05 10:07:35.021318    #### VMOPS exit copy_vhd_from_secondarystorage ####  
The cause of the exception is calling vhd-util. with an invalid option -S.  Indeed, the version of vhd-util in use on this machine does not provide for the '-S' option.  E.g.
 [root@camctxlabs ~]# ./vhd-util create -help  
 options: <-n name> <-s size (MB)> [-r reserve] [-h help]  
Whereas another XenServer in my data center does not have this problem.  E.g.
 [root@camctxlabs2 ~]# vhd-util create -help  
 options: <-n name> <-s size (MB)> [-r reserve] [-h help] [<-S size (MB) for metadata preallocation (see vhd-util resize)>]  
From this, I learnt that I need to find an updated version of vhd-util.

Final Remarks


The system VMs and web container for the management server have additional log files.  In the case of Citrix CloudPlatform these are listed here.

Thursday, August 01, 2013

Diagnosing Maven Dependency Problems

Background:

Maven provides dependency management.  You specify the immediate dependencies of each project, and Maven works out the transitive dependencies.

A transitive dependency list includes indirect dependencies.  E.g. if A depends on B and B depends on C, A's transitive dependency list is B and C.

Problem:

After regular maintenance, the Maven projects do not satisfy all dependencies at runtime..

In my case, Maven failed to make available org.apache.commons.io.FileUtils of the commons-io package.  E.g. 
 mvn -pl :cloud-client-ui jetty:run  
 ...  
 INFO [ConfigurationServerImpl] (Timer-2:null) SSL keystore located at /root/github/cshv3/client/target/generated-webapp/WEB-INF/classes/cloud.keystore  
 Exception in thread "Timer-2" java.lang.NoClassDefFoundError: org/apache/commons/io/FileUtils  
     at com.cloud.server.ConfigurationServerImpl.getBase64Keystore(ConfigurationServerImpl.java:453)  
 ...  
 Caused by: java.lang.ClassNotFoundException: org.apache.commons.io.FileUtils  
     at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:50)  
 ...  
However, commons-io is a known dependency, which can be demonstrated by using the mvn dependency:tree command.  E.g.
 root@mgmtserver:~/github/cshv3/client# mvn dependency:tree  
 [INFO] Scanning for projects...  
 [INFO]  
 [INFO] ------------------------------------------------------------------------  
 [INFO] Building Apache CloudStack Client UI 4.2.0-SNAPSHOT  
 [INFO] ------------------------------------------------------------------------  
 [INFO]  
 [INFO] --- maven-dependency-plugin:2.5.1:tree (default-cli) @ cloud-client-ui ---  
 [INFO] org.apache.cloudstack:cloud-client-ui:war:4.2.0-SNAPSHOT  
 ...  
 [INFO] +- commons-io:commons-io:jar:1.4:provided  
 ...  

Solution:

Inspect the effective POM to see how POM inheritance has modified the dependency specification.

The command mvn help:effective-pom will return the pom.xml as modified by properties inherited from it's parent and other pom.xml files in its inheritance hierarchy   Run the command in the same folder as the pom.xml you are interested.

E.g. for mvn -pl :cloud-client-ui jetty:run, first find the folder for the cloud-client-ui project.
 root@mgmtserver:~/github/cshv3# grep -R cloud-client-ui * --include=pom.xml  
 client/pom.xml: <artifactId>cloud-client-ui</artifactId>  

Then use mvn help:effective-pom to see how the dependency specification for commons-io differs from that of other files.


 root@mgmtserver:~/github/cshv3# cd client  
 root@mgmtserver:~/github/cshv3/client# mvn help:effective-pom  
 ...  
    <dependency>  
     <groupId>commons-configuration</groupId>  
     <artifactId>commons-configuration</artifactId>  
     <version>1.8</version>  
    </dependency>  
    <dependency>  
     <groupId>commons-io</groupId>  
     <artifactId>commons-io</artifactId>  
     <version>1.4</version>  
     <scope>provided</scope>  
    </dependency>  
 ...  
Notice that commons-io has the <scope> element, which changes the dependency scope.  Dependency scope tells Maven when it is responsible for satisfying a dependency.  A scope of provided tells Maven not to bother adding commons-io to the runtime environment.  Specifically, provided "indicates you expect the JDK or a container to provide the dependency at runtime."

Therefore, when you look at the list of libraries Maven makes available at runtime, commons-configuration is available, but commons-io is not.  E.g.
 root@mgmtserver:~/github/cshv3# ls -al client/target/cloud-client-ui-4.2.0-SNAPSHOT/WEB-INF/lib | grep commons  
 -rw-r--r-- 1 root root  168760 Jul 16 17:23 commons-beanutils-core-1.7.0.jar  
 -rw-r--r-- 1 root root  232771 Jul 16 17:23 commons-codec-1.6.jar  
 -rw-r--r-- 1 root root  571259 Jul 22 10:14 commons-collections-3.2.jar  
 -rw-r--r-- 1 root root  354491 Jul 16 17:23 commons-configuration-1.8.jar  
 -rw-r--r-- 1 root root  24242 Jul 16 20:06 commons-daemon-1.0.10.jar  
 -rw-r--r-- 1 root root  160519 Jul 16 17:23 commons-dbcp-1.4.jar  
 -rw-r--r-- 1 root root  53082 Jul 16 17:23 commons-fileupload-1.2.jar  
 -rw-r--r-- 1 root root  305001 Jul 16 17:24 commons-httpclient-3.1.jar  
 -rw-r--r-- 1 root root  284220 Jul 16 17:23 commons-lang-2.6.jar  
 -rw-r--r-- 1 root root  60686 Jul 16 17:23 commons-logging-1.1.1.jar  
 -rw-r--r-- 1 root root  111119 Jul 16 17:23 commons-pool-1.6.jar  
 -rw-r--r-- 1 root root  34407 Jul 16 20:08 ws-commons-util-1.0.2.jar  
Remove <scope>provided</scope> element from the inherited dependency, and the problem disappears.

Final Remarks:

If you run into a dependency problem, tell the Apache CloudStack mailing list.  Dependency problems are difficult to spot.  Pointing them out greatly helps the project keep the build in good shape.