Performance, Overhead, and csUnit

Over the last several months we have been working on improving the performance of csUnit. Not that it was really bad. However, we thought that it is about time to look at whether we can improve it. Generally we improved two areas, one of which is of interest to users, while the second area is more of interest for the csUnit development team. Area one is basically the overhead that csUnit adds to the code that you execute. As a user you would like this to be as low as possible as you would like your resources (computer, but also your time) is mostly spent on what you really want to achieve. For improving the performance we are using JetBrains' dotTrace. It's an easy to use tool, and provided us with just the information we needed to reduce the overhead. For a recipe with about 4 assemblies containing tests you can expect an overhead of about 2 to 3 seconds on a decent notebook. Faster machines lead to better results. Currently we have a few options left, which we consider for future improvements, but right now we believe that we played the major cards. The second area was improving csUnit's own test suite. Again with the help of the profile, but also with csUnit's own "Statistics" tab, we identified the most expensive tests, and worked through them one at a time. We increased the use of mocks, but we also improved the testability of csUnit by introducing more internal interfaces. Overall we are now able to run the test suite with a speed of approx. 6,000 tests per minute on a decent notebook. We are currently shooting for a new version in March. We have a short list of known issues we want to fix. Stay tuned!

Mandatory Use Of Attributes in csUnit (Proposal)

We are considering to drop csUnit's support for finding Tests and TestFixtures based on naming conventions, e.g. fixture names end with Test and tests starting with Test. Instead I'd like to change the specification so that it becomes a requirement to use attributes, e.g. TestFixtureAttribute and TestAttribute, to mark fixtures and tests. The benefit would be a more consistent use of csUnit, e.g. TestAssemblyFixtureAttribute can be expressed with attributes only. The impact would be that all test fixtures and all tests that still rely on naming convention would have to augmented by assigning the TestFixtureAttribute to all fixtures and TestAttribute to all tests. Depending on the number of tests this might a substantial work. Send your feedback and comments to manfred at csunit dot org. Thank you!

What About csUnit 2.1?

Admittedly there wasn't much news around csUnit 2.1 for a while. But be assured that we are working on it, and we are hoping to make a release available in September 2006. Currently we are refactoring the code base, in particular we would like to make the "categories" feature much more powerful. A collateral will be an improved user interface for both the stand-alone application (csUnitRunner) and the Visual Studio 2005 add-in. Stay tuned!

Startup Tests in csUnit 2.0.4

The GUI runner of the new version csUnit 2.0.4 executes some startup tests. These checks certain aspects of the installation. The dialog box showing the execution of those startup tests will disappear automatically after about 3-4 seconds. The next maintenance release will execute these tests only once when the GUI has been started for the first time after installation. Currently it is run each time the GUI is started. Also, these tests are available from an option in the Help menu. Adding these startup tests is an experiment for improving the user experience of csUnit. All kind of feedback is highly welcome and appreciated. Please send email to manfred at csunit dot org. Thank you!

csUnit 2.0.4 Released

Another version of csUnit is available: csUnit 2.0.4. This is a maintenance release for csUnit 2.0.x, which is the version that has addins and wizards for VS2002 and VS2003. This release contains one bugfix for csUnitVSWizard (SourceForge Tracker ID 1528508: Can't find Wizard.config). Furthermore the setup has been modified so that csUnit assemblies, which need to be be COM visible, no longer register with a CodeBase key in the registry. If you need support for VS2005 you may try csUnit 2.1.x, which contains addins and wizards for VS2005 and also supports .NET 2.0.

csUnit 2.1.1 BETA released

Another beta release of csUnit 2.1 is available at csUnit's web site. Remember, version 2.1 is the first csUnit version that supports .NET 2.0 and Visual Studio 2005. For this beta version we focused on two major areas:

• Addin for Visual Studio
• Performance Engineering

The addin for Visual Studio is available again with a similar feature set as in previous versions. After adding a reference to csUnit to your project you will have two additional entries in your context menu for the project: On the performance side, we decided that we wanted to measure and reveal the amount of overhead csUnit adds to your unit tests. csUnit is the first unit testing tool, which provides that feature. We used dotTrace by JetBrains and Microsoft's CLRProfiler to improve overall performance. Our data shows that we have been able to reduce the overhead caused by csUnitRunner, the standalone GUI application to about 1 second per 100 tests. Compared to almost 10 seconds in the past, this is a significant improvement and saves valuable development time. Concurrently we have also started to talk to different tool vendors, some of them already providing support for csUnit. The idea is to make sure that csUnit provides for the best possible integration in order to help increasing your productivity.

csUnit 2.1.0 BETA Released

Finally a .NET 2.0 compatible version of csUnit is available. Although it doesn't have an addin for Visual Studio 2005 yet, it's still worthwhile to take a look. For instance: Did you ever wish you had a setup or teardown method for all your tests in one assembly? Then try this: [TestAssemblyFixture] public class TestData { [SetUp] public void CreateInMemoryDB() { //... } [TearDown] public void CleanupInMemoryDB() { //... } private InMemoryDB _database = null; } This is a just an example for illustration purposes. Don't send me a message asking for my code for the InMemoryDB class. I don't have it! As for csUnit 2.1.0 BETA, you can find it at http://www.csunit.org/download.

csUnit 2.0.1 released

Finally we got another release of csUnit finished (see version 2.0.1 on csUnit's web site). It took a while but there were many requests in do so. Although we don't have support for VS.NET 2005 yet, we plan to have that ready by end of March 2006. The new release offers a few enhancements, e.g. categories for tests and imcreased flexibility in the user interface for selecting which tests to run. Next we'll focus on:

• updating the collaterals for version 2.0.1
• then version 2.1, which will add support for VS.NET 2005

Determining Name And Path Of Output File in VS.NET 2005 Automation

When implementing an extension, e.g. an addin for VS, you may want to determine the path (location) and name of the assembly that is being built as part of a project. One way would be to read the project file, e.g. *.csproj. As it happens to be an XML file it is easy to determine the correct node. A different approach though is using the Project automation object. Project has a set of properties and for C#/VB.NET/J# projects it contains a property named "OutputFileName". The value of that property is the assembly name. Next, you need to determine the path. First you need to know the path to the project, which you can determine using the directory part of Project.FileName. This gives you the location of the project file. In order to determine the output path for the assembly, you need the active configuration for the project, which you obtain using Project.ConfigurationManager.ActiveConfiguration. The ActiveConfiguration object has a property named "OutputPath", which is the relative path from the project file to the output folder. One might be tempted to use ActiveConfiguration.OutputGroups. For that to work the project needs to have been built successfully once, which might not always be the case. Furthermore, Configuration.OutputGroups doesn't work in a handler for the solutions event Opened. Therefore I settled on the following code for determining the path and the name of the target assembly:

string GetOutputPathAndFile(Project p) {
   string outputPath;
   string outputFileName;
   string outputPathAndFile;
   string projectLocation;
   try {
      Configuration activeConf = p.ConfigurationManager.ActiveConfiguration;
      projectLocation = (new FileInfo(p.FileName)).DirectoryName;
      outputFileName = p.Properties.Item("OutputFileName").Value.ToString();
      outputPath = activeConf.Properties.Item("OutputPath").Value.ToString();
      outputPath = Path.Combine(projectLocation, outputPath);
      outputPathAndFile = Path.Combine(outputPath, outputFileName);
   }
   catch(Exception ex) {
      System.Diagnostics.Debug.WriteLine(ex.Message);
   }
   return outputPathAndFile;
}

If you are aware of a better solution, please don't hesitate to comment! I'm eager to learn.

Update: User Control in Tool Window of a Visual Studio Addin

After some more experimentation it seems that the last parameter of CreateToolWindow2() is null, if the hosted control is in a separate assembly / project. For instance: Suppose you have a solution containing two projects, let's call them MyAddIn and MyControls. The first one is your add-in itself and the second one contains some controls you want to have in a separate assembly as you want to use them also for other solutions, e.g. a stand-alone application. Now, if you have a UserControl in the project MyControls, let's call it MyUserControl1, then you can still host that control in your add-in. However, in this case the method CreateToolWindow2() returns null as the value of the last parameter. If on the other hand you add a UserControl, let's call it MyUserControl2, to the add-in project 'MyAddIn', and then host MyUserControl2 in the add-in, the last parameter of CreateToolWindow2() will contain a reference to your MyUserControl2 instance. Of course MyUserControl2 can in turn contain one or more instances of MyUserControl1. Strange that whether or not a UserControl is contained in the add-in assembly/project or not makes a difference regarding the last parameter of CreateToolWindow2(). Am I missing something?

The invoked member is not supported in a dynamic assembly - Reprise

Here is another workaround for the issue with the form designer: In some cases it is sufficient to just close and re-open the solution and/or Visual Studio. From my perspective this indicates a bug in the IDE. So it's back to Microsoft.

The invoked member is not supported in a dynamic assembly.

You might encounter another issue with the forms designer in Visual Studio 2005. When you have located a user control on the form, and the user control is in a different project, and the user control uses GetExportedTypes() to determine visible types, then you might receive the following error message when trying to view the form:

The invoked member is not supported in a dynamic assembly.

The error message is typically accompanied with a stack dump including information about the line of code which causes the problem. In order to solve the issue you need to find a different way of coding. In my case I had to use GetTypes() instead of GetExportedTypes(). After that change the form designer worked again as expected.

Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.

Sometimes when working with a solution containing multiple projects, with several forms some of them containing user controls, Visual Studio .NET 2005 produces the following error when you try to open the form in the form designer:

Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.

In some cases the following approach helps solving this issue:

1. Close the form designer
2. Close the solution and Visual Studio
3. From the project that contains the problematic form delete the bin and the obj folder
4. Start Visual Studio again and load your solution
5. Rebuild the solution
6. Reload the form

Not nice, but at least it's a workaround. I don't know yet how to reproduce the issue, but it seems to be related to user controls in one project which are used in a form in a different project.

User Control in Tool Window of a Visual Studio Addin

Visual Studio .NET 2005 offers some new APIs for making it easier to host a managed add-in. For instance, the function CreateToolWindow2() allows hosting a custom control within a tool window. A shim control is no longer necessary. The function call might look like this:

_toolWindow = toolWins.CreateToolWindow2(
   _addInInstance,
   asmPath,
   ctlProgId,
   "AgileTest",
   guidString,
   ref objTemp);

The documentation states that the objTemp parameter will contain a reference to the user control hosted in the new tool window. With my experiments so far I have never found that variable to be anything other than null. In order to get a reference to my hosted user control I therefore used a different approach. First I defined an event on the class level for the user control like this:

public static event EventHandler ControlCreated;

private void AgileTestHostCtrl_Load(object sender, EventArgs e) {
   if(ControlCreated != null) {
      ControlCreated(this, null);
   }
}

For this event I defined an event handler in the Connect class of the add-in:

void AgileTestCtrl_ControlCreated(object sender, EventArgs e) {
   _hostedControl = (AgileTestCtrl)sender;
}

Registering the handler on a class level event is easy:

AgileTestCtrls.AgileTestCtrl.ControlCreated += 
   new EventHandler(AgileTestCtrl_ControlCreated);

Now when the user control is loaded the event is fired, and the event handler is called. The limitation of this solution is obviously that should you need several instances of the same tool window it might get more complicated to distinguish between them. One option to solve this might be to use an event that carries additional information in its properties. In my case as I don't need more than one instance of the tool window and hence of my user control, this solution works fine and reliable. Please drop me a note if you are aware of a better solution. Thank you!

This is about DotNet

This blog is about DotNet. I'm a founding member of the csUnit team. In this blog I'd like to focus on general and special .NET 2.0 questions. Feedback is most certainly welcome!