Adventures while building a Silverlight Enterprise application part #41

In this post we dive into this tiny change that caused a weird bug in the WCF communication between our Silverlight client and one of our web services.

In the last post I mentioned we were about to go live with a new product. It’s my pleasure to report we have gone live since then and everything is going well, except for some minor issues. I also described a process where we would do to improve our development process. At this point we are in full swing with doing that.

While we are doing that, we have also planned a new release to fix the minor issues I mentioned before and add some small features for which we didn’t have the time earlier. As I was working on a bug fix for one of these issues, I ran into a bug that puzzled me at first, but once I figured it out was trivial. Because it took a fair amount of debugging and analyzing to get to it and it highlights a problem with refactoring some parts of your code, I figured I’d share it with all of you.

Sypmtoms

As I was testing this bug fix I had been working on, I ran into some weird behavior. Some of the data in the Silverlight client would not load, or so it seemed. At first I thought this might have something to do with the fix I did.

I started debugging the service I had changed and found out that, for some reason an id we send to the service in order to get the right data in, was empty. This was weird, because I didn’t change anything to that part of the code. I debugged the Silverlight client and found out that it did send the id correctly! That was weird. The only thing I could conclude was that the id must have been getting lost somewhere along the line between the code on the client and the code in the web service.

Debugging the issue

As this appeared to be some kind of a communication issue, I figured I’d break out my old friend Fiddler. I reconfigured the client to access the web service involved through http://ipv4.fiddler, instead of http://localhost and I started reproducing the problem. I browsed through the requests and there it was, the parameter named Id was correctly being transferred and had the right value. For some reason though that value was gone as soon as it reached my web service.

The only conclusion I could make at this point was that it had to have something to do with the service contract. I took a look at the the interface for my service and immediately I spotted a potential problem. The parameter was not called Id anymore, but it was renamed to id! This was done as a result of a warning produced by FxCop. It marked the casing of the parameter name as incorrect, so I corrected it, but then completely forgot about it.

An easy fix

In order to get this fixed, I made sure the casing in my .svc code behind was corrected as well. Then it was all a matter of rebuilding and publishing the service and refreshing the service reference in the Silverlight client and the problem was gone.

As it turns out it was a good idea to do a release on the cleaned up code, because this way we run into some of the consequences early on. It also shows that there is always a danger to changing service contracts, because it creates incompatibilities that might not be so easy to spot.

Adventures while building a Silverlight Enterprise application part #39

In this post we look at an issue we had while implementing a specific form of data entry for our application. Basically there was an oversight in our reasoning about CRUD operations that needed fixing. It also shows the power of GUID’s.

The situation

We are currently in the final sprint towards delivering our first new product release to a group of customers. As one might expect, we are in the process of testing the functionality that was built so far and we ran into some issues. One issue we encountered was hidden deep down in the framework and hidden behind some other bug in the Silverlight UI.

To get a better understanding of the issue we found, it’s important to have some basic knowledge about how are framework works, especially in terms of creating and updating records. We decided early on that creating a record is basically the same as updating one. The only difference in our service call is that we provide an empty GUID instead of an existing GUID. The framework should then detect that there is a new record, create a GUID for it and insert it into the Entity Framework context and all works well (except for the incidental problem with insert).

Recently we added two modules to our UI, which consist of a data grid. The data grid is filled with a default set of rows from one table and once the user fills in certain values, records are created in another table. In order to make this work there is a view which is added to the entity framework to provide the correct rows. We then have some code in the service layer to make sure we don’t get double rows when there are records available in the second table.

The bug

A bug occurred when we tried to enter a value into a record that was not yet available in the second table, saved the modules data and then change that value. Now we ended up with two records in the second table instead of one. A short debugging session revealed the cause. Both update actions where send with an empty GUID, so both we’re interpreted as an insert.

The solution

Solving it was not as easy. In order to prevent these kind of bugs, we already return an updated (or new for that matter) business object from the service. However, for new records, there is no id known (it’s empty), so we have nothing to match to, meaning we can’t update the record in the data grid, which leads to our problem.

In order to fix it, we would want to create a new GUID in the Silverlight client, send it to the server and use it there as well, however then our service would not interpret this as an insert. What we can do is send this new GUID in the actual property of the business object which we want to update. This did require an update of the framework where, in case of an insert (the passed GUID is empty) we now check to see if the business object has a GUID as it’s key already available then we use that GUID instead of generating a new one.

So now, we can generate a GUID in the client, send it with the insert and know which object was updated as soon as it is received in the service call back. This allows us to match the rows in our data grid to the business object and update accordingly, meaning no more double records.

In the end, this his the code we created to make sure we get a GUID if one is available:

   1:              idPropertyName = string.Format("{0}{1}", idPropertyName, IdSuffix);

   2:              PropertyInfo idProperty = dataType.GetProperty(idPropertyName);

   3:              Guid id = Guid.NewGuid();

   4:              if (idProperty != null)

   5:              {

   6:                  if (changedData.PropertyDictionary.ContainsKey(idPropertyName))

   7:                  {

   8:                      PropertyObject idPropertyObject = (PropertyObject)changedData.PropertyDictionary[idPropertyName];

   9:                      Guid requestedId = (Guid)idPropertyObject.CurrentValue;

  10:                      if (!requestedId.Equals(Guid.Empty))

  11:                      {

  12:                          id = requestedId;

  13:                      }

  14:                  }

  15:                  idProperty.SetValue(data, id, null);

  16:              }

  17:              this.Id = id;

  18:              SetProperty(this.IdProperty, id);

Adventures while building a Silverlight Enterprise application part #37

In this post we are looking at Entity Framework and specifically at some performance quirks.

The story

Recently we were testing part of our new software. One thing we noticed is that, besides the fact that the first time we load data, things are slow because of services starting up and Entity Framework loading metadata, also the first time we save some data, it is also slow.

We decided that is was worth investigating why this was happening. The goal was to at least come up with an explanation and some more insight into when this was occurring.

Analysis

Some preliminary analysis I made:

• This was not the same thing as starting up the service and Entity Framework loading metadata because of that. I knew that because I had just loaded data from the same service and model that I was now saving to.
• It was caused by something in the service. I could tell because we have a busy indicator active whenever a service request is pending completion.
• It was not caused by any specific code for one of our business objects as it didn’t matter which object I would save. It was always slow the first time.

As you may know, if you are following this series for some time, we have a bunch of complex generic code that’s involved when either loading or saving data. I decided my first task was to identify exactly what line or section of code was causing the delay the first time around. To accomplish this I placed some debug code which would log the exact time to the debug output. This way I could easily calculate where a delay would occur.

By following this strategy I quickly found out that the delay for the first save occurred when calling SaveChanges on our model. However, reflecting the code for the method did not reveal any obvious cause for this. I searched on the internet to find out if anyone had already cracked this problem, to find out that only a handful of people had seen the same issue, but non of them actually resolved it.

As I still did not pinpoint the exact cause, I decided to build a test application to demonstrate the problem and hopefully get a better understanding. As we are currently still using version 2.0 of Entity Framework and running in .NET framework 3.5, I used Visual Studio 2008 to build my test application on top of the AdventureWorks database. I needed it to do basically two things:

1. Load a single object from the model to initialize the Entity Framework model
2. Save a change to a single object from the model and time the operation

To make my tests more flexible I decided on a WPF UI that allows me to control the above steps and it presents me with the results instantly. Check out the screenshot below:

The times displayed are in milliseconds. Let me explain what you see here. On top are two rows of buttons, one for the full AdventureWorks model and one for a model with only the Customers table (both models do connect to the same database). Below that is a listbox with performance results for each save action. At the bottom there is a general log to see what is happening.

Basically what you see in the performance logs is this. The first save action is done after loading a customer from the complete model. It takes a whopping 177ms. The next save action, after loading the same record again, takes only 2ms! This reproduces the issue we’ve been having.

Now the next step is doing exactly the same thing with a model that contains only the Customers table, so there is a lot less metadata to be loaded. It should be noted that both models are placed in separate assemblies to make sure metadata of one model is not interfering with the other. The time differences are a lot smaller between the first save (28ms) and the second save (4ms). Note that as performance is reaching lower limits times can fluctuate because of other processes on the system, which can account for the 2ms increase between the large model and the small model.

Because of this we can now conclude that Entity Framework loads some metadata as the first save occurs. Unfortunately we can not do much about that, but at least we now have an explanation.

Taking the next step

But the research didn’t end there. One of my colleagues suggested testing this with .NET 4.0 and Entity Framework 4.0 to see if there would be an improvement there. So I converted the project to Visual Studio 2010 and recreated both models using Entity Framework 4.0. The times are like this:

• Saving the first time on the complete model: 82ms
• Saving the second time on the complete model: 7ms
• Saving the first time on the single table model: 36ms
• Saving the second time on the single table model: 3ms

So work has been done in Entity Framework 4.0 to improve performance while loading metadata. Please note that my measurements were made on a system that is not exactly high performance and thus the measurements are not very accurate, however the differences between EF 2.0 and EF 4.0 with the first save are still significant enough.

Downloads

You can download the source of both versions of my test application:

• Entity framework 2.0 test application
• Entity framework 4.0 test application

Please be aware that the first project is a Visual Studio 2008 project and the second one is a Visual Studio 2010 project. You can however upgrade the first project, without changing the results.

Conclusion

So, although we can’t improve the performance of the first save action on an entity framework model right away, at least we can now explain what happens and what is the actual impact. As this only happens when IIS recycles the Application Pool for our web service, the impact is minimal. Also we can soften the pain as soon as we migrate our service to Entity framework 4.0.

I hope you have learned as much as I have writing this post. If you have any questions, please leave a comment below and I’ll be happy to help you out.

Adventures while building a Silverlight Enterprise application part #34

In this post we’ll look into analyzing performance in a Silverlight Line Of Business application, including multiple service layers and we come across a common problem using ADO.NET Entity Framework including a not so obvious solution.

The story

At the moment we are working on getting out a deliverable to a group of customers as soon as possible. However, one of the issues that keeps coming up is performance. A general response by a lot of developers is to just scour the web for performance tips on different technologies used in the stack and apply as many of them as they possibly can. However I find it much more useful to analyze the problem first.

Getting some data

To get a better feel for the problem, I first needed to get some numbers. When starting out with a performance issue the first question to get answered is “where is most time being spend?” This usually leads to the point where most time is being wasted as well.

At first I started out with profiling our application with Visual Studio 2010, but this wasn’t very useful to us as it didn’t show us any data on communicating with the service. I do feel compelled to link to some useful information on profiling your Silverlight application as it isn’t available through the IDE just yet and it requires you to jump through some hoops to get it to actually show you any performance data on your code. You can find a decent post by Oren Nachman on this here.

I should mention that profiling our application revealed one small performance issue with creating dynamic objects, which we use to instantiate modules in our application based on authorization and application settings. However, looking at the big picture it is hardly relevant to us, while hard to remedy it in a good way for our application.

The next step for me was to actually get data on communication with our services. As we designed our application in a way that we abstracted our service connections in seperate generic classes and we use generic service calls to get to our data, the easiest way for me was to simply record the DateTime.Now.Ticks. To keep the impact of this as low as possible on any production scenarios, I decided to write this information to the VS debug window, through Debug.WriteLine. Also to make sure this can be turned off easily to allow for other verbose debugging information, I introduced a conditional compilation symbol called DEBUGGINGPERFORMANCE and wrapped all my recording code with it. The code for me looked something like this:

   1: #if DEBUGGINGPERFORMANCE

   2:             Debug.WriteLine("BusinessLayerConnection.GetDataAsync<{0}>({1}): {2}", typeof(T).Name, id, DateTime.Now.Ticks);

   3: #endif

Running the application with this debugging code gave me some interesting data pointing me towards a specific operation, which gets all the data for a particular module at once. Within that set of modules one stood out that was taking far longer then the other modules, so I decided to investigate that further. I decided to use Fiddler to make sure none of this is actually caused by bandwidth issues. The statistics clearly showed my that the problem was with the service taking up a lot of time before actually starting to send a response.

This led me to put similar code in the webservice. Note that Debug.WriteLine doesn’t support the format approach by default and you should wrap that in string.Format. To actually make for a better performance test, I decided to build a small test tool to just make the same service calls several times in a row. This makes sure we’re working with correct numbers instead of chasing some random external factor.

As I started testing like this, I found out there where a lot of messages in the debug output stating that ‘A first change exception of type ‘ had occurred. In this case a large part of these exceptions where related to another service we use, which couldn’t access one of our databases. This was obviously an easy fix and it did improve performance considerably.

Fixing a common Entity Framework problem

After getting one exception out of the way, I found there where some more exceptions being thrown. The exact message that was displayed was: A first chance exception of type 'System.NotSupportedException' occurred in mscorlib.dll. Performance data indicated that this took roughly 3 seconds for six exceptions in each request!

I found it weird to see an exception like this. A web search didn’t return anything useful, except that it could be related to Entity Framework somehow. A forum thread pointed to this bug report about dynamic assemblies, which in itself wasn’t very useful information either.

I also found people asking questions about this message all over the internet, but without any real solutions. I turned on the System.NotSupportedException in Visual Studio, attached it to the service for debugging and ran my test again. As expected it halted in the debugger on the point where the exception was thrown. As expected it halted in mscorlib.dll, in this case in the System.Reflection.Emit.AssemblyBuilder class in the method GetManifestResourceNames(). This does confirm that it is actually caused by the bug I mentioned earlier, however it doesn’t really help us with a solution yet. The question now is, “why is someone calling this method on a dynamic assembly?”.

Browsing up the call stack makes it very clear that this code is being called from the Entity Framework. In fact it is called from the System.Data.Objects.ObjectContext constructor, but why? Taking a closer look at the call stack there is some indication that it must have something to do with initializing the metadata needed for the Entity Framework. It also reveals that the exception occurred in the System.Data.EntityClient.EntityConnection class in a method called SplitPaths. Looking at the code in that method and the state it was in as the exception occurred actually showed me what caused the problem. In the SplitPaths method there is a string[] called results (what’s in a name, right?) which contained three strings: res://*/Model.csdl, res://*/Model.ssdl and res://*/Model.mdl.

Looks familiar? That’s because these url’s are found in the default connection string for Entity Framework and they should point to the three files that make up the Entity Framework metadata. For some more documentation on that look here.

What was actually happening here was that a dynamic assembly was asked for it’s resources as the Entity Framework code was looking for a metadata file in all the loaded resources, as indicated by it’s connection string. The dynamic assembly that was loaded in this case was the binary for the WCF web service that hosts our DAL. The fix was simple. Just point to the exact assembly by it’s full name in the connection string for each metadata file and it’s fixed. This is how the metadata entry in our connection string looks now (obviously with some name changes):

   1: metadata=res://OurApp.Data, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null/OurModel.csdlres://OurApp.Data, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null/OurModel.ssdlres://OurApp.Data, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null/OurModel.msl;

I hope you found this search for this fix as useful as I did and it helps you out. Please leave me a comment if you have any questions.

30-07-2010 EDIT: Added the metadata part of the connection string to make the solution more clear. Thanks to greg-joyal for pointing that out to me.

Message framework design considerations

A while back I posted some thoughts on using a message framework as a way of abstracting part of the distributed computing problem. I stated I would start and write some code and so I have. On the way I had to make some interesting design decisions I wanted to share with you. This article is about these decisions and what where some of the reasons I made these.

Targeting the right objects
One of the first things I had to make a choice about, was how to target the right objects. Basically I needed two scenarios.

The first scenario is that Object A needs some task to be done by an object of type B. In this case Object A doesn't care what instance of type B actually does the job. It doesn't even care where this instance might be. In this case Object A should send out a message targeting type B. This means using a fully qualified class name as the receiver address.

The second scenario is that Object A sends out a message to notify other objects but it doesn't know which objects might be interested in the message. In this case other objects should register themselves with a dispatcher, stating that they are interested in some type of message. In this case Object A shouldn't include a receiver address at all.

But what about interaction with the system? This wouldn't be sufficient in a multi user scenario, because you can't target a specific instance of an object, which would be needed to actually give feedback to the user. I've decided that being able to target a specific object is not part of the low level messaging. To achieve a scenario where you need users to only receive their own messages back, there should be a layer on top of the existing messaging.
I chose this approach because it would be highly impractical to keep track of individual object instances across the entire ecosystem. Remember that one statement is that an object can be anywhere and it shouldn't matter for sending a message. By introducing unique addresses per instance, now it starts to matter, because a sender needs to know these unique addresses.

Interprocess communication
Another important aspect of the messaging framework is the communication between different processes. Again an object can be anywhere, but also anything .NET should work. I've investigated on using WCF, because this is obviously a very flexible and configurable way of communicating, scaling out from local (on the same machine) to global (across the internet). However it does put some bloat on the framework for people who don't want to use it.

I've also considered using Microsoft Messaging Queue, which kind of makes sense for a messaging framework. However this would involve building several tools around MQ to make sure all process types would be able to access the message queue and it would also put a deployment strain on the framework, beyond what I find is acceptable.

I settled on WCF, which doesn't mean that MQ is completely discarded. In the future it might prove valuable to actually build libraries around MQ to incorporate it into the framework. It does mean that by default every process includes code for both a WCF host and a client. However these are only instantiated as soon as a process is attached to another process.

IMessageReceiver
To indicate that an object can recieve messages, it is needed that it implements the IMessageReceiver interface, which only contains one method (at least for now) that's called RecieveMessage. It takes a single parameter of type IMessage. The IMessage type has two properties, Sender and Receiver which are both strings (for containing the fully qualified classnames of the objects involved in the communication).

MessageDispatcher
To make sure objects don't have to worry about getting a message to the receiver, every process should have exactly one MessageDispatcher object, who's sole responsibillity is to collect messages and dispatch them to the right objects and/or to other processes as needed.

To attach to another process in the ecosystem, the process in question has to make a request to the other process. Because we already have a message in the processes interface, that's exactly what I want to use. I've introduced a FrameworkMessage type that implements IMessage, so I can send it across to another process, in effect registering as a client to that process. To make sure the other process can also communicate back, it's response is to send a registration message as well, to make itself known as a client process as well.

Conclusion
As you can see there is a lot to think about when building a messaging framework. I'll keep working on this every now and then. Next time I'll try to get some code in.

Pondering on distributed computing in .NET

After working on a highly scalable application for almost a year now, using Silverlight, WCF, Entity Framework and Sql Server and after reading and replying to numerous questions on these topics and seeing the struggle developers are having with these new ways of doing things, I started thinking about how this could be made easier.

What I figured was that I don't really care about what part of my software runs where, which is basically a 'cloud' way of looking at things. So why not do everything in the cloud then? Well, there are lots of reasons not to, including costs and customer trust, but that's not what this post is about. It boils down to us not wanting to do everything in the cloud just yet.

That thought however, led to something which is not very new, but hasn't really been very popular either. I guess object messaging describes it for what it is. Basically what I'm saying is that objects may or may not know each other, but any object can send out messages to other objects, it being send directly or through a message dispatcher. Some (but maybe not all) objects would be able to recieve messages.
I wanted to figure out why it is not very popular today and instead of just reading hundreds of webpages, trying to find out why, I thought a cool way of ganing this understanding would be to simply design and build my own messaging framework.

An object can be anywhere
The first thing to do is to make it clear to myself what the concept is going to be. My first statement is that an object can be anywhere. It can be on the same thread, in a different thread, on the same machine or on a different machine, it really shoudn't matter. All I want to do is tell the framework to send message X to object Y and it should be taken care off.

.NET == .NET == .NET
What I mean by this is that as a Microsoft fanboy I obviously wanted to build this in .NET (C#). I also want this to work on a wide variaty of process types, wheter it is a command line application, a Silverlight client, a service hosted in IIS or in Azure, or an NT Service, shouldn't matter. It should all connect implicitly.

Challenges
This does bring along some technical challenges. For example a Silverlight client can connect to a service perfictly but connecting to a Silverlight client in that same fasion is not going to work, so recieving messages there is going to be a challenge, especially if keeping it transparent comes into play.

Dealing with concurrency is another big thing that comes into play. All of a sudden everything that's not in my object is asynchronous, which can be a challenge. This brings a whole new way of developing objects. Whenever a message is recieved, do I need to stop work and check if it is relevant for what I'm doing right now, or should I just ignore it untill I have time to deal with it? How do I keep track of what I've send out in relation to what comes back in? These are all questions that all of a sudden need answering.

I'll be looking into this stuff in the near future, writing some code and hopefully sharing it with you. Please let me know your thoughts on the topic.

Adventures while building a Silverlight Enterprise application part #21

I came back from a short vacation today and read this post from Tim Heuer. I then realized that in all twenty parts I've written on our adventure of building a Silverlight Enterprise application, I've actually never elaborated about why and how we chose Silverlight and how we (or at least I) learned to use Silverlight technology. So now seems as good a time as any.

Let's be a lazy author and just follow the questions that Tim put up there.

Decision resources
Obviously when you go about to start a project you have to decide on what you are going to use and why. If you are looking at Silverlight, what factors into your decision?

In our case, one of the most important topics was to move from a client-server based solution with a (very) fat client to a multi-tier solution with a web based client. The first choice was to go with DotNetNuke as some experience was already available, however, as you may very well know, this uses regular ASP.NET Ajax like solutions, which we found very limiting in UI perspective. At the time Silverlight 2 was just about to come out of the Beta stage and we felt we should at least take a good hard look at it. We needed to have technology that was relatively easy to implement and yet have enough flexibility to make for a very useful UI.

So our major decision factors where:

1. UI possibilities / user friendliness
   
2. Development speed
3. Flexibility

Why did your company choose to adopt Silverlight (or choose not to)? Was there another technology that was chosen to be better? Why/why not?

We choose to go with Silverlight because it checked the boxes on the above list. We felt it was easier to build a good user friendly UI with Silverlight, in comparison with ASP.NET / Ajax. Also we experienced in our prototyping / prove of concept face, that building a UI with Silverlight is taking less time as well, because of the use of XAML and C# only. No HTML or JavaScript code made live a lot easier for us.

Another elementary aspect of Silverlight proved valuable to us. It was the fact that you can embed controls inside other controls and work with templating. In combination with declarative databinding this became an important topic for us during decision making.

Again we where comparing to DotNetNuke, which had some advantages over Silverlight as well. One of the things that DotNetNuke had which was considered an advantage, is the fact that you could than access the database directly from your client side code. In hindsight I'm glad that we can't do this with Silverlight and just have all the logic in a service layer. This proved to be much more flexible for a lot of scenarios we already ran into.

What is the most important thing in deciding if Silverlight is right? Feature set? Existing technologies? Rapid development? Other reasons?

At the time, Silverlight 2 was not exactly existing technology as it was just coming out of Beta. This was a concern for us, but the fact that this was Silverlight 2 and that there was already an active and growing community, combined with the fact that Microsoft had plans for updates layed out already, pulled us across. The feature set was definitely important. The way databinding is implemented is important to us, as we are building a data rich application. Also the controls available at the time as default, but also trough the Silverlight Toolkit and the fact that third party companies literally jumped at Silverlight 2 helped easing the choice even further.

Rapid development was obviously something that needed prove as no real world examples where available at the time. We build prototypes of several scales and found that it was indeed fast enough to build something with Silverlight 2, even without experience with the technology.

Learning resources
On learning – how do you best learn? Do you prefer “atomic” samples? These are the ones that you can just pop in and figure out a task-based situation (i.e. how do I open a file in Silverlight). Or do you prefer more of a “lesson plan” approach to things? This would be a series based on a task (i.e. Build a Media Player in Silverlight).

For me personally a lesson plan approach was new. I did part of the digg API sample series, but after several steps I lost interest, mainly because some of the topics where not of interest at the time and some of the topics where just to easy for me. This makes me lean towards atomic samples, altough as a starting place I can see why people like this lesson plan thing. Later on atomic samples are obviously the way to go. They have the advantage of showing only the bare minimum of implementation without the burden of having to go trough previous steps to understand code that was already there.

On medium – in either types of these learning paths, what is your preference? Video? Written step-by-step guides? Labs?
When you are completely new to a topic, watching a video is great. It takes away some of the effort of having to follow along with some article and just being showed how things work. This is also great for some more indept topics, with less code and more explanation.

If looking for a reference, or if I need to actually start coding on something, I do prefer written material, as it is a lot easier to just scan trough and look for that bid of code, or that bid of explanation you needed, instead of plowing trough twenty minutes of video trying to find that five second shot of the code you need.

I've tried labs in the past and I found that they are not my thing. I tend to get bored very quickly because usually the level of instructions is making things to easy. Also setting up to do a lab feels like a lot of work most of the time.

On topics – what are the top 3 topics you expect when learning a new technology? How do you on-ramp yourself when you know nothing about it? Do you expect to learn the tools first? Or jump right in to data access?

This is actually a very tough question to answer. It realy depends on what a technology looks like. When looking at Silverlight I felt that I already knew enough about the most important tool: Visual Studio 2008. Of course Blend became part of the toolset I use and over time I did watch videos and read tutorials on Blend to learn some tricks here and there. So for me, while learning Silverlight, diving right into important topics was very helpful. I do feel it is important to include tips on how to use tools, where this is important to the topic at hand.

For example if you are talking about databinding and data access, it is handy to demonstrate on how to do this in Blend including using sample data. If something like this becomes to elaborate, at least make sure you point out to the public that this is a handy feature they should know about and tell them where they can find more information about it.

Other notes
Something that bugged me for quite some time when getting started with Silverlight was the number of resources that where available for Silverlight 2 Beta 1 and 2. While being a great help to win over management to go with Silverlight, this made live a lot tougher when trying to find useful resources for Silverlight 2 RTM as a lot of times the important bits would not work in the RTM.

Something else that I found is that, altough silverlight.net is a great resource when learning Silverlight, it is a bit of a mess. Videos are not categorized in a way that makes them easy to find and there is not a proper search function on the site. Also I think it would be very helpful to have a search engine that can search for Silverlight content for your specific version only. So if I'm working on Silverlight 3, I might not be very interested on Silverlight 1.1 and 2 content. Or at times I might not care if it's Silverlight 2 or 3 content, because they are likely to be compatible any way.

As a final note I would like to encourage anyone who hasn't already done this, please go to the article of Tim Heuer and comment. Or you can comment on this article as well.

Adventures while building a Silverlight Enterprise application part #20

This post is about some of the less technical (a.k.a functional) stuff around code generation, especially when using XAML, it being in either Silverlight or WPF. This is a no-code post, that is intended to give you some food for thought when planning for code generation on different levels.

The basics
If you're involved in applications that have large amounts of, well, something, it may very well be feasible to use some form of code generation. You could use this for many things and most of them, if not all, have been done already. Some common items in software that may be candidate to code generation are:

• Business objects
• Screens / windows / forms in the GUI
• Reports
• Database scripts

And obviously there are a lot more. There are only two basic requirements to code generation. What you plan to generate needs to be in some format you can actually supply from some peace of code and what you plan to generate needs to have some form of repetitiveness to it.

The most common thing to generate is your Business objects. They tend to be very repetitive and as they are in some programming language it usually is enough to generate simple text. If you're application is based on some relational database, you might even be able to leverage the metadata in your database as a source for generating your business objects.

Setting goals
The first thing you should do when thinking about code generation is set goals. What I mean by that is you should think about not only what you're going to generate but also why. Is this generated code only a starting point and are you never going to regenerate? Or are you interested in regenerating code at some point? The answers you give on these questions have a big impact on how your generation process should look.

If you're generating only once, you need to make sure that any metadata you use in the generation process is definitive. Experience tells us this is nearly impossible to achieve, so at least make sure the impact of a change is as minimal as possible and also make sure your generated code is readable.

If you are going to regenerate, consider the fact that not only do you need to run the process and make sure you're not loosing any customizations on the generated code, you also need to make sure that any source of metadata is consumed again. More on that later. Another concern when regenerating is integration testing. When you regenerate part of your applications source, you've altered all that source and changes are something is going to fail because of it. Make sure you think about how you're going to at least test for failures, or better yet, prevent failures in the first place.

Have a business case
You may or may not have to pitch code generation in your organization, however you should always have a business case for a code generation process. What I mean by that is you should at least have some clue as to how much effort it will take to build the generation process and how much time it will safe you in the end. Basically you need to have a Return On Investment calculation, at least for yourself, before you go off and invest a lot of time into this. I know it may seem that you won't invest that much at first, but trust me, you'll most likely end up spending at least twice the time you figured before you actually set down and made a decent estimate.

When looking at this business case, make sure you do not only include the obvious. Everyone can figure out that it saves time not having to write a specific amount of code. What is getting left out a lot is the quality of the code that is generated. It's consistent, which means you can test one instance of the code and you'll know the rest will work. This also means it reduces the amount of bugs and if bugs are found in generated code, it reduces the amount of effort to fix these bugs.

Having worked for several software companies in different roles I know from experience that having less bugs is an even bigger win than spending less money on writing code. Having a bug doesn't only cost money, it also impacts reputation and customer satisfaction at some point. Having less bugs is good for everybody.

When things get tricky
You obviously can't generate every bit of code in your application, however most of us will be tempted at some point to try and generate something to hard. One of the most underestimated peaces of code to generate is part of the GUI.
Let's say you're building a Line Of Business application that has one hundred forms in the application that are being used for data entry. This is obviously a very common scenario. Not many developers like to go out and build that many forms. It just feels silly doing repetitive work if your job is all about automating repetitive tasks.

However, there is more then meets the eye here. Let's say you want to do this for a Silverlight application. At first glance, all you need to generate is some XAML for each form and it needs to contain some TextBlocks to contain the labels and some other controls to be used for input. All seems fine so far.
Now consider the metadata you need to actually generate this. You may say that you already have this as it is the same metadata you've used for generating your business objects. Is it? I doubt that. The only thing your business object tells you is what fields it contains and what types they are. They don't tell you anything about their position in a form. There is no information about any special behavior that this one specific control needs different from most others.

All that extra information needs to come from a functional designer, it being some application expert or even one of the developers in the team. Now let´s say you have this metadata in some usable form. You´re still not capable of generating a functional form. Now you need to bind to your business objects. Initially this isn´t that hard, but things start to get complicated when your datamodel evolves or when the functional designer desides to link in that one field from some other business object on the same form, because that is easier for the user.

Now let´s say that you even got that under control. Now you still need to ty in behavior. You need to attach events, which means you also need to generate the code behind for your application, but you also want to be able to extend the code behind at some point. The metadata you need for that process has to come from a developer at some point.

If you now look over the big picture for generating XAML forms, you have not one but three sources of metadata:

• The datamodel
• A functional designer
• A developer

And these three sources of information need to be brought together so they can be used in a single generation process. Building something like this quickly becomes a project on it's own, rather than just a meens to an end.

Putting it in perspective
So shouldn't you do this? That's a good question and there is no simple yes or no answer. It all boils down to your ROI calculation as I explained it earlier. How much effort goes into building and implementing this way of building your GUI in the organization? And how much effort is going away from building and maintianing it in a traditional fashion?
Obviously if you have an increadibelly large application, this might actually work, but another case in which this may work is if you're a one-man-show. If you do the datamodel, the functional design and develop the application, then this might just be worth the effort. The reason for that is because you control all the variables and there is no team to keep in check on how to work with this.

So what if you figure out this is not a good approach for your project, should you just go off and build a hundred forms by hand? Well, not so fast. If you can eliminate one or two of the sources of information, by not generating that part of the code (or at least not in the same component), you may just be able to make it feasible. The thing to keep in mind is that combining metadata sources makes things more complicated, so try and keep them seperated.

I hope this article helps you out with your choices in wether or not to generate specific parts of your application code. If you have any questions, remarks, etc. you know the drill.

Adventures while building a Silverlight Enterprise application part #13

At times you run into decisions made at the early stages of design, that you regret, or at least find inconvenient. This happens to the best of us, especially when using new technologies. In our case we ran into the downside of using ADO.NET Entity Framework (EF) in our application and part #13 already looked like a good one to look at why this is the case.

Why EF doesn't always make sense
Because we are talking a big application here (several hundred forms and grids), and because we are using Silverlight 2 as our client, we decided to make a Business Layer Service, that takes care of our CRUD operations and make this generic. This means we now only have five operations instead of having over a thousand. This still makes us very happy as maintenance is low and you don't have to bother with a lot of code in this part of the system, however...

... in EF, whenever you do a save you need to supply all the connected entities or at least a reference it can use. For a small application with a very limited number of entities this is not a problem. Just code it up for each class and you're done. For our application however, we have a generic save and although we are able to write code for separate business objects (which translate into entities in our service), we'd rather not do this for something that should be the same for all our objects. I should state that this choice was a direct result of us using Silverlight 2 as our client. Writing several hundred save routines sounds like a job for an intern, which we don't have. :-)

We did end up building a generic save, but it did take some hassle to make it work in all scenarios. Especially combining both saving a new entity with saving an already existing one proved to be somewhat problematic.

If I then look further back in the process and see the effort that wend in to building the generic loading in several scenarios, getting a single item, getting a list of items based on some criteria, etc., I'm realy not sure if it was worth the effort with using EF. Why not just build your own?

The good
There is one reason however, and that's the model. The fact that you have in intermittend model that all your code is based on is something that may very well prove very valuable in the future. You could make changes to your datamodel and just change your mappings to go with it and presto, it still works.

If you're reading this and think to yourself: "Design your datamodel right and there is no need to do this", then you realy should get a few more years of experience in the new world. It is no longer acceptable to say that something can't be done, because the datamodel doesn't support it. Customers will not understand / don't care about this and just tell us to fix it and they are right (they eventually pay my salary :-) ).

Conclusion
So was it a bad choice? Should you or shouldn't you use EF in your larger applications? In our case I'd say it hasn't payed itself back yet, but it very well may still do that. I guess it still depends on your application and you should always have a good look at requirements like lifespan, scalability, performance, etc.. In general I think that EF can work for larger applications (as it does work for us), but you shouldn't be caught by what the demos try to make us believe. You can't build your data access services in a matter of minutes, especially if you need your service to work with Silverlight 2. It can still take months to do it with EF.

I hope this post is helpful to you when making your technology choices. Please leave any comments and/or questions below. I'm always happy to read and reply to them.

Adventures while building a Silverlight Enterprise application part #10

Yeah! It's part 10 of my Adventure series. Sorry, I'm not giving a party or anything, but it feels like a milestone to me.

In this episode we are going to look at an issue I ran into with WCF and Silverlight 2 when using a specific structure in the datacontract. In our case, we wanted to use a Dictionary that would contain another Dictionary as the value. Have a look at our data contract to have a clearer picture:


[DataContract]
[KnownType(typeof(Dictionary<string, object>))]
public class CompositeType
{
private Dictionary<string, object> _properties;
private string _stringValue = "Hello ";

public CompositeType()
{
_properties = new Dictionary<string, object>();
}

[DataMember]
public string StringValue
{
get { return _stringValue; }
set { _stringValue = value; }
}

[DataMember]
public Dictionary<string, object> Properties
{
get
{
return _properties;
}
}
}

As you can see, the dictionary type is a part of our DataContract, trough the use of the DataMember attribute. We also added that same type to our known types list, trough the use of the KnownTypes attribute. We could have used the ServiceKnownTypeAttribute on our interface type as well, with the same result.

If we would now use a Silverlight 2 client with a service reference to this service and make a call to our service, everything seems fine...

...until, that is, we use a Dictionary instance as a value in the Properties property.
In that case an exception occurs, stating:

There was an error while trying to deserialize parameter http://tempuri.org/:GetDataUsingDataContractResult. The InnerException message was 'Error in line 1 position 584. Element 'http://schemas.microsoft.com/2003/10/Serialization/Arrays:Value' contains data of the 'http://schemas.microsoft.com/2003/10/Serialization/Arrays:ArrayOfKeyValueOfstringanyType' data contract. The deserializer has no knowledge of any type that maps to this contract. Add the type corresponding to 'ArrayOfKeyValueOfstringanyType' to the list of known types - for example, by using the KnownTypeAttribute attribute or by adding it to the list of known types passed to DataContractSerializer.'. Please see InnerException for more details.

If we wouldn't have added the dictionary type to our known types list, the service call would have timed out, without a proper exception, indicating that some problem would have occurred in the service.

I've been digging to see if I could come up with a solution. Googling the exception I got plenty of examples with derived types, but non with nested types and late binding, like I'm using. After some reading on datacontracts and digging into the reference code in the client, I figured that known types in the service might just not be distributed to the client. To prove my theory I started looking at the generated service client instance at runtime. I looked in the EndPoint property which contains the contract, which in turn contains the operations for the contract. As known types are usually scoped at the operations level, I checked to see if it contained any items in the KnownTypes property and in fact it didn't. This means my theory is correct and I should add the dictionary type to the known types list of the operation. As I didn't want to do this in generated code, I decided to do it at runtime:


private Service1Client GetServiceClient()
{
Service1Client client = new Service1Client();

foreach (var operation in client.Endpoint.Contract.Operations)
{
operation.KnownTypes.Add(typeof(Dictionary<string, object>));
}

return client;
}


If I now run the client, everything runs like a charm. What we actually did, by adding these known types, is telling the DataContractSerializer to include our dictionary type when deserializing any incoming xml messages.

You can download some sample code here:


I hope this helps any of you encountering the same issue. If you have any questions or comments, please leave me a comment below.

Adventures while building a Silverlight Enterprise application part #8

Well, it's been a while again. Sorry about that, but it's been really hectic around here.
At the moment we are digging into the back end of our application. You may remember from previous episodes that we use WCF services in our back end. One of the most important services in our project is our Business Layer Service. This service provides access on our database through ADO.NET Entity Framework.

So that's great. We can generate our data layer code, because we use EF, but how about the rest of our code? As we have a large amount of entities, one thing we didn't want to do was to provide service calls for each of them. It would be a lot of boring work, with high maintenance code as a result. So how would we make this generic?

Well, the WCF part of things isn't that hard. We defined a base class that provides us with a property bag. As we pass an instance of this class over the line, all that is send is a dictionary of keys and values. All our derived classes do, is access the property bag from the base class through properties. These derived classes, we generate from a meta data store.

Loading these objects from the EF is not as straight forward, but still not that much of an issue. We simply provide a string with the name of the object we want to the service, which we use to load the object through the EF method GetObjectByKey. We then reflect trough the properties of the resulting objects and fill the property bag of our generic class.

The real problems start when you want to provide a generic way to build lists of objects. We looked at many possible solutions, only to come up blank. Eventually it hit me, we needed a hybrid solution, between generic code and generated code. So I went looking for a solution using facilities provided in our base class to get to a list of specific EF entities and here is what I came up with:

public static IEnumerable<BusinessObjectBase> CreateList(Dictionary<string, object> criteria)
{
IEnumerable<BusinessObjectBase> result;
string queryText = GetQueryText("MyEntity", criteria);
using (CobraXeContext context = new CobraXeContext())
{
ObjectParameter[] parameters = GetParametersFromCriteria(criteria);
ObjectQuery<MyEntity> query = context.CreateQuery<MyEntity>(queryText.ToString(), parameters);
ObjectResult<MyEntity> objectResult = query.Distinct().Execute(MergeOption.PreserveChanges);
result = GetListFromObjectResult(objectResult);
}
return result;
}


On the service this looks easy, but actually getting to this point proved quite a challenge. Especially generating an EF query that worked, even if I needed to filter on properties from a different, but related entity was difficult. The documentation on joining in EF SQL is vague when it comes to it's exact syntax and it's also not completely intuitive as how to get to related information.
Eventually I came up with the following rule for building a query, which is implemented in the method above:

select value [entity] from [entity] join [jointable] on [entity].[joinproperty]=[jointable] where [predicates]


In my code I've assumed that the name of a joinproperty is always equal to the jointable name.

After building this and having my business classes regenerated, I started wondering about what code is to be generic, what is to be generated and what code is specific enough to be developed by hand, for a specific use. After thinking about this for several days, I came to the conclusion that there is no straight answer to this question. Maybe this decision is what has become part of our 'art', as I like to call it. Each situation is different and each program is different. I've worked on code for three months, only to have it run for two hours and never again. I've also worked on code for an hour and saw the same code being used years after.

Yep, it's been an interesting week. I hope you have learned as much as I have and I'm looking forward to read your comments.

Adventures while building a Silverlight Enterprise application part #7

Today I wanted to state this, although for some of you it may seem obvious, others may get caught up in this pitfall. Always make sure that any webservice you use from Silverlight is thread safe, even in the same session.

Here is what could happen. You have some UI action, for example the click of a button, which triggers a call to your webservice. Inside your webservice you use some non thread safe object, to do some operation, for example a SqlConnection to read some data from your database. All is well, but as you build your webservice you may consider to share this SqlConnection, because you need to do two queries within one call.

That's where things start going wrong. On first sight, everything seems to work fine. Until that is, you start bashing away on that button and all of a sudden your Silverlight application crashes mysteriously in the completed event of your service call.
After some debugging you may find out that this is caused by the fact that your SqlConnection failed for some reason. Some more debugging may tell you that the SqlConnection is actually closed, or in the state of being opened, or it may just throw an exception, telling you to remove your results and try again. Any of these is caused by the fact that you use a single SqlConnection to do more than one operation on.

"But what about performance?" I hear you ask. Well, what about it? There is never, and I mean never, a reason to reuse an instance of SqlConnection. The reason is that Microsoft provided us with a great feature called a connection pool, which will just cycle trough a set of already created connections for each request it gets. All you have to do is make sure you open the connection as late as possible and close it as soon as possible. This way you never hold on to a connection to long and the change of a connection pool running out of connections is as small as possible.

But what if you are using some other non thread safe object, that would impact performance when you re instantiate it for every call? One thing I would probably want to do is, disable the control that triggers the action, as soon as an action takes place and only enable it again as the action is complete, or maybe even at a later time, if that's appropriate. This way you make sure that a single user can not run multiple operations and performance should improve already.

Another solution may be to actually rethink the design of your service. Maybe you need a more extensive model than a standard service. For example, you could choose to build an actual Windows Service with a WCF interface. This would allow you to queue up any calls comming in and having them handled by other threads. This model would be useful to provide, for example, a search service.

Hope this was helpful to at least some of you and I'm looking forward to reading your comments.

Random hints and tips on Silverlight

Lately I ran into a few quirks while doing some prototyping in Silverlight and I taught it would be nice to share them with you people.

Silverlight Application Name
Actually this is more about the name of the website that VS2008 creates for you when you create a Silverlight application. If the name of the Silverlight application is relatively long, it may very well result in a to long name for your ASP.NET website. This will result in an error as that project is created.

IntelliSense with Silverlight assemblies
If you use a Silverlight class library to build some usercontrol in and you reference it in a Silverlight application, IntelliSense will not show the class while your typing your namespace alias declaration in xaml. You have to compile the class library before it shows up.

Getting a template applied on your custom control
This may seem obvious, but it had me struggling for quite a bit. Whenever you added a template in generic.xaml to apply it to some custom control it is important to know it has to go into a folder called themes. Otherwise the template won't be found. This is only since release and is not well documented.

Refactoring doesn't effect XAML
Personally I'd say this is an oversight in Visual Studio 2008 SP1. Whenever you use refactoring to rename a property that is used in XAML, it wont change the name in the XAML, breaking any references you had to the property. I guess it's back to manually refactoring those.

Tab navigation on a popup doens't work by default
Whenever you use a popup control to display some controls, you can't use tab navigation by default. You'll need to add a content control to host this functionality. Jeff Handly explains it on his blog.

Allways make sure IsolatedStorageSettings.Save() is called at some point
Altough Save is called automatically called whenever a Silverlight application ends, this could actually fail, without you noticing. If, for some reason, saving settings throws an exception, you will not be able to tell this, because the application already ended. But if you call Save, it will throw the exception at that point and you can resolve any issues.

Make sure your webservices are thread safe
This may not seem like a Silverlight tip, but I put it here, because in most cases the Silverlight programmer is confronted with the initial exceptions. Inside the webservice, if you use a non thread safe object, make sure you do not share instances throughout your code. Create new once as needed and destruct them as soon as you're done. Read more about it here.
You may want to check back here, as chances are I'll post new information here from time to time.