Writing code != developing software

There has been some fuzz online about whether or not should everybody learn how to write code. In this post I’d like to add my point of view and take into account what the profession of developer involves (or should involve).

If you have not followed this discussion, you might want to read up on some other posts firsts:

Please Don't Learn To Code - Jeff Atwood

Please Learn To Think About Abstractions - Scott Hanselman

In a nutshell the fuzz is caused by public figures announcing they are going to learn how to write code.

The car mechanic

There has been several comparisons with other professionals already. However I think they are not really suited for what is happening here. I like to explain why learning how to write code doesn’t mean anything by projecting this to the profession of the car mechanic.

Let’s say I announce I’m going to learn how to work on an old car engine. If you analyze the skills I need to work on a car engine it almost becomes laughable:

• I need to be able to open the bonnet of the car
• I need a basic understanding of tools in the toolbox (a spanner, a screwdriver, etc.)
• I need to be able to start the car and drive it in order to test my work

Anyone can see that it I work on a car engine with only the above skills, I’m very likely to ruin the engine. At the very least we can conclude that being able to work on an old car engine doesn’t make me a car mechanic.

Let’s project this example back to the software developers profession. If you want to write code you need the following skills:

• I need to be able to install and run an IDE
• I need to be able to type code in the syntax for the language I’m using
• I need to be able to use the IDE to compile and/or run my code

If you are going to use only the above skills to write a program, changes are you are not going to achieve the results you want. In any case the above skills are not nearly enough to become a software developer.

The software developers profession

Jeff Atwood makes an excellent point in his post when he states that being a software developer is not about writing code, but about solving problems. I completely agree with Jeff on this point. If you read the post he refers to when making this point (here), you’ll see he paints a very bleak picture about developers being extremely disconnected from the users they are building software for. I personally do not have that experience, but then I’ve always worked in relatively small teams (six developers being the max).

In my career I’ve coached several developers and I’ve always been an advocate of what to me is the single most important skill any developer should have is the analytical skill. I’ve even given short trainings in analysis. When another developer asks for my help solving some problem, I always try to be very verbose on how I get to my analysis.

And as Jeff Atwood also states in his post, writing less code is better. In order to do that, being analytical about what you’re trying to achieve is crucial.

Obviously you can have all the analytical ability in the world, but if you don’t have a proper understanding about what you’re dealing with, it being the inner workings of an operating system, or the politics around some customers feature request, you can not make a proper analysis. This means you do need knowledge.

This brings me to a problem that is fairly unique to the software developers profession. Code is not knowledge, however it is transferred and documented online in vast quantities. This has created a whole breed of so called developers who simply go online and copy code for what they think solves a particular problem they’re having, without actually understanding what the code does, nor why they are having a certain problem. The term Lamer comes to mind.

The danger with that is that someone who does that might think that they solved a problem, however they might very well have not. And because of the lack of understanding they are unable to verify if the code solves the problem, nor are they able to troubleshoot the code if needed.

The software developer needs to analyze the users problems, then come up with a solution that fits the users needs. Again it’s not about the code. It’s about the solution. It requires a deep understanding of computers and the users needs and it requires analytical skills and a creative mind to come up with a fitting solution.

This is just my two cents. Please leave any comments, questions or rants below. I’d love to hear how others perceive this.

Moving from Silverlight to MVC3

In this post I’d like to explain why we felt the need to abandon our Silverlight development in favor of MVC3.

As some of you may very well know, I’ve been pushing Silverlight within our company from as early as Silverlight 2 Beta 1, back in 2008. Still, this last summer I felt I had to raise the question: Is this still the right technology for our company?

Technology choices are about requirements

When we first chose Silverlight as our UI technology, we came up with a set of requirements. Our application should have a (near) zero footprint on client systems. It should be accessible through the web. Deployment on clients should be effortless or none existent. It should look great without a whole lot of effort. Silverlight checks all these boxes (it still does).

We write summer of 2010. The iPad has been around for several months now and it has been a great success. Smartphones are widely adopted to the point that we predict our customers will get interested in support for platforms like these when using our software.

Another thing we see is people working on more different platforms than before. Was it ok to ask for a certain browser in 2009, in 2010 our customers began to ask for a browser independent UI, not only on Windows, but also on a lot of other platforms.

To see what exactly would be the response of our customers we decided to build a native iPhone application with a limited set of functionality. We started the project early 2011 and published the first Dutch HRM app for any smartphone platform in may. It was a hit, not only because of responses by our clients, but also because of responses by our competitors.

Requirements shift, the world changes

This lead us to conclude that we needed to do something to support a new requirement. Our application, at least in part, should be available on mobile devices and tablets. And we should make sure we could drop the browser requirement we had active, even with our Silverlight 4 UI*.

That is why, during one of our meetings about what to build next, I felt the need to ask “Shouldn’t we move away from Silverlight and adopt ASP.NET MVC3 with HTML5?” It sure was painful, after investing over two and a half years in Silverlight to even suggest dropping it. However I felt then, and I still do, that a change had to be made. After some more debate on the consequences and some research, we are now putting in the effort to build the first parts of an MVC3 UI with the first steps into HTML 5 and CSS 3.

Luckily not all our efforts with Silverlight where in vain. We did deliver a good working product and we are committed to using, expanding and improving the middleware we developed for it. (Do you see the power of a Service Oriented Architecture?).

So is Silverlight really dead now?

Some of you may remember this post. In that post I state that Silverlight is actually the better technology for writing Line Of Business Applications. Even after researching ASP.NET MVC3 and HTML 5 further, I do still believe that, however because of the changes in how people use computers it simply fails to check one box for our customers.

Silverlight is no where near dead. It will still be used in many ways, including new platforms like Windows 8 and, for example Microsoft CRM 2011. Microsoft is integrating Silverlight in more and more of their own platforms. Also with the introduction of Silverlight 5 there will be many cool new features coming to the browser. Therefor I doubt Microsoft will drop Silverlight any time soon.

Consequences for Developers 42

Unfortunately our move away from Silverlight does have some consequences for this blog. First of all, my long running series of posts “Adventures while building a Silverlight Enterprise application has ended with part 41 (which is slightly painful for an obvious reason).

It also means I will no longer be writing posts for SilverlightShow.net. I’d like to use this opportunity to thank Svetla for supporting me for the past years.

And finally, it will also mean a lot less content on Silverlight, but hopefully a lot more content on MVC3, HTML 5, CSS 3, jQuery, etc..

It’s the end of another interesting period in my professional live and the start of a new one. I hope you have enjoyed my articles on Silverlight and I hope to see you around.

 

 

* The IE dependency was not actually caused by Silverlight, but by the integration with the Microsoft Report Viewer we used.

Code generation is a basic developer skill

“Give a good developer a 40 hour task and he’ll spend 39 hours writing a program that can do the task in 1 hour.” – unknown

Really? This seems risky!

It can look risky, can’t it. You might end up with a half working program and not enough time to do the job. There are approaches that reduce this risk, but more on that later.

Why should you even bother? Simple math tells us it’s not faster to do so (at least not in the above example). However, experience tells us that in general most jobs will be repeated. Now, with the above example, if it’s only repeated once, you have a 39 hour profit, as you can now do the job in 1 hour.

Obviously, things don’t work like that in the real world. Automating a process usually takes longer than actually manually doing the work. In fact, when you want to automate something, that usually means you first have to do it by hand at least once, so you know what you’re automating.

But it’s also unlikely that you will only repeat a process just once. Often tasks are done many times and then it becomes worth the effort of automation. Let’s look at a current example.

A common case

As some of you may know, I’m working on a line of business application with a Silverlight UI and a WCF service layer (although the used technology doesn’t really matter for this case). Part of writing most line of business applications is writing business objects that represent your model and writing logic to retrieve data from the database, either directly or through some ORM.

Another part is writing lot’s of “forms” if you will, to input data into. Usually these forms follow a common pattern. Both of these parts, the business objects and the forms, contain a lot of repetitive work to do manually. Some of the code can be moved into generic super classes, but most of the code is of a nature that doesn’t allow for generalization. This is where automation comes into play. In this case, why write all that code yourself if you can also generate it?

When we first started out with this application, we did set out to build some code generation. Unfortunately most of us didn’t have a lot of experience with code generation as such, so wrong choices were made (C# generation with a StringBuilder is not the way to go and the same goes for Code DOM). After the initial versions of our code generation engine, we were forced to spend all our time on getting a first release out to our customers and no more work on automation in any form was a predictable result.

So, are you a good developer?

We all fall into this trap at some point. I know I have several times. You let a deadline pressure you to much, or you go back to your old naïve thinking saying that this will just be some demo code and it won’t end up in production anyway. Whatever the reason, we all end up breaking our own rules from time to time.

Don’t get me wrong, that doesn’t make you a bad developer. In fact it means you are focusing on how to providing value to your customers, which is key for a good developer. Also, breaking these rules from time to time keeps us alert on why they are rules in the first place.

However, we should also take time every now and then to take a step back and have a look at what we are doing. In my case I found we should put a lot more effort in making our development process more efficient. This means making more code generic, refactoring existing generic code to fit our needs better, and generating more code.

Generating code, a daunting task?

To some developers generating code may seem like a daunting task. Don’t worry. That’s a good thing, because you won’t just dive in without giving it some thought. As you can see from our example, having a developer that things “Oh, this is easy. I’ll just take a StringBuilder and push out lots of .cs files" is far from ideal.

Actually the first step in code generation is not writing a code generation process. It all starts with meta data. If you don’t have something to base your code generation process on, then how are you going to generate code in the first place?

In case of a line of business application, often your data model can provide you with some meta data. In fact we started out with a stored procedure that would take meta data from our data model in SQL Server and put it in another database. Then we would generate source from that data.

Another source for meta data may actually come from people entering that data into an application. We use that too, to allow our functional specialists to provide us with information to generate the forms in the GUI.

Once you have your meta data in some form or the other, you can write your generation process. Actually, that part doesn’t have to be hard. In the end it’s just creating text from data, which most web developers have been doing for years. The main problem is integrating your process with your IDE. That is where T4 comes in.

As some of you may know T4 is already used in Visual Studio to generate source code. It also provides you with a simple button to trigger the execution of all T4 templates within your solution. This is obviously a lot easier than adding files generated by some external program.

As for editing these templates, Visual Studio doesn’t come with standard support for this (it’s treated as text by default, so no IntelliSense). If you are serious about T4, you should check out Tangible. They have an editor for T4, integrated in Visual Studio, both in a free edition as well as a pro edition with more features. Both features do come with IntelliSense and Highlighting, but the free edition only supports limited namespaces and assemblies. You can find them here.

But what about those risks?

I hear you. Starting with this it can be scary. The trick is to start small and then keep on growing your code generation process to support more and more scenario’s. This way you can work on your code generation in iterations and have a working end result often. This reduces the risk, because you can, at any point, decide that you will no longer extend the code generation process, but extend the generated code by hand.

Also, before generating any code, you should have at least written and tested it once, to make sure you know the structure of your code and what meta data is needed in order to generate it. This way you also know you will end up with code that works.

Conclusion

I hope you realize the potential of code generation and how this can really make you more productive. I can tell you from experience that it also makes software development more fun, even if you have to build a lot of the same. With the right tools I’m confident that any development team could benefit from automating their work.

Adventures while building a Silverlight Enterprise application part #40

As we are close to releasing our first version and we will have some time on our hands to do clean up on our sources, today we want to look at some of the things we would like to do in order to improve our codebase.

Well, I never thought this series of posts would last this long and have this many parts. We might even make it up to part 42 :-).

The situation

As I wrote above, we are close to releasing our first version to the client. If you’ve followed this series of posts, you may know that we’ve been working under the gun for the past eighteen months or so to actually get to this point. To actually have our team meet deadline after deadline, we had to take some shortcuts here and there.

“What?!”, I hear you scream, “Deadlines are never an excuse for shortcuts!”. Sure, in the academic world that’s true, but in the real world business of software, this doesn’t hold up. Even large software companies like Microsoft constantly make choices about if they can do without a certain workload before releasing the next version. We make these choices on a daily basis as well.

But now we are faced with a rare (or dare I say unique) opportunity. We are actually allowed to reserve time to go back and improve the quality of our codebase. We also are allowed to invest in improving our software development platform in order to improve quality or increase productivity.

The approach

This begs the question, what are we going to do with this time? I’ve composed a list of things we want to do and I’d like to dive into each of these points in this post. Please keep in mind that the following points are not in order of importance. They are in this order for different reasons.

1. Remove any compiler warnings
2. Do a complete code review, dealing with the following
   1. Make sure all code adheres to our coding standards
   2. Make sure all code can deal with multi language and culture differences
   3. Decide on what to do with TODO’s
   4. Check if workarounds for bugs in external code can be removed
   5. Deal with obvious performance killers
   6. Mark unclear code for refactoring
3. Migrate our main web services to .NET 4.0
4. Migrate to Entity Framework 4
5. Analyze what code can be made more generic and refactor
6. Improve our code generators
7. Get automated testing
8. Get nightly builds

Remove any compiler warnings

One might argue we should have removed compiler warnings as we went along. And when we started out we did, however as we were confronted with deadlines, this was one of the things we decided we could drop. The reason for this is that we don’t need this for our application to work correctly.

So why fix it? Well, some of these warnings can prevent bugs and some warn about inefficient code. In order to get all our code up to a higher standard it therefore is useful to adhere to these warnings and fix them as necessary.

Do a complete code review

We do this after we have fixed the compiler warnings as we then end up with code as it should be. We may have fixed things we otherwise would have to fix in the review. It also works the other way around. We also now have a codebase without compiler warnings, so we can actually keep it that way as we change code.

Make sure all code adheres to our coding standards

This is and obvious advantage to create. We want all code to follow some rules so we can make assumptions about it. This does not only make it easier to make changes or write new code. It also makes it easier to read, which makes up for around 80% of any developers workload.

Make sure all code can deal with multi language and culture differences

We have international customers who want to use our software in a different language (the default is Dutch). Also their culture settings may differ. We therefore need to make sure we check our existing code base to see if this is taken in account and if not, fix it.

Decide on what to do with TODO’s

During the development we have had times where we though “we should change this part of the code to work so and so.” As we where up against deadlines and most of these changes would have serious impact, we marked these spots with TODO comments.

Again, I can here you say, “well, execute them!”, however we will find TODO’s that are outdated, either because they ended up being implemented directly, because we had too, or because they are no longer relevant because of some decision or insight we had.

Check if workarounds for bugs in external code can be removed

As we work with some third party code, either written by Microsoft or bought from a third party, we encounter bugs in these parts every now and then. We found it good practice to mark any code that we use as a workaround with a comment, describing what we did to work around the bug. Over time these parts might have been updated and chances are we can remove some of these workarounds, benefitting both performance and maintainability of the source.

Deal with obvious performance killers

By this I mean we do a quick analysis of what is happening in a piece of code and see if we can find any obvious inefficiencies. If we do, we remove them on the spot.

We won’t do any performance measurements, so the inefficient code has to be obvious. For example, we might encounter code that uses two queries to retrieve data from the database, where we could have used only one. You don’t need a masters degree to figure out, that you should make this into a single query.

Migrate our web services to .NET 4.0

We’ve already migrated our client to Silverlight 4.0 and Visual Studio 2010. However, because we didn’t want to take any risks with our web services, we didn’t migrate them at the same time. Now that we have the change bringing our web services to .NET 4.0 (and obviously moving them to Visual Studio 2010 in the process) will make our lives a lot easier.

Migrate to Entity Framework 4.0

Right now we are still on Entity Framework 2.0, which was the latest version as we started development late 2008. We haven’t upgraded since, but because there are several improvements in both performance and functionality, we would like to move to Entity Framework 4.0.

Analyze what code can be made more generic and refactor

Right now we are not keeping it DRY all throughout the code base. Especially when it comes to implementing specific functionality we ended up solving the same problems more then once and sometimes even in different ways.

Also, although we tried to take as much care as possible to prevent repeating code, we still had some cases where there was no time to implement a proper generic solution for a recurring problem.

Now we have time to analyze these cases and refactor code as needed.

Improving our code generators

Some of you may know that we use code generators for several parts of our code. Currently we generate our business classes, their collections, XAML for our UI (including an empty code behind) and resource files to go with the XAML.

To create this code, we use metadata from our database models and metadata that’s supplied by our functional specialists. They supply us with information to generate the UI and the resource files.

In order to make this process more efficient we would like to redesign the user interface for the tool that the functional specialists use and change the way we bring all the metadata together. Then we can actually use the combination of this data to generate better and more source code.

Get automated testing

I know, we should have done this from the beginning. However, everyone knows this is one of the first things management will eliminate if time becomes an issue, especially if the team is not experienced in the field of TTD and needs some time to get used to it.

This does pose a problem right now, because it seems almost impossible to introduce actual unit testing after such a long time of development, because we would need to introduce dependency injection in such a large code base. We’ll have some research to do in this area and we’ll have to make some though choices.

Get nightly builds

This one will be a though one to realize, considering all the parts we have and the way we work together, but it does have some important value in integrating source and making sure all the parts keep working together.

A mistake or a consequence?

As you can see there is still a lot of improvement to make. So where did this come from? Well, most of the issues I addressed were results from conscious choices we made. We knew this would have consequences later down the line.

Other things, like the quality of the code generation and a need to rethink our approach was due to new insights and working with some of the concepts in our organization for the first time.

And some things, like code reviews have been a process to get implemented in the culture of our organization. Changes like these tend to take time (and lots of it).

Because of this I have to conclude that the fact we have to do all this work is a consequence of choices we made. And because we knew most of the consequences as we made those choices, I feel it’s a good thing, especially if you consider a lot of development teams do not get this chance.

Have you ever been in a similar situation? Let us know your experience by leaving a comment.

Adventures while building a Silverlight Enterprise application part #39

In this post we look at an issue we had while implementing a specific form of data entry for our application. Basically there was an oversight in our reasoning about CRUD operations that needed fixing. It also shows the power of GUID’s.

The situation

We are currently in the final sprint towards delivering our first new product release to a group of customers. As one might expect, we are in the process of testing the functionality that was built so far and we ran into some issues. One issue we encountered was hidden deep down in the framework and hidden behind some other bug in the Silverlight UI.

To get a better understanding of the issue we found, it’s important to have some basic knowledge about how are framework works, especially in terms of creating and updating records. We decided early on that creating a record is basically the same as updating one. The only difference in our service call is that we provide an empty GUID instead of an existing GUID. The framework should then detect that there is a new record, create a GUID for it and insert it into the Entity Framework context and all works well (except for the incidental problem with insert).

Recently we added two modules to our UI, which consist of a data grid. The data grid is filled with a default set of rows from one table and once the user fills in certain values, records are created in another table. In order to make this work there is a view which is added to the entity framework to provide the correct rows. We then have some code in the service layer to make sure we don’t get double rows when there are records available in the second table.

The bug

A bug occurred when we tried to enter a value into a record that was not yet available in the second table, saved the modules data and then change that value. Now we ended up with two records in the second table instead of one. A short debugging session revealed the cause. Both update actions where send with an empty GUID, so both we’re interpreted as an insert.

The solution

Solving it was not as easy. In order to prevent these kind of bugs, we already return an updated (or new for that matter) business object from the service. However, for new records, there is no id known (it’s empty), so we have nothing to match to, meaning we can’t update the record in the data grid, which leads to our problem.

In order to fix it, we would want to create a new GUID in the Silverlight client, send it to the server and use it there as well, however then our service would not interpret this as an insert. What we can do is send this new GUID in the actual property of the business object which we want to update. This did require an update of the framework where, in case of an insert (the passed GUID is empty) we now check to see if the business object has a GUID as it’s key already available then we use that GUID instead of generating a new one.

So now, we can generate a GUID in the client, send it with the insert and know which object was updated as soon as it is received in the service call back. This allows us to match the rows in our data grid to the business object and update accordingly, meaning no more double records.

In the end, this his the code we created to make sure we get a GUID if one is available:

   1:              idPropertyName = string.Format("{0}{1}", idPropertyName, IdSuffix);

   2:              PropertyInfo idProperty = dataType.GetProperty(idPropertyName);

   3:              Guid id = Guid.NewGuid();

   4:              if (idProperty != null)

   5:              {

   6:                  if (changedData.PropertyDictionary.ContainsKey(idPropertyName))

   7:                  {

   8:                      PropertyObject idPropertyObject = (PropertyObject)changedData.PropertyDictionary[idPropertyName];

   9:                      Guid requestedId = (Guid)idPropertyObject.CurrentValue;

  10:                      if (!requestedId.Equals(Guid.Empty))

  11:                      {

  12:                          id = requestedId;

  13:                      }

  14:                  }

  15:                  idProperty.SetValue(data, id, null);

  16:              }

  17:              this.Id = id;

  18:              SetProperty(this.IdProperty, id);

Adventures while building a Silverlight Enterprise application part #38

In this post I want to look into a problem most of us will encounter while building a Line Of Business application in Silverlight. It involves UI paradigms combined with the asynchronous communication model.

The problem

We have all run into it at some point. You have a form that a user needs to fill out, which contains a couple of fields and something like a save button. When the user clicks the save button you validate their input and if the validation fails, you block the save action.

All is well. The user can correct the input and try again at will. However, validation has a downside. It’s limited to either true or false and no way in between.

Let’s consider a scenario where you would like to have confirmation from the user. Now you could hack this into the validation system by using a flag and if the user doesn’t change the value and saves a second time it must be right. However I don’t think this is a very good solution and it would confuse users.

I need a dialog!

So we want some other way of asking the user for input. The usual way of doing this would be to present the user with a dialog where the user can press a button to choose whether or not it’s ok to continue. We worked with them for many years and they are intuitive to use.

In Silverlight however they pose a problem. Of course you can use the dialog included in the framework. However this uses the browser to present the dialog and it’s not a good looking thing. More importantly it’s very limited in it’s functionality. Fortunately it’s not that big of a deal to just build a user control that provides the functionality we want. We can even make it modal by just placing a Canvas over the entire UI and then putting the user control in a Popup (or use a childwindow for that matter).

Then the issues start. If you are to build a user control to display dialogs, like we have done, you most likely implemented it using a callback to signal the application that the user has clicked a button in your dialog and you’re done. Although this is a classic asynchronous model, working with it can become problematic.

Splitting code

Let’s go back to our scenario. We have a form with some fields. Some fields get validated and some can lead to a dialog. The steps we would take to save our data before introducing the dialog look like this:

1. Validate the input
2. If all fields are correct start saving the data
3. Once the save is completed, refresh the current data

Now if we introduce a single dialog that allows the user to choose between stop and continue the steps are like this:

1. Validate the input
2. If all fields are correct, check if the dialog is needed
3. If the dialog is needed, show the dialog
4. Once the user has made a choice and wants to continue, start saving the data. If the user does not want to continue, stop.
5. Once the save is completed, refresh the current data

Looking at these steps, we go from one event handler for the save completion, to adding a callback for the dialog. So instead of having two methods with code for saving, we now have three methods. And with every dialog we add, we add another callback, thus another method containing code for our save action. You can see how this gets out of hand quickly.

So make it synchronous…

Well, that would solve our problem, however… Silverlight only has one dispatcher for the UI-thread. This means that as soon as you block the UI-thread, your application becomes unresponsive (basically it just hangs until you unblock). While displaying a dialog that is not a very practical thing to do, as the user would not be able to click a button (defeating the whole purpose, right?).

In effect this means there is no straight forward to provide synchronous UI interaction. It’s going to be event based and thus asynchronous. There is a way around this, however. You can’t block the UI thread, but you can block any other thread, so if we move all the code involved in saving to a background thread we can eliminate a callback for the dialog. Here is some example code to show you how calling such a dialog would look:

   1: private void saveButton_Click(object sender, RoutedEventArgs e)

   2: {

   3:     // Make sure to create your dialog while still running on the UI thread

   4:     _dialog = new DialogWindow();

   5:     

   6:     // Pass off the save logic to another thread

   7:     ThreadPool.QueueUserWorkItem(SaveData, this);

   8: }

   9:  

  10: private void SaveData(object stateInfo)

  11: {

  12:     if (DialogWindow.ShowDialog(_dialog, stateInfo))

  13:     {

  14:         // Actually save the data

  15:     }

  16: }

So basically you run anything involved with the save action (except for validation, perhaps) on a background thread. I also pass a reference to the calling UI control. The reason for that will become clear soon.

While running in the background thread, you can call my DialogWindow.ShowDialog method and it will block your thread. This means you can use its result right away.

Run code on the UI thread

In order for this to work we obviously need to be able to run some code on the UI thread and then block our thread while waiting for a result. Here is the code for the ShowDialog mehtod:

   1: public static bool ShowDialog(DialogWindow window, object stateInfo)

   2: {

   3:     bool result = false;

   4:     DependencyObject dependencyObject = stateInfo as DependencyObject;

   5:  

   6:     // If we got passed a dependency object and we have access 

   7:     // to it's dispatcher then we are on the UI thread and we can't block

   8:     if (dependencyObject == null  dependencyObject.Dispatcher.CheckAccess())

   9:     {

  10:         window.Show();

  11:         return false;

  12:     }

  13:  

  14:     Action action = new Action(window.Show);

  15:     // Run the Show method on the UI thread

  16:     dependencyObject.Dispatcher.BeginInvoke(action);

  17:     // Block this thread until we have a result

  18:     while (!window.DialogResult.HasValue)

  19:     {

  20:         Thread.Sleep(50);

  21:     }

  22:     result = window.DialogResult.Value;

  23:  

  24:     return result;

  25: }

First thing we do is check if we are actually on the UI thread. If we are, we don’t want to block. Be aware that the CheckAccess method is not shown by IntelliSense for some reason. It compiles without any problems though.

If we are not on the UI thread, we can actually use the dispatcher of the dependency object passed to us, to actually run the Show method on the UI thread. Then we simply wait for the result and pass it back.

Conclusion

In order to have a synchronous dialog we simply introduced a thread to run our code on which runs some code back on the UI thread and waits for it to complete. This should make it easier for us to compose more complex flows in our code.

You can download the code here.

What’s the fuss around Silverlight about?

Since day one of PDC10 there has been a lot of press attention for what appears to be a change in strategy for Microsoft around Silverlight. As I’ve gotten several questions from people concerning the stories published by several respectable sources and how this would impact the business, I decided to write down my take on this.

The PDC10 debacle

So what are people on about? It basically comes down to two things. One is the “lack” of attention for Silverlight on the PDC10. The other is Microsoft announcing that they completely back HTML5. It was elaborated on by saying that there has been a shift in the strategy around platform independent web technology. Also, because there has not been an announcement on Silverlight 5, people are scared into believing there will not be a Silverlight 5.

Most people interpreted this by concluding that Microsoft will limit their investments in Silverlight to the Windows Phone platform. Let’s assume they are right in their conclusion. If that is true, then it’s devastating for all those developers and companies who have invested heavily in Silverlight in the past years.

My analysis on Silverlight

So how do I look at this? Well, there is a number of things I see differently from most people. Let’s have a look at some of those.

HTML5 vs Silverlight

A lot of people see HTML5 and think it replaces Silverlight for the browser completely. Having worked with both HTML4 and Silverlight and having looked into what HTML5 brings to the table so far (it’s not done yet), I have to disagree.

First of all most people that make this statement think of Silverlight as a platform for media and games, similar to Flash. However there is a lot more to Silverlight. For one it is a lot better suited for Line Of Business (LOB) applications. Another major advantage is that everything works and looks exactly the same in different browsers and on the desktop.

Another major advantage about using Silverlight for web developemt is that you can use one programming language across different tiers, complimented by one markup language. Compare that to HTML and you immediately spot the problem. Not only do you need HTML for markup, you need CSS for styling and Javascript for client interaction. To top it all off you need some server side language to work with data. That’s four languages for something that Silverlight does with two languages. Are the decision makers of IT-world paying attention? Less is definitely more in this case.

Because of all this, and a great IDE in the Visual Studio / Blend combination, productivity in Silverlight is much higher and more business logic focused than the classic web application. This allows developers to provide more value more quickly, making more money for the companies and their customers.

But Microsoft has given up on Silverlight, right??

Well, did they? Let’s look at some facts here.

First of all this statement is all based on Microsoft communicating their support for HTML as the only true platform independent technology. People think that this is change in strategy, but in fact it’s not. Let’s face it, that is just reality. The downside on Silverlight is that there is no complete support for all platforms. I’m not only talking about Linux here. It’s also about a lot of mobile platforms. And it’s hardly realistic to expect to get that support for all these platforms. It’s not going to be cost effective to do so. In my book, this statement is stating the obvious and it would be stupid to say anything else.

But will Microsoft stop investing heavily Silverlight because of HTML5? I don’t think so. First of all HTML5 is far from complete. It will take at least another ten years to make it to the main stream in it’s full glory. Second of all HTML5 will have trouble adapting, because it is a global standard and everyone wants to have their say about it. Silverlight doesn’t have this issue.

And then there is the business side of things. Microsoft has invested so heavily in Silverlight, it would not make sense for them to stop now. They have cranked out three major releases of Silverlight in the past two years, including tooling for Visual Studio and support in Blend. They have invested in the Silverlight Toolkit. They brought Silverlight to the Windows Phone platform. And recently they introduced Visual Studio LightSwitch, which can generate complete Silverlight applications. Really, they have hundreds of millions of dollars invested in Silverlight, with great success as a result. And frankly, there is no reason for them to stop doing so, for all of the reasons mentioned before. As long as it keeps selling software for them, they’ll keep working on it.

Conclusion

So we can conclude that Microsoft will not stop pushing Silverlight forward in the foreseeable future. However, it is up to decision makers to stick with the facts and not go with the press buzz, or they will be investing in the wrong technologies, loosing a lot of money in the process. And it’s up to developers to stand by their choice of technology. There really is no need to suddenly change anything.

Just remind yourselves, what made you choose Silverlight as a platform in the first place? Exactly.

Microsoft introduces LightSwitch: Game changer or gadget showoff?

On VSLive 2010 Microsoft announced Visual Studio LightSwitch. In this article I want to take a short look at what it is, based on the information provided by Microsoft so far. Then I’ll share some of my thoughts on this with you about how this might affect us as developers.

So what is this LightSwitch thingy about?

Well, basically it is a new edition of Visual Studio 2010 that enables people to quickly build data oriented applications (are there any other types?). ‘Oh,’ I hear you say, ‘another Oracle Forms clone. That’s not going to be any good.’ Well, first of all I’d say go and take a look at the VSLive 2010 Keynote. Also there is a great introduction video here. And for more of a deep dive you can check out this video.

‘Ok, so it’s just the next Microsoft Access?’, I hear you ask. Well, not quite either. The big difference with the previously mentioned platforms is the fact that you get an application with a multi-tier architecture and you get to choose what your client is and what your data source is. You can even have multiple data sources and link them together. And if you need to take your application to the next level, there are all kinds of extensibility points in both the underlying framework and the IDE itself.

On the UI end they have figured a way to do better as well. In the IDE you don’t have a WYSIWYG editor, but a sort of a tree view so you can better see and understand what you’re doing. The generated UI is also based on templates, so you can easily update the look and feel of your application at one point. And then when you do want a WYSIWYG editor you just start up your application, click a button and your on your way.

All in all it seems that Microsoft has succeeded in building a tool attempted and failed by a lot of companies.

The holy grail of database oriented systems?

For decades a lot of companies have attempted to build a system where you would simply define a data model, press a button and presto, there is your new software. Most companies failed and some partly succeeded, but no one was able to build a system that would actually work in a broad spectrum of cases and still provide enough customizability to implement the very specific features needed in very specific applications.

Most of the time failure occurred because the software would be to generic and the resulting applications would not meet enough of the users needs. And when teams would attempt to implement code to satisfy these needs, they would either become to specific or to expensive.

But is it really that simple? I don’t think so. Most database oriented systems I’ve worked on/with always have some form of extra processing, connect to external systems in special ways or need some very specific UI. In practice I think that most application that professional developers work on will not be built with LightSwitch all of a sudden. There may be some, but not many. There will be many reasons not to build an application in LightSwitch, but the main reason will be that developers feel they might not have enough control to actually build was is required. Another important reason is that you don’t get to choose what technology is used to build your application.

‘So it’s not the holy grail, then? It must fail miserably’ I hear you say. Well, no.

LightSwitch as a game changer

LightSwitch will be a game changer. We will see less complicated systems being build in LightSwitch all over the place. For Microsoft I guess it is all about getting the smaller companies to go with them, instead of going open source. And they do give them an important advantage in allowing them to cheaply and easily build their own software and still keep that software scalable. So in that respect it is a game changer.

Another effect will not be seen until we have moved on a couple of years. Right now we see a lot of applications build in Microsoft Access, which have reached their limits and need a ‘professional’ rebuild. Usually that means bringing in a team of developers to get the job done. Changes are these applications will not be rebuild in LightSwitch. However, what will happen with the LightSwitch applications in a couple of years, when they have reached their limits? Exactly, developers will need to be extending those systems. Mind you, I do think this will greatly reduce the number of rebuilds of applications, as I expect Microsoft to come up with a good upgrade strategy for future versions of LightSwitch, making sure they are always running on the latest and greatest technology.

And finally I think we will be seeing more and more ‘secondary’ applications. What I mean by that is applications which are used to fill in some gaps in our current processes. For example, if you are building a LOB application and you have some tables in your databases, that your company needs to maintain for your customers, why would you not build the UI for that in LightSwitch? Or if you need to collect metadata for generating code, why not do that in LightSwitch? And there will be many other uses like this for LightSwitch where it can add to existing systems as well.

Conclusion

As with a lot of the new technologies that we’ve seen in the past decade, LightSwitch does provide us with another useful tool in our toolbox. It again takes away some of the repeating work we as developers do, so we can focus on what makes our software unique. This again will be a game changer, just not as a lot of people expect.

So what are your thoughts on LightSwitch? Let us know by leaving a comment.

Adventures while building a Silverlight Enterprise application part #35

In this post we’ll look into something I ran into while doing rework on our framework. It involves the order in which code gets executed when using events and such.

The situation

We’re currently in the process of getting our first deliverable out the door on time. One of the things that needed to be addressed before we can actually start testing this code properly, is to get a backlog of issues out of the way. I decided to do that myself as it gives me a better feel for what are common problems and what is the quality of work we as a team put out. Also I realized that issues may had arisen because of wrinkles in the framework architecture, where we’ve had to take shortcuts to get to our first deliverable and I wanted to be able to address these as needed.

Some architecture insight

For you to make sense of what I did to solve one of the problems I encountered, it is important to have some understanding of how our application works (at least at the client side). We host several ‘applications’ insight our Silverlight client. In the end we will have probably about six of these applications. Each application follows the same design. On the left there is navigation, consisting of a ‘menu’ and some UI to select records. Depending on the type of record the user selects and whatever authorization the user has we load modules to work with the data.

To create new records, we use a concept we call a task. This is basically a wizard like UI, which runs as a popup on top of the other UI. It consists of some of the same modules as used when the user navigates to a record, only in a different visual state. This makes for some interesting scenario’s as it is likely that there are two instances of the same module using and updating resources that only have a single instance.

One of these scenario’s is updating the navigation after creating a record through a task. As the task does not have any knowledge on what the module does, it is the modules responsibility to update navigation after creating a new record. To do that, it updates something we named ViewState, which in turn updates both the navigation and the modules that are needed to present this new record to the user.

However, creating the record is usually done in the first module of the task, before other modules can save their data (because they might need to create a relationship with the new record). This means that whenever the modules get the signal to load new data, that most of them can’t because the data is not there yet. It has everything to do with the order in which things are executed, especially because we use events to directly update our modules.

A design oversight

The fix is actually simple once you realize we had a design oversight in our ViewState. Why should you update navigation and load data into modules, when you have a popup presented to the user and the user can’t interact with the newly loaded data? It doesn’t make any sense. So the fix was to get the ViewState to hold any updates to the rest of the application if a popup is being presented and trigger the events once the popup is closed.

The first step to do that was to create a generic way to trigger the events in the ViewState. This is what I came up with:

   1: private void InvokeEvent<T>(MulticastDelegate target, T args) where T:EventArgs

   2: {

   3:     if (target != null)

   4:     {

   5:         if (PopupBridge.Bridge.IsPopupOpen)

   6:         {

   7:             EventHandlerQueueItem<EventArgs> queueItem = new EventHandlerQueueItem<EventArgs>();

   8:             queueItem.EventHandler = target;

   9:             queueItem.EventHandlerArguments = args;

  10:             _eventQueue.Enqueue(queueItem);

  11:         }

  12:         else

  13:         {

  14:             target.DynamicInvoke(new object[] { this, args });

  15:         }

  16:     }

  17: }

The InvokeEvent<T> method takes a MulticastDelegate (which represents the actual event instance) and an args argument based on T where T should always be an EventArgs or derive from it. To make this as easy to use as possible I first check to see if there is actually a valid target (which is null if no handlers are available). Next I check to see if there is any popup open. In our case there is a PopupBridge that has that information for us. Now if there is no poup open I simply call DynamicInvoke on the target and pass in a reference to the ViewState and the args argument.

If there is a popup open, though, I create a EventHandlerQueueItem. This is how that struct looks:

   1: internal struct EventHandlerQueueItem<T> where T:EventArgs

   2: {

   3:     public MulticastDelegate EventHandler;

   4:     public T EventHandlerArguments;

   5: }

As you can see it is a simple struct that has both a MulticastDelegate and can contain arguments to pass off to any handlers. Once we had that, we could now declare a Queue<T> to hold these instances. As we can’t write something like Queue<EventHandlerQueueItem<T>> I decided to use covariance and go with Queue<EventHandlerQueueItem<EventArgs>>.

Now all that remains is to trigger the events once the popup is closed. In our case the PopupBridge knows about this and triggers an event for it. The handler looks like this:

   1: void Bridge_PopupClosing(object sender, PopupClosingEventArgs e)

   2: {

   3:     if (!PopupBridge.Bridge.IsPopupOpen)

   4:     {

   5:         while (_eventQueue.Count > 0)

   6:         {

   7:             EventHandlerQueueItem<EventArgs> item = _eventQueue.Dequeue();

   8:             if (item.EventHandler != null)

   9:             {

  10:                 item.EventHandler.DynamicInvoke(new object[] { this, item.EventHandlerArguments });

  11:             }

  12:         }

  13:     }

  14: }

As there can be multiple popups open I still need to check if there is a popup open at that point. If they are all closed, we can just keep calling Dequeue and invoke the handlers until the queue is empty.

Now that we have solved this design oversight, modules that need to load data that was created in a task, will always be able to get that data from the server and everything works as expected.

I hope you find these pieces of code useful. If you have any questions or comments, please leave them below.

Adventures while building a Silverlight Enterprise application part #34

In this post we’ll look into analyzing performance in a Silverlight Line Of Business application, including multiple service layers and we come across a common problem using ADO.NET Entity Framework including a not so obvious solution.

The story

At the moment we are working on getting out a deliverable to a group of customers as soon as possible. However, one of the issues that keeps coming up is performance. A general response by a lot of developers is to just scour the web for performance tips on different technologies used in the stack and apply as many of them as they possibly can. However I find it much more useful to analyze the problem first.

Getting some data

To get a better feel for the problem, I first needed to get some numbers. When starting out with a performance issue the first question to get answered is “where is most time being spend?” This usually leads to the point where most time is being wasted as well.

At first I started out with profiling our application with Visual Studio 2010, but this wasn’t very useful to us as it didn’t show us any data on communicating with the service. I do feel compelled to link to some useful information on profiling your Silverlight application as it isn’t available through the IDE just yet and it requires you to jump through some hoops to get it to actually show you any performance data on your code. You can find a decent post by Oren Nachman on this here.

I should mention that profiling our application revealed one small performance issue with creating dynamic objects, which we use to instantiate modules in our application based on authorization and application settings. However, looking at the big picture it is hardly relevant to us, while hard to remedy it in a good way for our application.

The next step for me was to actually get data on communication with our services. As we designed our application in a way that we abstracted our service connections in seperate generic classes and we use generic service calls to get to our data, the easiest way for me was to simply record the DateTime.Now.Ticks. To keep the impact of this as low as possible on any production scenarios, I decided to write this information to the VS debug window, through Debug.WriteLine. Also to make sure this can be turned off easily to allow for other verbose debugging information, I introduced a conditional compilation symbol called DEBUGGINGPERFORMANCE and wrapped all my recording code with it. The code for me looked something like this:

   1: #if DEBUGGINGPERFORMANCE

   2:             Debug.WriteLine("BusinessLayerConnection.GetDataAsync<{0}>({1}): {2}", typeof(T).Name, id, DateTime.Now.Ticks);

   3: #endif

Running the application with this debugging code gave me some interesting data pointing me towards a specific operation, which gets all the data for a particular module at once. Within that set of modules one stood out that was taking far longer then the other modules, so I decided to investigate that further. I decided to use Fiddler to make sure none of this is actually caused by bandwidth issues. The statistics clearly showed my that the problem was with the service taking up a lot of time before actually starting to send a response.

This led me to put similar code in the webservice. Note that Debug.WriteLine doesn’t support the format approach by default and you should wrap that in string.Format. To actually make for a better performance test, I decided to build a small test tool to just make the same service calls several times in a row. This makes sure we’re working with correct numbers instead of chasing some random external factor.

As I started testing like this, I found out there where a lot of messages in the debug output stating that ‘A first change exception of type ‘ had occurred. In this case a large part of these exceptions where related to another service we use, which couldn’t access one of our databases. This was obviously an easy fix and it did improve performance considerably.

Fixing a common Entity Framework problem

After getting one exception out of the way, I found there where some more exceptions being thrown. The exact message that was displayed was: A first chance exception of type 'System.NotSupportedException' occurred in mscorlib.dll. Performance data indicated that this took roughly 3 seconds for six exceptions in each request!

I found it weird to see an exception like this. A web search didn’t return anything useful, except that it could be related to Entity Framework somehow. A forum thread pointed to this bug report about dynamic assemblies, which in itself wasn’t very useful information either.

I also found people asking questions about this message all over the internet, but without any real solutions. I turned on the System.NotSupportedException in Visual Studio, attached it to the service for debugging and ran my test again. As expected it halted in the debugger on the point where the exception was thrown. As expected it halted in mscorlib.dll, in this case in the System.Reflection.Emit.AssemblyBuilder class in the method GetManifestResourceNames(). This does confirm that it is actually caused by the bug I mentioned earlier, however it doesn’t really help us with a solution yet. The question now is, “why is someone calling this method on a dynamic assembly?”.

Browsing up the call stack makes it very clear that this code is being called from the Entity Framework. In fact it is called from the System.Data.Objects.ObjectContext constructor, but why? Taking a closer look at the call stack there is some indication that it must have something to do with initializing the metadata needed for the Entity Framework. It also reveals that the exception occurred in the System.Data.EntityClient.EntityConnection class in a method called SplitPaths. Looking at the code in that method and the state it was in as the exception occurred actually showed me what caused the problem. In the SplitPaths method there is a string[] called results (what’s in a name, right?) which contained three strings: res://*/Model.csdl, res://*/Model.ssdl and res://*/Model.mdl.

Looks familiar? That’s because these url’s are found in the default connection string for Entity Framework and they should point to the three files that make up the Entity Framework metadata. For some more documentation on that look here.

What was actually happening here was that a dynamic assembly was asked for it’s resources as the Entity Framework code was looking for a metadata file in all the loaded resources, as indicated by it’s connection string. The dynamic assembly that was loaded in this case was the binary for the WCF web service that hosts our DAL. The fix was simple. Just point to the exact assembly by it’s full name in the connection string for each metadata file and it’s fixed. This is how the metadata entry in our connection string looks now (obviously with some name changes):

   1: metadata=res://OurApp.Data, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null/OurModel.csdlres://OurApp.Data, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null/OurModel.ssdlres://OurApp.Data, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null/OurModel.msl;

I hope you found this search for this fix as useful as I did and it helps you out. Please leave me a comment if you have any questions.

30-07-2010 EDIT: Added the metadata part of the connection string to make the solution more clear. Thanks to greg-joyal for pointing that out to me.