Pondering on distributed computing in .NET

After working on a highly scalable application for almost a year now, using Silverlight, WCF, Entity Framework and Sql Server and after reading and replying to numerous questions on these topics and seeing the struggle developers are having with these new ways of doing things, I started thinking about how this could be made easier.

What I figured was that I don't really care about what part of my software runs where, which is basically a 'cloud' way of looking at things. So why not do everything in the cloud then? Well, there are lots of reasons not to, including costs and customer trust, but that's not what this post is about. It boils down to us not wanting to do everything in the cloud just yet.

That thought however, led to something which is not very new, but hasn't really been very popular either. I guess object messaging describes it for what it is. Basically what I'm saying is that objects may or may not know each other, but any object can send out messages to other objects, it being send directly or through a message dispatcher. Some (but maybe not all) objects would be able to recieve messages.
I wanted to figure out why it is not very popular today and instead of just reading hundreds of webpages, trying to find out why, I thought a cool way of ganing this understanding would be to simply design and build my own messaging framework.

An object can be anywhere
The first thing to do is to make it clear to myself what the concept is going to be. My first statement is that an object can be anywhere. It can be on the same thread, in a different thread, on the same machine or on a different machine, it really shoudn't matter. All I want to do is tell the framework to send message X to object Y and it should be taken care off.

.NET == .NET == .NET
What I mean by this is that as a Microsoft fanboy I obviously wanted to build this in .NET (C#). I also want this to work on a wide variaty of process types, wheter it is a command line application, a Silverlight client, a service hosted in IIS or in Azure, or an NT Service, shouldn't matter. It should all connect implicitly.

Challenges
This does bring along some technical challenges. For example a Silverlight client can connect to a service perfictly but connecting to a Silverlight client in that same fasion is not going to work, so recieving messages there is going to be a challenge, especially if keeping it transparent comes into play.

Dealing with concurrency is another big thing that comes into play. All of a sudden everything that's not in my object is asynchronous, which can be a challenge. This brings a whole new way of developing objects. Whenever a message is recieved, do I need to stop work and check if it is relevant for what I'm doing right now, or should I just ignore it untill I have time to deal with it? How do I keep track of what I've send out in relation to what comes back in? These are all questions that all of a sudden need answering.

I'll be looking into this stuff in the near future, writing some code and hopefully sharing it with you. Please let me know your thoughts on the topic.

Adventures while building a Silverlight Enterprise application part #23

Today we look into how collations in Sql Server bugged me while trying to get code generation to work on a database different from what it was originally build on. We also look at how I solved the problem (which is with some cool C# and WPF code).

The problem context
Lets take a look at the big picture first, so we all are on the same page as to why I ran into this problem. In the application we are building we have multiple databases to store our LOB data in. Depending on someones authorization, it's possible he or she has access to one or more of these databases (or sometimes only subsets of these databases). Because of this, we needed some central data store to hold information on security, the complete installation and on where what LOB database is and what data is in it. We call this data store the Repository and stated that any data we need to go across LOB database boundaries, should be in the Repository.

So far, so good. In the early stages of the project we build a WCF service, based on Entity Framework, that allows us to do any data operations on the LOB databases in a generic way. To achieve this we build a code generator to generate all the business classes we needed. The code generation process we use for the service is based on the fact that we use a automatically generated EF model. In other words, the EF model is a direct depiction of the SQL Server data model. Because of this fact we can extract metadata from SQL Server to feed our code generation process. To do this, our database guys build a script that extracts the metadata from SQL Server (using the system views) and insert it into a separate database which we use to power our code generation.

Up to this point, still no problems. Now what we wanted to do, was copy all the code we've build to access the LOB databases and from that build a WCF service to access our Repository and because of that I needed to generate business classes from the model as it is inside our Repository database. Now normally we create and update our database models through a Sql Server 2005 project in Visual Studio 2008 and do schema compares.
However, data is added to this metadata by several people and in that case we tend to use a backup to distribute this data, which was exactly what I did when preparing to add the metadata from our Repository.

The probem
After restoring the metadata database is when the problems started. I needed to alter the stored procedure written to extract the metadata from a database to change the source database name to make it access the repository database. As soon as I ran the alter procedure statement to update the stored procedure, I got several collation conflict errors. It turns out that a backup I got from the Repository database was created in a different collation, from the one I normally use. To be more specific the Repository database was created with Latin1_General_CI_AS (my local default) where as the metadata database was created with SQL_Latin1_General_CP1_CI_AS.

To solve this issue I would need to go through each column one by one and change the collation. Because the only data accessed from the Repository is in system views and you can't (and don't want to) change their collations, I had to change the collations on the metadata database. I didn't feel much for doing this by hand and because I foresee this happening more often, I figured I might aswell write a small tool to handle this for me. Here is a screen shot of what it looks like:
Basically it allows you to type in a Sql Instance name, after which it retrieves the database list from that instance and also it retrieves all available collations to fill the two comboboxes. As soon as you select a database and at least a source collation, you can then click the Find Columns button to retrieve any columns with the source collation. After you've selected a target collation and unchecked whatever columns you do not want to change, you can then click the Change Collations button and it will trigger an alter table / alter column query to try and do that for you.

I guess what gives this tool it's flair is the use of some queries directly on ADO. Here is a snippet of code with these queries:

private const string GetAllDatabasesSqlCommand = "select name from sys.databases";
private const string GetAllCollationsSqlCommand = "select name from ::fn_helpcollations()";
private const string GetAllColumnNamesSqlCommand = "select o.name, c.name, t.name, c.max_length, c.is_nullable"
+ " from sys.columns c"
+ " left join sys.objects o on c.object_id=o.object_id"
+ " left join sys.types t on c.system_type_id = t.system_type_id"
+ " where c.collation_name=@collation_name";

The first query retrieves all database names. This obviously only works when connected to the master database.
The second query returns all collation names. This can be done on any database (it should always return the same list).
The final query finds all columns that use a specific collation. The result set includes the table name, the column name, the type name, the max length and whether or not a column can contain NULL. This is all the information needed to generate an alter table alter column statement which changes only the collation.

Something I never used before was the SqlConnectionStringBuilder class to dynamically create my connection string. Here is the code I used for that:

SqlConnectionStringBuilder connectionStringBuilder = new SqlConnectionStringBuilder();
connectionStringBuilder.DataSource = hostnameTextBox.Text;
connectionStringBuilder.InitialCatalog = MasterDatabaseName;
connectionStringBuilder.IntegratedSecurity = true;

As you can see, using this class is really straight forward. Another thing I never used before was the ChangeDatabase method on the SqlConnection class. This came in handy as I could simply keep one connection throughout the flow of the application and switch databases quickly.

I've uploaded the code here for your viewing pleasure. Note that this tool was slammed together in a hurry so most of the code is in code behind for the main window and most of it isn't very well written, however there are some nice concepts in there as well

I hoped you enjoyed yourself again. I know I have.

Adventures while building a Silverlight Enterprise application part #22

Yesterday I needed to check the available style keys in our main app.xaml file and see which are no longer needed. As there currently are 66 style keys in that file and it's growing, I didn't feel much for taking each key and searching through our source code to find out. Time to build a small tool. This article describes how I build this tool.

Requirements
The tool needs to be able to search a directory tree for files with a certain extensions (.xaml*) for a pattern or literal string. Before it does this it also needs to be able to open a .xaml file and retrieve any style elements so it can then read their keys.

To achieve this, two classes are needed. One class will read a .xaml file and get all keys from style elements and the other class will search trough the file system for files containing these keys.

Building it
I'll spare the obvious details and dive right into the highlights. To read the keys from style elements in basically any xml document, I used LinqToXml. Here is the code I used:

private void LoadStyleKeysFromDocument()
{
XNamespace winFxNamespace = "http://schemas.microsoft.com/winfx/2006/xaml";
XName keyAttributeName = winFxNamespace + "Key";

var result = from node in _document.Descendants()
where node.Name.LocalName.Equals("Style")
select node;

var distinctResult = result.Distinct();

StyleKeys.Clear();
foreach (XElement styleElement in distinctResult)
{
StyleKeys.Add(styleElement.Attributes(keyAttributeName).First().Value);
}
}

The first two lines make an XName object that is needed to include the xml namespace when retrieving the x:Key from the element. Note that this works independently from the prefix (x) as it was assigned in the document. This means that this code will still work if someone would decide to change the prefix on this namespace.

Next, a Linq query is used to retrieve any nodes in the document that have the name Style. The query is followed by a statement to make sure I only get unique results.

Finally I fill the StyleKeys collection with any key attributes value found inside an element in the query result.

Searching for a particular pattern in the file system is done in the following method:

public void Search(string pattern, string rootFolder, string fileFilter)
{
// Get all files matching the filter
string[] fileNames = Directory.GetFiles(rootFolder, fileFilter, SearchOption.AllDirectories);
// For each file
foreach (string fileName in fileNames)
{
// Open file
string fileData = File.ReadAllText(fileName);
// Match pattern
MatchCollection matches = Regex.Matches(fileData, pattern);
// Register count
PatternSearchResultEntry resultEntry = newPatternSearchResultEntry()
{
FileName = fileName,
HitCount = matches.Count,
Pattern = pattern
};
Results.Add(resultEntry);
}
}

As you can see, the first line gets all the filenames that are anywhere in the directory hierarchy below the supplied root folder.
Looping through the filenames, I simply load all the text from each file and use the Regex class to count the number of hits. By doing so, this code is also very useful to find hit counts for other patterns.
All the results are added to a collection of a struct called PatternSearchResultEntry.

So thats the business end of things. Obviously we need a user interface of some sort.
I chose a WPF interface, because I like data binding.
To retrieve user input for the style file and the folder to look in, I build a class called BindableString, which contains a Name and a Value and implements the INotifyPropertyChanged interface. It allows me to create instances of these and bind them to my UI. This way I have a central point to access this information without having to worry about updates, etc..

To do the actual work I wrote the following Click event for a button:

private void analyseStyleUsageButton_Click(object sender, RoutedEventArgs e)
{
XamlStyleKeyReader reader = newXamlStyleKeyReader();
reader.ReadXamlFile(_stylesFilePath.Value);

PatternSearch patternSearch = newPatternSearch();
foreach (string styleKey in reader.StyleKeys)
{
patternSearch.Search(styleKey, _searchRootDirectory.Value, new string[] { "*.xaml" });
}

CollectionView view = (CollectionView)CollectionViewSource.GetDefaultView(
patternSearch.Results);
if (view.CanGroup)
{
view.GroupDescriptions.Add(new PropertyGroupDescription("Pattern"));
}
analyseStyleUsageDataGrid.ItemsSource = view.Groups;
}

It basically instantiates the XamlStyleKeyReader class and loads the style file in it. Next it instantiates the PatternSearch class and kicks of a search for each style key available in the XamlStyleKeyReader.

The code after that groups the results based on the search pattern. The reason I did it this way is because it is not very transparent to bind to the result of a group in Linq. Binding to this is easy once you know how. As you can see the items source for the datagrid that displays my results, is actually the collection of groups.
This collection is declared as having objects, which isn't very helpful, however diving into the API documentation reviels that this collection contains instances of the CollectionViewGroup class. From that class I need the name (obviously) and a hit count, which of course it doesn't have.
To get a hit count I bound to the Items property from the group, which contains all the items that belong to that group and then I use a value converter to get the total hit count for that group.

I've uploaded the complete source for this tool here.

Be aware that this tool is far from finished. I would like to save the last settings and have some progress indication, which means moving the search code to it's own thread. Styling of the UI can be improved, etc., etc.

I do hope you find this code useful and you've learned something along the way.

Why you want code reviews

Lately there has been some discussion in our team on whether or not code reviews are needed and even practical and why you should want them in your development team. This may sound like obvious knowledge, but unfortunately it appears that some developers have an aversion against code reviews, whether it being their code being reviewed or it being them reviewing someone else's code. In this article we'll look into some of the reasons that makes developers feel this way and also we look into some of the benefits of having code reviews. As we go through all this, we will also give some attention at how to prevent bad experiences with code reviews.

Why not?
As I mentioned in the introduction, there is still a fairly large group of developers who have an aversion against code reviews. A lot of the time, this is because they feel that they did the best job they could on the code they have build, given the circumstances and they don't need some coworker grinding their hard labor to peaces and giving them a hard time.

If that's the case, something is obviously wrong. If this is feeling is based on previous experience, then they have experienced a bad code review by some reviewer who clearly missed the point.
Others may not want to be at that end of the table where they have to tell a coworker they did a bad job. Again, if you feel like this, you missed the point.
It is NEVER the goal to find something to be able to accuse the developer of bad work.

A third reason why a developer may not want code reviews is that they think it will be boring to do a review. If that's you, you guessed it, you've missed the point as well.

The goals of a code review
So what are we trying to accomplish trough a code review? There are several very important goals one could target:

1. Code quality
2. Code consistency
3. Architecture consistency
4. Awareness
5. Knowledge sharing

Code quality
This may seem like the obvious first reason and causes the most aversion. Sure a developer tries to do the best job he or she can, however we are all human so we make mistakes. The goal here is not to bang on someones bad peaces of code, but to prevent bugs from happening. Bugs cause a lot of work for a software company. If they get catched by the testing department, they have to generate a case file for it and communicate that back to the developers, who in turn have to do rework and system test that. It then gets handed back to testing and they have to retest.
If a bug is discovered by a user, things get even worse. They will file a case with the support desk, who has to triage it and then it has to be planned for a release and generates a whole bunch of work. So in the end it is in everyones best interest to prevent bugs from happening, which is exactly what the first goal of a code review is all about.

Code consistency
To keep code maintainable it is imperative that chosen solutions are consistent with each other. However it is hard to know everything about a large code base that keeps expanding and changing while a team of developers is working on it. By having code reviews in place, a large part of inconsistency can be catched and fixed before the code actually gets to the testing department.

Architecture consistency
From experience I know it is hard to keep the architecture that was planned consistent throughout the system. Developers often tend to do things their way, especially when they either have difficulties understanding the architecture or how to solve a particular problem in the architecture, or because they run into time constraints.
To make sure that architecture at least stays consistent throughout the system, it is important to look at how certain functionality is built into the system. If a developer did something that affects the consistency, but brings a good argument for doing it that way, it may result in changes in the overall architecture because then changes are the architecture wasn't fitting the application. Again this is important to the whole team as some others may end up feeling the same pains the first developer encountered if it doesn't get addressed.

Awareness
This may sound like a very generic term, so let me explain. Because we work in a rapidly changing field there is no way to keep up with everything that's going on so at times we may miss certain information which would have helped us solving a particular problem a lot easier or more efficient or prevent some problem in the future. To keep us more up to speed a reviewer may spot some of these issues and point out other possible solutions to a problem of which we might not have been aware.
Looking at this from a reviewers point of view you might actually see some features you where not aware of.

Knowledge sharing
This is a more obvious benefit. Not only does the developer get a change learn from the reviewer, but it also works the other way around. And this doesn't only apply to technical knowledge, it also goes for specific knowledge about the application that's being reviewed, which is an important business benefit.

Tips on code reviews
To make code reviews be a pleasant experience both the developer and the review need to keep an open mind. That doesn't mean you can't defend something you think is good work, or you can't say it when you see something you feel should be improved.

As a developer look at this as an opportunity to showcase your code to one of your peers. Reviewers, ask questions. They allow the developer to explain choices he or she made in the process. Both parties shouldn't be pushing something without a good explanation. If you can't come to an agreement on an issue, involve a third party you both trust and stick to that decision. Also don't take it personal, opinions can differ and there are more ways then one to reach a goal.

To take full advantage of code reviews, there should be a group of reviewers, who should be experienced developers in general and in the technology used. Also make sure the right reviewers are pared with the right developers. It's likely that a young but bright developer could review a more senior developer and technically this may make sense. However in the 'food chain' this might cause problems, so it's better to avoid this. On the other end of the spectrum a young developer with less experience on the technology used might be intimidated by a review from the most senior developers on the team. This might lead to the young developer not speaking his or her mind and just following the senior developers. Although this might get the junior developer a long way, it will not teach him or her to have a mind of their own.
For educational purposes it would be best to have a medior developer review a junior developer and have senior developers review medior developers. This way junior developers will be less intimidated and medior developers can learn the review skills from their senior peers.

Conclusion
I agree that there is a lot to think about when implementing code reviews, however I do feel it is worth the effort. A team doing good code reviews will get great benefit from them. Consider it and if you feel like it, please share your experiences with us.

Adventures while building a Silverlight Enterprise application part #21

I came back from a short vacation today and read this post from Tim Heuer. I then realized that in all twenty parts I've written on our adventure of building a Silverlight Enterprise application, I've actually never elaborated about why and how we chose Silverlight and how we (or at least I) learned to use Silverlight technology. So now seems as good a time as any.

Let's be a lazy author and just follow the questions that Tim put up there.

Decision resources
Obviously when you go about to start a project you have to decide on what you are going to use and why. If you are looking at Silverlight, what factors into your decision?

In our case, one of the most important topics was to move from a client-server based solution with a (very) fat client to a multi-tier solution with a web based client. The first choice was to go with DotNetNuke as some experience was already available, however, as you may very well know, this uses regular ASP.NET Ajax like solutions, which we found very limiting in UI perspective. At the time Silverlight 2 was just about to come out of the Beta stage and we felt we should at least take a good hard look at it. We needed to have technology that was relatively easy to implement and yet have enough flexibility to make for a very useful UI.

So our major decision factors where:

1. UI possibilities / user friendliness
   
2. Development speed
3. Flexibility

Why did your company choose to adopt Silverlight (or choose not to)? Was there another technology that was chosen to be better? Why/why not?

We choose to go with Silverlight because it checked the boxes on the above list. We felt it was easier to build a good user friendly UI with Silverlight, in comparison with ASP.NET / Ajax. Also we experienced in our prototyping / prove of concept face, that building a UI with Silverlight is taking less time as well, because of the use of XAML and C# only. No HTML or JavaScript code made live a lot easier for us.

Another elementary aspect of Silverlight proved valuable to us. It was the fact that you can embed controls inside other controls and work with templating. In combination with declarative databinding this became an important topic for us during decision making.

Again we where comparing to DotNetNuke, which had some advantages over Silverlight as well. One of the things that DotNetNuke had which was considered an advantage, is the fact that you could than access the database directly from your client side code. In hindsight I'm glad that we can't do this with Silverlight and just have all the logic in a service layer. This proved to be much more flexible for a lot of scenarios we already ran into.

What is the most important thing in deciding if Silverlight is right? Feature set? Existing technologies? Rapid development? Other reasons?

At the time, Silverlight 2 was not exactly existing technology as it was just coming out of Beta. This was a concern for us, but the fact that this was Silverlight 2 and that there was already an active and growing community, combined with the fact that Microsoft had plans for updates layed out already, pulled us across. The feature set was definitely important. The way databinding is implemented is important to us, as we are building a data rich application. Also the controls available at the time as default, but also trough the Silverlight Toolkit and the fact that third party companies literally jumped at Silverlight 2 helped easing the choice even further.

Rapid development was obviously something that needed prove as no real world examples where available at the time. We build prototypes of several scales and found that it was indeed fast enough to build something with Silverlight 2, even without experience with the technology.

Learning resources
On learning – how do you best learn? Do you prefer “atomic” samples? These are the ones that you can just pop in and figure out a task-based situation (i.e. how do I open a file in Silverlight). Or do you prefer more of a “lesson plan” approach to things? This would be a series based on a task (i.e. Build a Media Player in Silverlight).

For me personally a lesson plan approach was new. I did part of the digg API sample series, but after several steps I lost interest, mainly because some of the topics where not of interest at the time and some of the topics where just to easy for me. This makes me lean towards atomic samples, altough as a starting place I can see why people like this lesson plan thing. Later on atomic samples are obviously the way to go. They have the advantage of showing only the bare minimum of implementation without the burden of having to go trough previous steps to understand code that was already there.

On medium – in either types of these learning paths, what is your preference? Video? Written step-by-step guides? Labs?
When you are completely new to a topic, watching a video is great. It takes away some of the effort of having to follow along with some article and just being showed how things work. This is also great for some more indept topics, with less code and more explanation.

If looking for a reference, or if I need to actually start coding on something, I do prefer written material, as it is a lot easier to just scan trough and look for that bid of code, or that bid of explanation you needed, instead of plowing trough twenty minutes of video trying to find that five second shot of the code you need.

I've tried labs in the past and I found that they are not my thing. I tend to get bored very quickly because usually the level of instructions is making things to easy. Also setting up to do a lab feels like a lot of work most of the time.

On topics – what are the top 3 topics you expect when learning a new technology? How do you on-ramp yourself when you know nothing about it? Do you expect to learn the tools first? Or jump right in to data access?

This is actually a very tough question to answer. It realy depends on what a technology looks like. When looking at Silverlight I felt that I already knew enough about the most important tool: Visual Studio 2008. Of course Blend became part of the toolset I use and over time I did watch videos and read tutorials on Blend to learn some tricks here and there. So for me, while learning Silverlight, diving right into important topics was very helpful. I do feel it is important to include tips on how to use tools, where this is important to the topic at hand.

For example if you are talking about databinding and data access, it is handy to demonstrate on how to do this in Blend including using sample data. If something like this becomes to elaborate, at least make sure you point out to the public that this is a handy feature they should know about and tell them where they can find more information about it.

Other notes
Something that bugged me for quite some time when getting started with Silverlight was the number of resources that where available for Silverlight 2 Beta 1 and 2. While being a great help to win over management to go with Silverlight, this made live a lot tougher when trying to find useful resources for Silverlight 2 RTM as a lot of times the important bits would not work in the RTM.

Something else that I found is that, altough silverlight.net is a great resource when learning Silverlight, it is a bit of a mess. Videos are not categorized in a way that makes them easy to find and there is not a proper search function on the site. Also I think it would be very helpful to have a search engine that can search for Silverlight content for your specific version only. So if I'm working on Silverlight 3, I might not be very interested on Silverlight 1.1 and 2 content. Or at times I might not care if it's Silverlight 2 or 3 content, because they are likely to be compatible any way.

As a final note I would like to encourage anyone who hasn't already done this, please go to the article of Tim Heuer and comment. Or you can comment on this article as well.