Moving blog to Github Pages

The blog has been pretty much inactive. I would like to start blogging a bit more again, but want to use a more powerful blogging platform that allows me to use Markdown. After some investigation, I've decided on using GitHub pages with Jekyll. For the few still subscribed to my feed, you can subscribed to the new blog located at https://cjlotz.github.io/. I'm leaving the existing blog content as is.

Messaging Plugin for Xamarin 3.0

I've just published an update to the Messaging Plugin for Xamarin that includes support for the Windows Universal Platform (thanks to @JamesMontemagno). The update introduces a breaking change whereby the default namespace for the plugin changes to Plugin.Messaging to align with the naming strategy for Xamarin plugins proposed by James Montemagno in his latest round of Xamarin plugin updates. Due to this breaking change, the version of the plugin has been bumped to v3.0.0.

Messaging Plugin for Xamarin and Windows 2.0.0

I’ve been having some fun taking part in the Xamarin Holiday contest for creating a Xamarin plugin.  My entry for the competition is a Messaging Plugin.  The Messaging plugin makes it possible to make a phone call, send a SMS or send an e-mail using the default messaging applications on the different mobile platforms.

I’ve been slowly adding new features and have just published a new version of the plugin that adds support for sending attachments as part of an e-mail.  In addition to adding attachments support I did some clean-up and refactoring on the plugin API.  Unfortunately this introduces a few breaking changes and I’ve therefore decided to make the new version a major update (2.0.0).

Here is a list of supported features for v2:

• Send SMS (supported on iOS, Android, WinPhone 8, WinPhone RT)
• Make Phone Call (supported on iOS, Android, WinPhone 8, WinPhone RT)
• Send Email (supported on iOS, Android, WinPhone 8, WinPhone RT and limited support on WinStore via mailto protocol)
• Send HTML Email (supported on iOS, Android)
• Send Email Attachments (supported on iOS, Android, WinPhone RT)

Here is the release notes for v2:

• Added support for attachments via IEmailAttachment abstraction
• Added IEmailMessage abstraction
• Breaking change: Deprecated EmailMessageRequest.  Construct IEmailMessage using EmailMessageBuilder instead.
• Breaking change: Changed IEmailTask.SendMail overload to use IEmailMessage.
• Breaking change: Deprecated Lotz.Xamarin.Messaging.Abstractions namespace. Use Lotz.Xamarin.Messaging instead.

So head over to NuGet to install the latest version.  Full documentation on how to use the API and examples for using the plugin for the different platforms can be found in the GitHub repository. 

TFS Release Notes Generator on Github

A while back I wrote an article on how to generate release notes from a TFS Work Item Query into a PDF output file. I recevied some requests for access to the full source code and not only the snippets showed in the blogpost. I've now uploaded the solution on Github. Have fun :-)

Silverlight 5 PivotViewer Localization

We’ve recently started using the Silverlight 5 PivotViewer control in On Key Express as a great way for visualizing the Work Orders showing the work list that needs to be completed/has been completed.  Here is some screenshots showing the different “cards” of information for the Work Orders at different zoom levels.

Small Card View

Large Card View

One of the great features of On Key Express is the ability for clients to translate the system into any language.  This is great feature for customers who have engineers working across the globe as they can customise the system into any language they which to support.  We supply the default English translation but the clients can even decide to change the English translation if they have certain client specific terminology they want to use.

This language feature however imposes an important restriction of any third party control that we use – we need the ability to translate any resource displayed by the control.  We make extensive use of the excellent Telerik RadControls for Silverlight.  Fortunately these controls are fully localizable by hooking up a custom ResourceManager to override the default resources being used by the Grids and other controls. 

The PivotViewer control has quite a few resources.  It ships with support for a few languages out-of-the-box through providing the resources in separate System.Windows.Controls.Pivot.resources.dll resource assemblies.  By setting the ItemCulture property you are able to use the different out-of-the-box translations shipped with the control.  However, unlike the Telerik controls there is however no easy way to hook into the control to override the resources being used.  We can create additional resources assemblies for different languages but as mentioned previously, the client decides what additional languages they want to support. 

We therefore needed a way to hook into the PivotViewer to inject our own Resource Provider that will use the client provided resource translations.  Technically we store the clients translations in a database that is periodically synchronised with the client devices.  After looking at the PivotViewer using Reflector, we confirmed that it makes use of the usual an internal static Resources class that is generated whenever you add .resx files to a project.  The Resources wrapper internally makes use of a ResourceManager that uses the current culture to access the language specific resource file.  We needed the ability to intercept the calls being made by this ResourceManager.  We also wanted the ability to still use the existing Microsoft resource assembly as the default fallback mechanism for any of the resources that we don’t want the clients to translate.  There includes the numerous exceptions and other design time resources which the clients aren’t interested in. 

With this in mind, we created the following PivotViewerResourceManager wrapper class:

Notice that the class is a simple ResourceManager wrapper around the existing Microsoft resource assembly.  When override the GetString method to allow us to first do an external lookup for the resource using our IResourceProvider interface.  If we do not have an external translation, we simply delegate to the Microsoft provided resource.  We adopted the convention of prefixing all the client provided translations for the PivotViewer with the “MSPivot” prefix as it makes it easier to identify all the translations related directly to the control.  With all of this in place we simply added a trigger to the PivotViewer XAML to load and inject our PivotViewerResourceManager when the control is loaded via the InjectResourceManager method.

Here is a sample screen shot of our ResourceManager in action with the PivotViewer.  Notice that some of the resources for which we provide translations are show in a different language whilst the rest of the control falls back to using the default MS provider resources. 

Sweet!

Automatically Generate TFS Release Notes to PDF

Background

At Pragma we've been improving on some of our deployment practices during the past few releases.  I've always been a great proponent of trying to automate as much of the deployment process as possible.  One of the aspects that we improved on recently was the ability to automatically generate the Release Notes from our Team Foundation Server repository with every daily build of our software. This automation gives us quite a few benefits:

1. Visibility into what's included with every build from the beginning of a release
2. Ability to QA the Release Notes earlier in the development cycle as part of our daily builds
3. Use of our existing TFS work items as the source for feedback, i.e. no context is lost between developers telling the technical writer what to include in the Release Notes.  The technical writer only needs to check the grammar and spelling of the TFS Work Item feedback.
4. No manual intervention required to get the bulk of the Release Notes document generated

After doing some research on the web and looking at solutions like TFSChangeLog and this CodeProject article, we decided to roll our own due to some unique requirements within our environment.  The solution turned out to work quite elegantly and I hope this blog post will provide some pointers for other folks on  how to go about implementing something similar for their own environments.

Requirements

Here is a run-down of some of our requirements:

1. We want to be able to merge some manually authored content into the Release Notes document.  Sections like What's New, Modifications and Maintenance typically contain functionality that are implemented by various TFS Work Items (Product Backlog Items, Tasks etc).  So we wanted the ability to manually author these sections and merge the content with remainder of the document that is automatically generated.
2. We want the ability to selectively filter out some Work Items from being included in the Release Notes document by running a custom TFS Query.  An example of Work Items that we want to exclude are the internal bugs that were discovered for new functionality not yet released into production. 
3. We want to use MS Word to author the manual content to make use of our existing corporate stylesheets
4. We want the output to be available as a PDF
5. We want command line support to allow us to automate the process as part of our daily builds using TeamCity
6. We want the PDF to use bookmarks to enable easy navigation between the different sections
7. We want the PDF to use our corporate branding/style

With the list of requirements,  we set out to build a solution to satisfy all of these.

Solution

Setting up a Word template to satisfy requirement #1 and #3 was a snap to do.  As we want the final output to be available as PDF, we save the Word document as a PDF once it has been updated.  To satisfy requirement #2 we created a TFS Query with the relevant query conditions to filter out unwanted Work Items. 

With the manually created PDF and the TFS Query returning the list of work items to use, we started looking at some code for generating a PDF from the Work Item contents.  We started off by creating a CommandLineOptions class to encapsulate all the different command line parameters to satisfy requirement #5:

The AppAction enumeration is used to indicate whether to run the app as a command line utility or to run the application using a GUI which allows you to manually specify the parameters to use for driving the report generation process - this is especially useful for testing purposes.  Just for reference, here is a quick view of the user interface:

The next step was to create a ReportRunner class to take these settings and generate the report in the PDF format:

Using the TFS API to programmatically query the contents of the work items was quite easy to do as illustrated through the ConnectToTfs and GenerateReleaseNoteWorkItem methods above.  With the Work Item information now processed and available in memory, the final step in the generation process was to take the list of processed work items, write them to a PDF file and finally merge the contents of this generated file with the output of the manually created PDF file.  This turned out to be quite an interesting exercise and by far the most challenging aspect of the whole solution :-)

Generating the PDF

After looking at various options for generating a PDF programmatically, we decided to use the C# open source port of the well-known Java iText library called iTextSharp.  Through reading various articles on the web (there is also this great Manning Book on the Java version), we eventually figured out how to generate, merge and create a PDF to satisfy the remainder of the requirements.

On a high level, the process for generating the PDF turned out as follows:

1. Create new blank PDF
2. Add a front page
3. Add (merge) the contents of the manual PDF into the new blank PDF whilst keeping track of the bookmarks contained within the manual document (see Requirement #6)
4. Run through the list of processed work items to add the document content for them into relevant sections in the new PDF
5. Re-bookmark the whole document to enable easy navigation throughout the whole PDF in support of requirement #6.

This high-level process was implemented in the Publish method of the ReleaseNotesPdfWriter class.


To add the front page and also include a custom footer on every page containing the build version, generation time stamp as well as page number, we created a ReleaseNotesPdfPageEvents class to implement the IPdfPageEvent interface that iTextSharp provides for executing custom logic when a PDF document is opened/closed, new paragraphs, sections are added etc.  This class is then assigned to PageEvent of the PDF writer to ensure that the custom logic executes as part of the PDF generation process (see line 16).

After adding the front page, the content of the manual PDF is merged into the new document (lines 30-35). Keeping track of the bookmarks turned out to be quite an interesting exercise and is done as part of the Merge method.

After merging the existing content, we process the in-memory list of work item information by firstly grouping the Work Items based on their resolution types into sections like "How do I", "Bug Fixes" etc. (lines 38-41) and thereafter writing these sections into the document using the WriteWorkItems method:


For every new section we add an additional bookmark into the Bookmarks collection to make sure that we have a complete list of bookmarks for all the available sections in the document (see line 14).  The final step in the generation process is to add these bookmarks into the new document using the CreateBookmarks method.

Conclusion

Here is a screenshot of an example report generated for our latest release to give you an idea of what is possible using the solution described above.  I think you'll agree that the content looks quite professional.

I trust you find the above mentioned pointers useful in creating your own solution.

Async support for running Silverlight Unit Tests

One of the great new features included with .NET 4.5 and Visual Studio 2012 is the async/await support for writing more elegant multi-threaded code in C# or Visual Basic.  Support for writing tests that make use of the async/await keywords is available for the Microsoft Test Framework as well as most of the other popular xUnit testing frameworks like NUnit.  In addition to providing support for async/await baked into the compilers for Visual Studio 2012 and .NET 4.5, Microsoft has also released the Async Targeting pack for Visual Studio 2012 that enables projects targeting .NET 4.0 or Silverlight 5 to use the Async language feature in C# and Visual Basic code.

Using it in your Silverlight 5 code is quite handy – especially when invoking web services from the client.  However, the Microsoft Test Framework for Silverlight has not been extended to support running these asynchronous tests.  However, in a recent blog post by Morten Nielsen he shows how it is possible to go about adding this support to the Silverlight Test Framework.  He provides a customized version of the Silverlight Test Framework, but also urges us to go vote for getting this added directly to the official Silverlight Toolkit. 

I know a lot of development shops like ourselves still have quite a substantial amount of Silverlight code to maintain, so please go and vote for the issue to get Microsoft to add support for it in the Silverlight Toolkit.

FxCop Standalone for VS 2012, .NET 4.5 and Portable Libraries

In our current team environment we are using the VS 2012 Professional SKU with a MSDN subscription due to the cost savings it affords us.  That implies that we miss out on features like the integrated Code Analysis.  Up till VS 2010, Microsoft released an updated edition of the Code Analysis engine with the standalone FxCop installation.  That implied that even though we didn’t get the tooling integrated into Visual Studio directly, at least we could use the standalone edition of FxCop to execute the same set of Code Analysis rules on our code base - albeit in a bit more laborious fashion.

With VS 2012 Microsoft has not yet released an update to the standalone FxCop.  Here’s the MSDN forum post where I raised the issue as to when an update will be released.  The current standalone edition also doesn’t seem to support the new Portable Library format being used to cross target different .NET platforms.  Following the advise on the forum post I’ve created an UserVoice issue to request an update.  So if you are in the same boat as to using the standalone edition, please head over to UserVoice and go vote for the issue to get it resolved.  Thanks.

PS: Btw, if you would like to run the standalone FxCop in VS, you might want to consider the FxCop Integrator.  Haven’t used it, but it seems promising although there hasn’t been an update for a while.

Practical Performance Profiling E-book

The folks at RedGate, makers of some quality .NET and SQL Server tools like Reflector, ANTS Memory and Performance Profiler, SQL Monitor (and much more) have published an excellent, free book on performance profiling.  Practical Performance Profiling by Jean-Philippe Gouigoux seems like a wonderful comprehensive guide for improving application performance by understanding performance bottlenecks from every possible angle.   Do yourself a favour and register to get your own copy soon.  Happy profiling

Silverlight 5 Upgrade Woes

We recently struggled to upgrade the UI of On Key, Pragma’s Enterprise Asset Management System (EAMS) to Silverlight 5.  We struggled with getting Silverlight 5 binaries for a lot of the open source projects (Moq, Castle.Core), but it was the lack of development tooling support in Visual Studio 2010 that really disappointed us.  I can understand that the open source projects are slack in providing support for Silverlight 5.  We solved the problem by getting the source code and creating Silverlight 5 binaries ourselves.  However our hands are cut off when it comes the development tooling in Visual Studio.  Expression Blend 5 is still only in preview edition with no word on when a final version that supports the SL 5 RTW will be provided.  Code Analysis (FxCop) and Code Metrics analysis are broken for Silverlight 5 projects.  I find it very disappointing that MS didn’t make sure that all parts of the development tooling surrounding Silverlight 5 was working by the time that Silverlight 5 went RTW.  Hopefully these issues will be resolved soon, but one sort of gets the impression that with no more momentum behind Silverlight it might take quite a while

Elementary, my dear Watson

<rant> I don’t know about any other .NET developer out there, but I’m finding the current speculation with regards to what’s going to happen with vNext of the .NET developer tool stack to be totally frustrating to say the least.  This is causing so much angst and damage in the .NET developer community.  The uncertainty about what’s going to happen with technology stacks like SL, WPF etc. and the unwillingness of MS to be open about their future is killing a lot of momentum and goodwill and just adding more fuel onto the “oh-what-is-MS-up-to-now-again” discussions.  Yes, I know, all will become crystal clear at the BUILD conference somewhere in September, but why didn’t MS keep quiet until then?  Figure out what you want to do with the different technology stacks, restructure and get enough prototyping done BEFORE starting to communicate to the developer community.  Why even mention something like “HTML 5 and JavaScript is the way to go” with the first demo preview of Windows 8.  Surely MS should have expected the reaction by developers wanting to know what the future holds for the other technology stacks?  Anyway, I hope MS gets this right and that v1 of the next, seemingly consolidated UX platform, does not set us back another 2/3 years in waiting for it to become a really usable technology stack. </rant>

So until BUILD in September we are left to act as Sherlock Holmes – looking at information surfacing via leaked e-mails and other blog posts.  Alternatively we can just let it be and hope that everything will indeed fall together in a big “ah-hah moment” come September.  I just hope that an already fatigued .NET developer community will have enough energy left to buy into whatever MS is going to preach next and that it will be “Elementary, my dear Watson.  Elementary indeed.”  Time to get some rest before September then

Pragma On Key Silverlight Localization

One of the big features that we are adding to Version 5.4 of On Key, Pragma’s Enterprise Asset Management System, is the ability to use the system in different languages.  It seems like the first language we will support in addition to English will be Brazilian Portuguese  to support customers for our Pragma Brazil service company.  So I’ve been spending some time during the past week or two looking at the various aspects of On Key that we need to localize to determine the best solutions for getting everything localized.  In this post I will cover some useful information I discovered in my research.

Pragma Software Development: Check-in Procedure

When working together in a software development team it is important to have a common definition for when something is DONE.  An important aspect of getting things DONE is a well-defined check-in procedure that all developers follow to commit something into the source control repository.  Here is the procedure that we use at Pragma for committing changes into our Team Foundation Source Control repository:

1. Code has been integrated with a recent version of the code in the TFS repository – at least a version of the TFS repository on the same day.
2. Code has been formatted according to our ReSharper Pragma Full Cleanup profile.  This automatically formats all code to follow the same formatting and layout standards.
3. Code compiles without any compiler warnings.
4. Code adheres to our coding standards - I’ll write more about this in a future post.  We try to automate the application of our coding standards (where possible) using tools like ReSharper.  Alternatively, use a code review checklist to quickly identify areas of concern. 
5. No ReSharper code analysis Error or Warning violations (i.e. green square in scrollbar). Any violation needs to be identified and agreed upon. Suggestions and Hints can be ignored if deemed unnecessary.
6. No FxCop violations. Any violation needs to be identified, agreed upon and justified using the in-code SuppressMessage attribute.
7. New functionality has been covered by tests and the code coverage has been verified using NCover.
8. Bugs have been covered by tests and verified using NCover.
9. Code adherers to the Logging strategy and the developer has inspected the usefulness of the log statements using Log4View.
10. Code adheres to the Exception handling strategy for managing exceptions.
11. Error Messages and translations have been added to the resource files.
12. The associated TFS work item (development ticket/task) has been updated to reflect the effort involved (actual hours, comments etc.)

Once the code satisfies the above mentioned criteria, the developer runs a gated check-in using the TeamCity Visual Studio Plug-in.  This ensures that all the unit and integration tests are executed by the TeamCity build server after integrating the local changes with the latest TFS repository changeset.  Only if all the tests pass will TeamCity commit the changes onto the Mainline.   The observant reader may have noticed that we don’t do an upfront code review.  As we work from home for 2-3 days a week, we conduct our code reviews on the checked-in artefacts.  I’ll write more about the code review procedure in a future blog posting. 

Lastly, for service pack development, the developer needs to consider whether the change needs to be merged across to the Mainline for the next major version and merge the changes across if required.

Pragma Software Development Tools

Last time around I mentioned I would blog about the software development tools we are using.  They say a picture is worth a thousand words, so with time to blog being a bit short in the week running up to the final release of On Key, here’s  a mind map showing the different software development tools we are using in the different software development disciplines!  Btw, if anybody knows about a decent Live Writer plug-in to insert a thumbnail, be sure to leave me a comment.

Pragma Software Development Team

Following on my intention to blog a bit about our internal development environment here at PRAGMA, let’s start with a look at the software development team, how we communicate, where we work and what Software Development Lifecycle we follow.

 

Team

The software development team has grown quite a bit during the past 2 years.  Currently we have a Software Development Manager who takes the responsibility of managing the project, prioritizing the feature set, interacting with clients, ensuring the product vision etc.  Reporting directly to the Software Development Manager is a Development, Test and Analyst lead. They are responsible for managing the 13 developers, 3 testers and 1 analyst that compromise the rest of the team.   The management function forms only a small part of their responsibility as they are all hands-on, doing full-time development, testing and analysis together with the rest of the team.  We believe in a flat structure and in the team taking collective ownership for the solution.  As Development Lead, I share the overall responsibility for the architecture with another senior developer in the team.  We also have a Project Manager/Assistant who helps with administrating the project plan and also currently plays the role of SCRUM Master.

Communication

One of the benefits of working at PRAGMA is that we get to work from home for 2 days a week.  Initially I found it very hard to adapt to not having the whole team co-located.  Especially in the early stages of the project, where I had to do a lot of mentoring, I missed the simple “luxury” of pairing with a developer to illustrate some concepts or to fledge out a design on a whiteboard or through some code.  As the project progressed and things settled in terms of architecture and skills the issue became less of a problem.  I still feel a co-located team is the most productive setup, but there is also a lot of merit in allowing people to work remotely for a day or two in the week by giving them the flexibility of not having to commute into office every day.

When working remotely keeping in touch is obviously very important.  We use Skype for IM and VOIP calls to interact between individuals and small groups of people.  I specifically mention small groups as we struggled to use Skype for calls that involved the whole team.  The call quality was not good and people would frequently drop and had to reconnect to the conference call. This might be due to bad connections from the different ISP’s that the remote workers are using so your own mileage might differ.  Fortunately for us PRAGMA makes use of Interwise Participant for web conferencing so our daily feedback session currently involves a combination of using Interwise for visuals and PRAGMA’s normal teleconference facilities for audio.  This obviously incurs a bit of cost for the remote workers but it doesn’t nearly weigh up against the other benefits like not having to commute.  As the broadband scenario in South Africa is getting better and better we re-evaluate this setup every now and then to see whether we can’t get away with only using Skype.  For simple remote pairing/debugging sessions we find Microsoft SharedView to be an excellent solution.

Office Space

Our offices are located in the Bellville Business Park in the Northern Suburbs of Cape Town so fortunately we miss out on all city traffic.  We moved into our current premises end of November 2009 and the new, modern offices are equipped with a very nice canteen and other facilities that make working at the office a pleasure.  Of course there is some good, free coffee as well   The seating inside the building is based on an open plan setup.  The software development team sit together in what is refer to as the Lab with each person having his/her own small cubicle.  This is a picture of my cubicle in the Lab.

The cubicle barriers are about 1.6 m high and allow you to work without being distracted when somebody walks by your desk.  This setup works quite nicely although there are some days that I feel the cubicle barriers should rather be taken down to create a feeling of more openness and to “increase communication” in the team. 

We have whiteboards against all the walls for design discussions and two small adjacent meeting rooms to the Lab for ad-hoc design discussions and meetings.  We also have our build monitors setup against a wall in the Lab that shows our TeamCity builds and other dashboard related statistics relevant to our ongoing development efforts. 

 

We will hopefully be replacing the two smallish monitors with one big monitor during this year to make it easier to read as we also use these monitors as an electronic SCRUM task board during our daily at-the-office SCRUM stand-up sessions.

SDLC

Talking about SCRUM, one of my first responsibilities when I joined the team way back in July 2008 was to look at adopting a more formal software development lifecycle.  I’m a firm believer in an agile development methodology and all the disciplines involved with doing agile development so we adopted SCRUM.  We went through quite a few Sprints trying out the different aspects of SCRUM to find out what works best in our environment.  After starting with 2 week Sprints we eventually settled on 3 weeks as our ideal Sprint length.  We tried Planning Poker for a few Sprints but eventually dropped this in favour of just estimating the tasks for a Sprint in Ideal hours.   We had some big debates about using Story Points versus Ideal Hours and how we should track our Velocity.  We took quite a long time to get a Prioritized Product Backlog in place and also struggled to map the backlog back into a project plan that was required due to immense pressure from a big international client.   We eventually got into some kind of SCRUM rhythm, but we still have quite a few areas where we want to improve on going forward. 

For a start I think we need to break the team into smaller groups that focus on specific functional areas of On Key.  We can then use a Scrum of Scrums meeting to discuss the areas of overlap and integration.  We also need to get better at breaking down the functionality in coarser tasks (or work items).  I feel we are breaking down our tasks into too fine grained work items and this causes unnecessary management overhead and also totally clutters the task board with too much detail.   I think a lot of this is as a result of us rewriting and expanding on an existing system, i.e. we have a very good idea of what is required, but I think we must still try and manage the work at a higher level.   This might ease the pain of integrating back into the project plan as well.   Having said all of this, we have mastered a lot of the technical disciplines (like continuous integration, TDD, automated acceptance testing etc.) required for agile development in place and we do succeed at delivering small increments of working functionality in our 3 week Sprint cycle.

Sprint

So how does a typical 3-week Sprint look taking into account that we work remotely for about half of the time?  The following 3 week roster gives a quick summary of the high-level activities involved:

Mon	Tue	Wed	Thu	Fri
At the office Individual Planning and Design Discussions	At the office Daily Scrum @ 10:00 Daily Build @ 22:00	Working remotely Daily Scrum @ 10:00 Daily Build @ 22:00	At the office Daily Scrum @ 10:00 Daily Build @ 22:00	Working remotely Daily Scrum @ 10:00 Daily Build @ 22:00
Working Remotely Daily Scrum @ 10:00 Daily Build @ 22:00	At the office Daily Scrum @ 10:00 Daily Build @ 22:00	Working remotely Daily Scrum @ 10:00 Daily Build @ 22:00	At the office Daily Scrum @ 10:00 Daily Build @ 22:00	Working remotely Daily Scrum @ 10:00 Daily Build @ 22:00
Working Remotely Daily Scrum @ 10:00 Daily Build @ 22:00	At the office Daily Scrum @ 10:00 Daily Build @ 22:00	Working remotely Daily Scrum @ 10:00 Daily Build @ 22:00	At the office Daily Scrum @ 10:00 Sprint Build @ 22:00	At the office Retrospective @ 09:00 High level Planning @ 10:00 Demo @ 13:00 Design, Training @ 14:00 Drinks @ 16:00

Well, I think that’s enough for now about the overall work environment.  In my next post I’ll look at the Development Tools that we are using.

Pragma On Key Software Development Environment

As I mentioned towards the end of last year, we delivered an important Release 3 milestone of Pragma On Key V5.  We are currently working hard to further improve the application performance and to fix the remaining bugs with the aim of going live with a big international customer 1 March 2011.  For those unfamiliar with PRAGMA and On Key [Source]:

PRAGMA is a global engineering company providing physical asset management improvement services and products to our clients. We believe that the most important value derived from our service is peace of mind. A client's CEO and his production or service team can focus on delivering their own client promise, whilst we, along with the in-house maintenance team, take care of optimising the performance and longevity of their assets over the life cycle of those assets.

Pragma On Key is our homegrown Enterprise Asset Management System (EAMS). Apart from the traditional functionalities generally expected from an EAMS, it has a variety of sophisticated modules that allow the client virtual and real-time access to asset information.

We are based in the beautiful Cape Town, South Africa and we have customers across a wide spectrum of industries in South Africa and also overseas. 

We currently support two major versions of On Key:

1. Version 4 – A Windows forms client/server application that runs on top of SQL Server as DBMS.  The application was developed using Delphi and has been in production for quite a few years already.
2. Version 5 – A new web-based solution architected as a RIA application.  It uses Silverlight as presentation technology and a layered server architecture using .NET 4.0 on top of SQL Server as DBMS.  

Both versions support interfacing with other ERP systems like SYSPRO and SAP via custom interfaces that we build.  In addition to rewriting a lot of the V4 features onto V5, we also added some really nifty new features to provide our customers with great flexibility in managing their physical assets.  In V5 we also started creating a SDK that allows third parties to build additional services on top of the core On Key application platform.   

Besides these cool features in On Key itself, we have also IMO come along way in creating a solid software development environment within PRAGMA.  The team closed to doubled in size during the last 3 years (we are now 22 people) and we simply had to mature our SDLC practices.  There are of course still a lot of areas where we can and will improve on, but I think we have a good foundation to build on for the future.

I want to start a series of blog posts where I write a bit about our software development environment - reflecting on the work environment, what tools and development practices we use and also areas that we want to improve on in future.  I also want to spend some time writing about the design of some of the core features within On Key.   I don’t quite know how all of this is going to pan out and how many posts I am going to create, but off the cuff, here is a short list of areas that I want to cover:

1. Work environment – Team Composition, Office Space, SDLC (Scrum), Typical Sprint…
2. Development Tools – Source Control, Build Server, Wiki, Work Item Management…
3. Development Practices – Check-in procedure, Code Reviews, Coding Standards, Code Analysis (Static/Dynamic), Code Generation, TDD…
4. Build Automation – Versioning, Continuous Integration, Continuous Performance Testing, Build Pipelines…
5. Testing – Automated Functional Regression Testing…
6. On Key Architecture – Client Architecture, Server Architecture, Localization, Exception Handling, Logging, Security…
7. On Key Features – Rule Engine, Validation Framework, Background Tasks, Rollout, Synchronization…

This list is quite long!  Time permitting, I’ll try and keep posting on some topic hopefully once a week.  Till later…

My 2011 Software Development Reading List

A wonderful 2011 for all!  I’ve not made many New Year resolutions, but one resolution that I do want to try and keep is to work through my library of ever growing IT books that have been gathering some dust on my desk and hard-drive.  In stead of spending time reading through blogs, I want to rather concentrate on broadening/sharpening my skills through working through some of my books.  I’ve got quite a few good books where I’ve only read some part of the book and I also want to complete these as well.  So without further ado, here is my list of IT books for 2011 that I want to read:

1. [Partially Read] Release It!: Michael T. Nygard  - Excellent advise on getting and keeping your software running in production.
2. [Partially Read] Architecting Applications for the Enterprise: Dino Esposito, Andrea Saltarello – A great resource for all budding architects and a book that I want to re-read to evaluate our current architecture against some of the best practices mentioned in here.
3. [Partially Read] Continuous Delivery: Jez Humble, David Farley – Excellent advise on using build, test and deployment automation to manage your software releases.  We’ve got quite a good deployment environment at work as I’m a big fan of continuous integration, but it is always nice to read some further insights from experts to see what areas we can still improve on.
4. The Art of Application Performance Monitoring: Ian Molyneaux – Performance monitoring is high on the agenda for our Enterprise Asset Management System (EAMS).
5. Algorithms In A Nutshell: George Heineman, Gary Pollice & Stanley Selkow – Time to learn and brush up on the Graph, Search, Path finding and Network Flow algorithms.  I’ve got a sneaky suspicion that we are missing out on some opportunities to use better algorithms within our EAMS.
6. Pragmatic Thinking and Learning: Andy Hunt – Time to figure out the best way to think about solving problems.
7. Growing Software – Proven Strategies for Managing Software Engineers: Louis Testa
8. Programming Ruby 1.9 (3rd Edition): Dave Thomas – Lots of people are raving about Ruby and I want to start learning why.
9. The Ruby Programming Language: David Flanagan & Yukihiro Masumoto

Happy reading :-)

Pragma On Key – Release 3 Milestone Reached

I’m watching the final TeamCity daily build as I type this and I’m super excited about the next major milestone that we’ve accomplished for Pragma On Key, our company’s Enterprise Asset Management System (EAMS).  It’s been 30 months of hard work to re-write and extend the system using Silverlight and .NET.  We started off as early adopters with the Silverlight Beta’s and .NET 3.5 and with the latest build we are using Silverlight 3 and .NET 4.0 as the technology platform.  We evaluated Silverlight 4 but due to some unresolved memory issues in the framework we decided to stick with Silverlight 3 for now.  We hope to be upgrading to Silverlight 4 soon here after.

I’m very proud of the way the team pulled through all the difficult times to end up with what I feel is a solid platform to build on for the future.  As always there is still a lot of things we can and will improve on going forward, but I think we have come along way since the start of the project.  From adopting a new SDLC (SCRUM) to learning a complete new technology skill set (C#; Silverlight, TDD, CI etc.) and in the process close to doubling in team size. Currently we have 2 Analysts, 5 Testers, 13 Developers (me included) and a Software Development Manager.   I can honestly say that within the 2.5 years we only had one or two isolated incidents with the team dynamics.  I think this is a great testimony to the quality of the people working here at Pragma.

We are now entering the final phases of testing the new release of On Key at a big international client who is planning to go live with the system end of Feb 2011.  So after more than 20 000 CI builds, 230 tested builds, 26 820 commits and with just more than 9000 unit and integration tests it is great to finally see the new version of On Key coming of age!  We’ve got some real nifty features included in this release and I think customers will enjoy the new web based interface as well.  I hope to be blogging a bit more about this and our internal development environment in the future.  Well done to all involved!

TransactionScope Fluent Interface

During the past 2 weeks I’ve been spending a lot of time investigating the performance of Pragma On Key, our company’s Enterprise Asset Management System (EAMS).  Part of the investigation was to resolve some issues we were encountering with deadlocks occurring for some of the longer running transactions.  We are using the TransactionScope class added in .NET 2.0 to mark blocks of code in our Application Controllers as participating in a transaction.  Out of the box, the default IsolationLevel used by new a TransactionScope instance is Serializable.  This ensures the best data integrity by preventing dirty reads, nonrepeatable reads and phantom rows.  However, all of this comes at the cost of concurrency as the range lock acquired by the DBMS prevents other transactions of updating or deleting rows in the range.   So I set off to carefully consider the locking requirements for the different transactions to make sure we are acquiring the right level of locking for the different transactions.

I’ve always found the different settings of the TransactionScope class to be rather confusing.  I always need to remind myself what the different items of the IsolationLevel and TransactionScopeOption enumerations actually mean.  There is also the tricky scenario where, when passing in a new TransactionOptions instance to set a different IsolationLevel, you need to remember to set the transaction Timeout value if you are using a custom Timeout setting in your web.config file.  So I decided to create a TransactionScopeBuilder class with a fluent interface to construct new TransactionScope instances with the idea to try and hide the complexity and also provide a more intent revealing interface of what the actual TransactionScope will do.  I’ll start by showing some examples usages of the builder and then conclude with the source code for the builder itself. 

Understanding MSIL Memory Leaks

We are using a custom RuntimeDataBuilder class that dynamically constructs a .NET type in-memory using Reflection.Emit to create simple property getters/setters for the information that we receive from a generic WCF Data Service. The service exposes a "DataSet like" structure consisting out of 1..m DataTableDefinition classes each containing 1..m DataTableColumnDefinition classes. When the information is received client side, we generate the dynamic type with its property setters/getters to improve the performance and facilitate binding on our Silverlight client. All of this works fine.

I’m currently trying to figure out how the memory management for these custom dynamic types work.  I’m thinking that we might be causing a possible memory leak if we regenerate the type. Reason why I am contemplating this is that the user can change the query parameters which may then result in more/less information coming across the wire. It therefore invalidates the previous type that I created and I want to make sure that I’m are able to free up the memory used by this previous type definition as this can happen quite frequently.  From this article on MSDN we gather that if you are using Light Weight Code Generation (LCG) the code is allocated on the managed heap which will be reclaimed by the GC when there is nothing holding a reference to it. But LCG only seems to apply to dynamic methods. My concern is for the Type with all its property getters/setters that is now not required anymore. If this is allocated on the unmanaged heap our only hope for reclaiming the memory seems to be to make sure that the type is loaded into a temporary AppDomain that we can unload when it is not required anymore.  This opens up a whole new can of worms for inter AppDomain communication which I would rather avoid.

So my question is any of my readers can shed some more light on this topic.  Is my assumption correct or is there another way of reclaiming the memory?  I also posted the question to StackOverflow if you are interested in checking out the responses over there.  Thx :-)