Posted by: Dennis | November 23, 2009

Application Library Cache and dynamically downloaded xap files

I had a problem with Silverlight and dynamically loading xap files… It simply didn’t work with the “Application library cache” feature, which meant that either you had to put ALL of your dependencies in the “shell”.

This is ofcourse very much against the whole point of the “Application Library Cache”, we end up with a behemoth of an assembly. The point is that we want the start using the application instantly, thus download to be as small as possible.

Let me illustrate the point with an example:

We have a shell xap (myshell.xap), which just loads the start page and login page. And when it starts up it is supposed to load the second xap (hopefully before the user ever knows that it needed to)

And then we have another xap (myapp.xap) which is a full feature RIA application using MEF.

And then we have a third xap (myreport.xap) which is also a full feature RIA application, MEF, AND it also needs a repoting third party.

You can now choose your poison in how to set this up using the standard methods. Either we can make myshell.xap include the ria, mef, and the reporting lib. This means the size of your initial download explodes (probably 1MB instead of 10kb).

The way that people then usually solve this is that they make a 3rd XAP, which then gets loaded before the myapp and myreport xaps. This 3rd xap then includes all of the assemblies as above. The problem with this, is that we we needed to use myapp only, we still have to pay for downloading the reporting modules.

Microsoft thought it would be great in Silverlight 3 to include a feature called “Application Library Cache” which is supposed to solve this problem. Which it does… As long as you only have 1 xap in your application (but possibly several applications).

So if we use the “Application Library cache” feature in the above scenario we end up with 3 XAP files and a zip file for each of the dependencies. The point was then when I loaded myapp.xap, it should automatically load the dependencies of the XAP file and also fetch those. The problem is just that Microsoft implemented it in the loader, which means this functionality is not available inside Silverlight.

So I set out to solve that problem 🙂 If you don’t care about all the details, then just skip to the bottom and copy the code…

First, go and read these two: How to use Library Cache in the first place and How to dynamically load a xap file.

A good read? You might have noticed that in the first place, there as a new component in the AppManifest.xml called ExtensionPart and that the second didn’t do anything about that.

So the first thing to do was actually put this code into the reading of the XML:

And with those two things in place, the rest of the code is rather trivial.

                List newdependencies = new List();
                reader.ReadToFollowing("ExtensionPart");
                do
                {
                    var uri = new Uri(reader.GetAttribute("Source"), UriKind.RelativeOrAbsolute);
                    lock (_dependencies)
                        if (_dependencies.Add(uri))
                            newdependencies.Add(uri);
                } while (reader.ReadToNextSibling("ExtensionPart"));

First of all we keep track of the dependencies we have already loaded from other xap files, there is no need to download things we already know is loaded. Unfortunately I cannot see if that particular assembly is loaded, simply because that information is not in the xml, only the link to its zip file. The second big problem is that once we actually load these zip files, there is no standard way of actually loading them in Silverlight. We can only load files from a zip file, where we ALREADY know the name. But these zip files were not packacked by us, and thus we cannot know what files are in the ZIP.

To get around this, we implement the absolutely minimal parsing of ZIP files, to read out the filenames. See GetFileName in the code.

/*
Copyright (c) 2009, Dennis Haney 
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
    * Redistributions of source code must retain the above copyright
      notice, this list of conditions and the following disclaimer.
    * Redistributions in binary form must reproduce the above copyright
      notice, this list of conditions and the following disclaimer in the
      documentation and/or other materials provided with the distribution.
    * Neither the name of the  nor the
      names of its contributors may be used to endorse or promote products
      derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY Dennis Haney ''AS IS'' AND ANY
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL Dennis Haney BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.IO;
using System.Linq;
using System.Net;
using System.Reflection;
using System.Text;
using System.Threading;
using System.Windows;
using System.Windows.Resources;
using System.Xml;

namespace Utilities
{
    /// 
    /// A downloader of XAP files, that is capable of also 
    /// fetching the "Reduce XAP size by using application library cache" dependencies.
    /// 
    public class XapLoader
    {
        private readonly List _webClients = new List();
        private readonly List> _webClientsProgress = new List>();
        private readonly List> _assemblies = new List>();
        private int _downloadcount;
        private Exception _firstError;
        private volatile bool _wasCancelled;
        private readonly HashSet _dependencies = new HashSet();

        /// 
        /// Occurs when the download of all files and dependencies are completed
        /// 
        public event Action> DownloadCompleted;

        /// 
        /// Occurs when the amount downloaded changes. 
        /// A little quirky since it cannot know the full size until we actually have downloaded everything.
        /// 
        public event Action DownloadProgressChanged;

        /// 
        /// Initiate downloads of the given xap files, and all of their dependencies.
        /// Can be called multiple times, but might get multiple callbacks if downloads completed between the calls.
        /// 
        public void StartDownloads(params Uri[] uris)
        {
            StartDownloads(uris.AsEnumerable());
        }


        /// 
        /// Initiate downloads of the given xap files, and all of their dependencies
        /// Can be called multiple times, but might get multiple callbacks if downloads completed between the calls.
        /// 
        public void StartDownloads(IEnumerable uris)
        {
            _wasCancelled = false;
            foreach (var uri in uris)
            {
                WebClient wc = new WebClient();
                wc.OpenReadCompleted += FetchCompleted;
                wc.DownloadProgressChanged += ProgressChanged;
                Interlocked.Increment(ref _downloadcount);
                lock (_webClients)
                    _webClients.Add(wc);
                wc.OpenReadAsync(uri);
            }
        }

        private void ProgressChanged(object sender, System.Net.DownloadProgressChangedEventArgs e)
        {
            Action evt = DownloadProgressChanged;
            if (evt == null)
                return;
            long bytesReceived = 0;
            long totalBytesToReceive = 0;
            lock (_webClientsProgress)
            {
                while (_webClientsProgress.Count < _webClients.Count)
                    _webClientsProgress.Add(new KeyValuePair(0, 0));
                int idx = _webClients.IndexOf((WebClient) sender);
                _webClientsProgress[idx] = new KeyValuePair(e.BytesReceived, e.TotalBytesToReceive);
                foreach (var pair in _webClientsProgress)
                {
                    bytesReceived += pair.Key;
                    totalBytesToReceive += pair.Value;
                }
            }
            evt(this, new DownloadProgressChangedEventArgs(bytesReceived, totalBytesToReceive));
            
        }

        public class DownloadProgressChangedEventArgs : ProgressChangedEventArgs
        {
            public DownloadProgressChangedEventArgs(long bytesReceived, long totalBytesToReceive)
                : base((int) (bytesReceived / totalBytesToReceive), null)
            {
                BytesReceived = bytesReceived;
                TotalBytesToReceive = totalBytesToReceive;
            }

            public long BytesReceived { get; private set; }
            public long TotalBytesToReceive { get; private set; }
        }        /// 
        /// Called when one of the downloads are done
        /// 
        private void FetchCompleted(object sender, OpenReadCompletedEventArgs e)
        {
            if (_wasCancelled) return;
            if (e.Error != null || e.Cancelled)
            {
                _firstError = e.Error;
                CancelAsync();
                return;
            }
            if (!_wasCancelled)
                LoadPackagedAssemblies(e.Result);

            int left = Interlocked.Decrement(ref _downloadcount);
            if (left > 0)
                return;
            //Since we added these as we got them, the dependencies are actually last, 
            //so reverse the order, so that depencies are loaded first
            _assemblies.Reverse(); 
            OnDownloadComplete(_wasCancelled);
        }

        private void OnDownloadComplete(bool wasCancelled)
        {
            IEnumerable assms = null;
            if (!wasCancelled)
                assms = from kvp in _assemblies select kvp.Key.Load(kvp.Value);
            _assemblies.Clear();
            _webClients.Clear();
            _webClientsProgress.Clear();
            Action> evt = DownloadCompleted;
            if (evt != null)
                evt(new AsyncCompletedEventArgs(_firstError, wasCancelled, null), assms);
        }

        /// 
        /// Cancel all pending downloads. Not threadsafe with calls to StartDownloads
        /// 
        public void CancelAsync()
        {
            _wasCancelled = true;
            lock (_webClients)
            {
                foreach (var wc in _webClients.Where(wc => wc.IsBusy))
                    wc.CancelAsync();
            }
            OnDownloadComplete(true);
        }

        /// 
        /// Load all dlls from zip files and xap files, for the xap files also initiate the download of any non-included dependencies
        /// 
        private void LoadPackagedAssemblies(Stream packageStream)
        {
            StreamResourceInfo packageStreamInfo = new StreamResourceInfo(packageStream, null);
            StreamResourceInfo manifestStreamInfo = Application.GetResourceStream(packageStreamInfo, new Uri("AppManifest.xaml", UriKind.Relative));
            if (manifestStreamInfo == null) //Zip file with DLLs only
            {
                foreach (var filename in GetFileNames(packageStream))
                    Add(packageStreamInfo, filename);
                return;
            }

            using (XmlReader reader = XmlReader.Create(manifestStreamInfo.Stream))
            {
                reader.ReadToFollowing("AssemblyPart");
                do
                {
                    string source = reader.GetAttribute("Source");
                    Add(packageStreamInfo, source);
                } while (reader.ReadToNextSibling("AssemblyPart"));

                //Unfortunately the way MS did this, they didn't bother writing what assemblies are actually in those links,
                //so we are forced to fetch the files even if they turn out to already be loaded
                List newdependencies = new List();
                reader.ReadToFollowing("ExtensionPart");
                do
                {
                    var uri = new Uri(reader.GetAttribute("Source"), UriKind.RelativeOrAbsolute);
                    lock (_dependencies)
                        if (_dependencies.Add(uri))
                            newdependencies.Add(uri);
                } while (reader.ReadToNextSibling("ExtensionPart"));

                if (!_wasCancelled)
                    StartDownloads(newdependencies);
            }
        }

        private void Add(StreamResourceInfo packageStreamInfo, string source)
        {
            Stream stream = Application.GetResourceStream(packageStreamInfo, new Uri(source, UriKind.Relative)).Stream;
            var assemblyPart = new AssemblyPart { Source = source };
            lock (_assemblies) //We dont load them here, so that we can load them in the right order at the end
                _assemblies.Add(new KeyValuePair(assemblyPart, stream));
        }

        /// 
        /// This really ougth to be in the silverlight library
        /// 
        private static IEnumerable GetFileNames(Stream stream)
        {
            stream.Seek(0, SeekOrigin.Begin); //rewind
            var ret = new List();
            var archiveStream = new BinaryReader(stream);
            while (true)
            {
                string file = GetFileName(archiveStream);
                if (file == null) break;
                ret.Add(file);
            }
            stream.Seek(0, SeekOrigin.Begin); //rewind
            return ret;
        }

        private static string GetFileName(BinaryReader reader)
        {
            // http://www.pkware.com/documents/casestudies/APPNOTE.TXT
            var headerSignature = reader.ReadInt32();  // local file header signature     4 bytes  (0x04034b50)
            if (headerSignature != 0x04034b50)
                return null; // Not a zip file
            reader.ReadInt16();                        // version needed to extract       2 bytes
            reader.ReadInt16();                        // general purpose bit flag        2 bytes
            reader.ReadInt16();                        // compression method              2 bytes
            reader.ReadInt16();                        // last mod file time              2 bytes
            reader.ReadInt16();                        // last mod file date              2 bytes
            reader.ReadInt32();                        // crc-32                          4 bytes 
            int compressedsize = reader.ReadInt32();   // compressed size                 4 bytes
            reader.ReadInt32();                        // uncompressed size               4 bytes
            short filenamelength = reader.ReadInt16();   // file name length                2 bytes
            short extrafieldlength = reader.ReadInt16(); // extra field length              2 bytes
            byte[] fn = reader.ReadBytes(filenamelength); // file name                    (variable size)
            string filename = Encoding.UTF8.GetString(fn, 0, filenamelength);
            //And then make sure to skip the actual data, so that we can loop over it
            reader.BaseStream.Seek(compressedsize + extrafieldlength, SeekOrigin.Current);

            return filename;
        }

    }
}
Advertisements

Categories

%d bloggers like this: