NuGet Tests

I. NuGet What?

Shipping software is a challenging operation, with a multitude of factors all coming together at a single moment in time: a new Release.

To test our features, we write automated tests and run them in the release pipeline. To guard our architecture, we write automated tests and run them in the build pipeline.

To ensure we conform to legal requirements, we may check with legal before using a third party integration. Or not. Or it was OK-ed by legal 7 years ago, but some NuGet package has since then changed its license type, carrying some implications along with it.. Totally missed it, and now you got an unexpected meeting request from CISO?

NuGet Test!

Sure you’ve opened Visual Studio and saw a warning: ‘This solution contains packages with vulnerabilities’. You head over to NuGet Package Manager and start looking for the culprit(s?). Boring?

NuGet Test!

Maybe you had some NuGet version mismatches while deploying to the server and your build crashed… Kind of sucks, especially that late in the release-process.. Totally not stressful.

NuGet Test!

In this article, I'll walk you through some helpful tests and checks to add to your build pipeline. By the end, you'll have a more robust release process and fewer headaches. The cost of writing these tests is very small, while the benefits can be substantial—potentially saving millions in legal fees.

II. Test Prep

Let me start off with some helper methods we will use to make our tests nice and readable. With these tools, we can automate the tasks a human would do in the NuGet Package Manager more easily:

ListSolutionProjectPaths

We get the path of every .csproj in our solution. _solutionDirectory is a private static field, set in the constructor.

private static string[] ListSolutionProjectPaths()
{
    return Directory.GetFiles(
        path: _solutionDirectory,
        searchPattern: "*.csproj",
        searchOption: SearchOption.AllDirectories
    );
}

ListNuGetPackages

We’ll use this method to gather and return basic info about the installed NuGet Packages in a given project. With the XDocument-class, we can read the XML version of our .csproj-files. Since we’re interested in the PackageReferences only, we’ll extract that info into an array of NuGetInfo, a class I made myself for keeping the code clean.

private static NuGetInfo[] ListNuGetPackages(string projectFilePath)
{
    return XDocument
        .Load(projectFilePath)
        .Descendants("PackageReference")
        .Select(packageReference=> new NuGetInfo
        {
            Project = projectFilePath.Split('\\').Last().Split(".csproj").First(),
            NuGetPackage = packageReference.Attribute("Include")?.Value,
            Version = packageReference.Attribute("Version")?.Value
        }).ToArray();
}

GetNugetResourceAsync

This method will be using the NuGet Service API to fetch metadata about packages in our projects.

private async Task<PackageMetadataResource> GetNugetResourceAsync()
{
    var repository = Repository.Factory.GetCoreV3("https://api.nuget.org/v3/index.json");

    return await repository.GetResourceAsync<PackageMetadataResource>();
}

GetNuGetMetadataAsync

This method will create a PackageIdentity the NuGet Service API can use to fetch metadata for a given NuGet package. IPackageSearchMetadata carries about every info you can find in the NuGet Package Manager and more. This is what we will use to take away human interference and automate what a developer does in the NuGet Package Manager.

private async Task<IPackageSearchMetadata> GetNuGetMetadataAsync(NuGetInfo packageReference, PackageMetadataResource resource)
{
    var packageIdentity = new PackageIdentity(
        id: packageReference.NuGetPackage,
        version: NuGetVersion.Parse(packageReference.Version)
    );

    return await resource.GetMetadataAsync(
        package: packageIdentity,
        sourceCacheContext: new SourceCacheContext(),
        log: NullLogger.Instance,
        token: default
    );
}

III. NuGet Tests

Here is the setup of our Test class:

public class NuGetTests
{
    private static string[] _projectFiles;
    private static string _solutionDirectory;

    public NuGetTests()
    {        
        _solutionDirectory = Directory.GetCurrentDirectory()
            .Split("MyProject.QA.Tests")
            .First();
        _projectFiles = ListSolutionProjectPaths();
    }
}

Now that we have our setup ready, the tests can speak for themselves.

Check Package Licenses

For each NuGet package in every project, we use the NuGet API to fetch the license Metadata. We check if they match with one of the allowed NuGet Package Licenses, and if they don’t, we collect them for the test results.

[Fact]
public async Task CheckPackageLicenses()
{
    //Arrange
    var restrictedNuGetPackages = new List<PackageLicenseInfo>();
    var allowedNugetPackageLicenses = new List<string>
    {
        "MIT",
        "Apache-2.0",
        "Microsoft"
    };

    var resource = await GetNugetResourceAsync();

    //Act
    foreach (var projectFile in _projectFiles)
    {
        foreach (var packageReference in ListNuGetPackages(projectFile))
        {
            var metadata = await GetNuGetMetadataAsync(packageReference, resource);
            var licenseMetadata = metadata.LicenseMetadata?.License ?? metadata.Authors;

            if (!allowedNugetPackageLicenses.Contains(licenseMetadata))
            {
                var nugetPackage = new PackageLicenseInfo
                {
                    NuGetPackage = packageReference.NuGetPackage,
                    Version = packageReference.Version,
                    Project = packageReference.Project,
                    License = licenseMetadata
                };

                restrictedNuGetPackages.Add(nugetPackage);
            }
        }
    }

    //Assert
    Assert.Empty(restrictedNuGetPackages);
}

Check Package Vulnerabilities

Next are the vulnerabilities. It can be smart to implement a no-vulnerability policy, but also costly in man-hours, especially if there’s a backlog to work through. With the following test, you’ll be upgrading your NuGet packages more frequent, making the workload come in smaller increments, with an increase in security as a bonus for CISO.

Next time you get the warning ‘This solution contains packages with vulnerabilities’, you just run the test and get to fixing!

[Fact]
public async Task CheckPackageVulnerabilities()
{
    // Arrange
    var vulnerableNuGetPackages = new List<PackageVulnerabilityInfo>();
    var resource = await GetNugetResourceAsync();

    // Act
    foreach (var projectFile in _projectFiles)
    {
        foreach (var packageReference in ListNuGetPackages(projectFile))
        {
            var metadata = await GetNuGetMetadataAsync(packageReference, resource);
            var vulnerabilities = metadata.Vulnerabilities ?? new List<PackageVulnerabilityMetadata>();

            if (vulnerabilities.Any())
            {
                var nugetPackage = new PackageVulnerabilityInfo
                {
                    NuGetPackage = packageReference.NuGetPackage,
                    Version = packageReference.Version,
                    Project = packageReference.Project,
                    Vulnerabilities = vulnerabilities
                };

                vulnerableNuGetPackages.Add(nugetPackage);
            }
        }
    }

    // Assert
    Assert.Empty(vulnerableNuGetPackages);
}

Check Package Version Mismatches

Another fun cause of heart failure is a production build failing because of package version mismatches. (Can I get a show of hands please?)

Even if the build succeeds, you might encounter runtime errors because the application might not be able to load the correct version of a dependency or other unexpected behavior in your application.

This method will in fact be a summary of what we want to gather from the Consolidate-tab in NuGet Package Manager. We want to check packages where multiple versions are installed over our solution projects, and if there are, gather them for the test result.

Nice bonuses: again an increase in security, more stable builds and thus smoother releases.

[Fact]
public void CheckPackageVersionMismatches()
{
    // Arrange
    var installedNuGetPackages = new List<NuGetInfo>();

    foreach (var projectFile in _projectFiles)
    {
        installedNuGetPackages.AddRange(ListNuGetPackages(projectFile));
    }

    // Act
    var packagesToConsolidate = installedNuGetPackages
        .GroupBy(package => package.NuGetPackage)
        .Where(packageGroup => packageGroup.Select(package => package.Version).Distinct().Count() > 1)
        .Select(packageToConsolidate => new
        {
            PackageName = packageToConsolidate.Key,
            Versions = packageToConsolidate.Select(package => $"{package.Project}: {package.Version}")
        }).ToList();

    // Assert
    Assert.Empty(packagesToConsolidate);
}

Other Usages

What else can we do with this toy?

Getting .NET Framework dependencies by reading await GetNuGetMetadataAsync(packageReference, resource).DependencySets.
Get a packages deprecation status by calling await GetNuGetMetadataAsync(packageReference, resource).GetDeprecationMetadataAsync().

Play NuGet-police: Lets say layers with Dto’s are not allowed to have NuGet packages installed. A simple test can enforce this policy and keep your architecture clean.

 [Fact]
 public void DtoLayers_NoInstalledNuGetPackages()
 {
     //Arrange & Act
     var dtoProjectFiles = Directory.GetFiles(_solutionDirectory, "*Dto.csproj", SearchOption.AllDirectories);
     var dtoLayerNugetPackages = new List<NuGetInfo>();

     foreach (var dtoProjectFile in dtoProjectFiles)
     {
         dtoLayerNugetPackages.AddRange(ListNuGetPackages(dtoProjectFile).ToList());
     }

     //Assert
     Assert.Empty(dtoLayerNugetPackages);
 }

IV. Take-aways

In the fast-paced world of software development, ensuring the stability, security, and legal compliance of your projects is paramount. By incorporating automated tests for NuGet packages, you can mitigate risks, streamline your release process, and sleep better at night knowing that your software is in good shape.

As developers, our goal is to deliver high-quality software that meets user expectations and stands the architectural test of time. By integrating these NuGet tests into your build pipeline, you take a proactive approach to safeguard your projects and ensure a smoother development journey, securing the stability of your software—all with little to no cost.

NuGet Tests

Table of contents

I. NuGet What?

II. Test Prep

ListSolutionProjectPaths

ListNuGetPackages

GetNugetResourceAsync

GetNuGetMetadataAsync

III. NuGet Tests

Check Package Licenses

Check Package Vulnerabilities

Check Package Version Mismatches

Other Usages

IV. Take-aways