Apr
08

Package Management for the Upcoming PC-BSD 9

Kris has written an article that discusses the new PBI format. It appears on pp 1617 of the April issue of BSD Mag and is reprinted here with permission.

The updated PBI format offers many new features such as digital signing, binary diff patching, and repository management, while preserving the original goals of format, by allowing packages to be installed and run in a self-contained manner. Lets first
take a look at how the PBI format differs from traditional package management, and then explore the changes in the upcoming 9.0 release.

The biggest difference between applications packaged in the PBI format vs traditional FreeBSD packages or RPMs is that each archive contains a complete set of libraries and dependent data required for for the target software to function. This means in effect, that a PBI is self-contained, and doesn’t require changing system packages to load the appropriate dependencies. A PBI file can be installed or removed at will, without effecting other applications or running the risk of causing breakage elsewhere in
the process. With the re-implementation of the format for the upcoming PC_BSD 9, this core concept has remained and been expanded upon.

One of the first major changes for the next PBI format has been the addition of library and file sharing between PBIs. In the previous implementation of the PBI format it was common that identical files existed between various applications. These duplicates, while necessary to provide self-contained functionality, still wasted both disk and runtime memory space. In the new format this waste of space has been greatly reduced through the usage of what has been dubbed the hash-dir. In this directory, libraries and common files are able to be shared between various PBIs through a system of hard-links. When an identical file is found in a PBI, the original will be removed and a hard-link created to the copy already in the hash-dir. After a PBI has been removed, any unneeded files left in the hash-dir are cleaned up by the pbid daemon, which monitors and maintains the integrity of the shared files.

Another new feature is the ability for administrators and PBI builders to create and manage their own repositories of PBIs. This repository system provides a number of tools for PBI distribution, release management and more. For the end user, tools are now provided which allow the browsing of PBIs in a repository, enabling auto-updates and mirror configuration. Tied into the new repository system is a new feature to digitally sign PBI files, and verify their contents. When a distributor creates a new repository, it includes an openssl public key file which is installed on the end users system. When the distributor runs a PBI build process, the resulting PBI file includes several signatures for the content archive and installation and removal scripts. During the PBI installation process, these signatures are checked to confirm that the archive has not been tampered with during transit. This key file is also used to associate a particular PBI with a parent repository for upgrade purposes, since it is possible that multiple repositories will have the same applications.

Since the very nature of self-contained packages tends to produce larger installation files, one thing which needed improvement was the updating process. In the previous implementation, updating an installed PBI would require the re-downloading of the entire installation archive. For larger applications this could be a rather time-consuming process, especially for users running over low bandwidth connections. In the typical application version update, only a small percentage of files had actually been changed, and a majority of these may simply be building time-stamps. In order to solve this problem the new PBI specification now supports updating via binary diff patches. Distributors running PBI builds can now enable options to generate small PBP (Push Button Patch) files, which are often a fraction of the size; in some cases less than 5% of the original PBI archive. When the end user begins the update of a PBI file from a repository, it automatically checks for for the presence of a PBP file, and attempts to use it, only falling back to the original archive should the process fail.

A more recent addition to the new PBI format is the ability for applications to be installed by user (non-root) accounts. Since each application is entirely self-contained, and doesn’t require changing around other installed packages, it became very easy to implement functionality for user-installation. By default the PBI format allows users apart of the systems operator group to perform installations and upgrades. This allows enhanced security in office or home situations, where users can now add/remove software on their desktop without being able to use the root account.

All these new features in the PBI format for PC-BSD 9.0 have already made it far superior to the existing legacy format. For traditional FreeBSD users though, perhaps the most important new feature is the implementation. The previous PBI incarnation was developed in QT/KDE C++, which made running on native FreeBSD cumbersome, especially on a system which had no need for X11 installed. The new format is implemented 100% in shell, and is able to run on traditional FreeBSD entirely from the command-line. The various functionality is broken up into 15+ command-line utilities with man-pages for each, which makes native FreeBSD usage a natural fit.

We’ve just taken a brief look at this reimplemented PBI format, and some of the new features it offers. This format will ship as the default for PC-BSD 9.0 and beyond and is currently available for beta testing in our PC-BSD 9-CURRENT snapshots. Once out of beta, it will be available to install on traditional FreeBSD via the ports system.

Share This Post:
  • Digg
  • Facebook
  • Twitter
  • email
  • LinkedIn
  • Slashdot

Written by dru. Posted in 9.0, pbis

Trackback from your site.

Comments (13)

  • mato
    April 14, 2011 at 2:05 pm |

    speaking of packages reminds me that i recently installed skype pbi on pc-bsd 8.2 and found out it is flawed. the thing is that it asked for root password upon installation as well as any subsequent launch and it seemed it was indeed running under root instead of user account. this makes it nearly useless in normal setup. :-(

    • admin
      April 15, 2011 at 6:56 am |

      mato » I’ll have to check into the skype package again, but in the past its needed root or was unable to take control of the mic / audio for some reason.

      • mato
        April 17, 2011 at 2:49 pm |

        pls do. i’ve been successful in installing and using skype on freebsd from ports for long time and it runs under regular user w/o any issues.

        on another note, webcamd is a bit unstable — sometimes i can see my camera in kopete, sometimes it’s not available. and i haven’t noticed any pattern. :-/

        • May 9, 2011 at 9:31 am |

          Which version of skype are you using from ports? I see that /net/skype has been expired as it has been broken for over a year.

          • mato
            May 10, 2011 at 3:38 am |

            Exactly that one. It works very well, you just need to find its tarball yourself. (Which is pretty easy via Google.)
            And AFAIR it’s the same version PBI is based on. The only issue with PBI is that it does run under root (which pretty much breaks the whole thing, at least with multi-user system). I do not know reasons for this behaviour as the port works OK.
            Hopefully Kris will look into it sooner than later. Cheers.

      • mato
        April 17, 2011 at 2:53 pm |

        and i forgot to say that skype is the main reason people i converted to using pc-bsd have to boot into windows :-(

  • Ben
    April 14, 2011 at 2:14 pm |

    The new format is implemented 100% in shell”

    Had you considered implementing it in Python? I guess doing it in shell scripts would be better, as far as reduced dependencies, but…you know you like Python (and it doesn’t bring in that many dependencies, either). ;-)

    • admin
      April 15, 2011 at 6:58 am |

      Ben » Well, two reasons I didn’t implement it in Python. First, I didn’t want the additional dependencies, just want it to run on plain FreeBSD. Second I really dislike python ;)

      • Ben
        April 15, 2011 at 7:09 am |

        Ah, well, Python isn’t for everyone, I guess. lol

  • Mustafa
    December 30, 2011 at 1:15 am |

    I know this post was here for months and I read it before, but I wanted to say WOW, you people are great, this is one of the most advanced ideas in package management, thank you all for all your efforts, may I suggest something? I think there should be the ability (only option) to update from a repository other than the PBI’s parent repository.
    Regards

  • December 30, 2011 at 7:52 am |

    Thanks!

    You can add your own repository using the Repository tab in Appcafe and can create your own repository to add using the pbi_makerepo command. See http://​wiki​.pcbsd​.org/​i​n​d​e​x​.​p​h​p​/​P​B​I​_​M​a​n​a​ger for more info about the repo commands.

  • uki
    April 28, 2012 at 8:26 am |

    A bit of an academic question. Have you considered linking statically instead? Works the same (self contained app), and takes less space (functions are stripped from the binary, only those that get used are left).

    • April 30, 2012 at 10:13 am |

      Yes, this was considered. It wasn’t used it because it wastes more space than doing intelligent library sharing. In addition, each binary in a PBI may have the same library linked into it multiple times– this would be a huge waste of space for something like QT4 apps.

Leave a comment

*

Please leave these two fields as-is:

Help the Project, Donate Today!