rename repsys to mgarepo, RepSys to MgaRepo, and update docs and examples for Mageia

1 files changed, 360 insertions, 0 deletions

diff --git a/README.BINREPO b/README.BINREPO
new file mode 100644
index 0000000..78b951d
--- /dev/null
+++ b/README.BINREPO
@@ -0,0 +1,360 @@
+================================
+The detached binaries repository
+================================
+
+.. contents::
+
+A brief description
+===================
+
+Ideally, all binaries from packages sources (ie. all the binary files inside
+SOURCES/) will be placed in another subversion repository. This repository
+is called "tarballs repository", "binaries repository" or just "binrepo".
+It will contain mostly the same directory structure of the main repository,
+but instead of having SOURCES and SPECS, it will only have a SOURCES
+directory. Every copy/move operation should happen in both repositories.
+
+In order to allow deceasing binaries from older distributions, each stable
+distro will have its own subversion repository for binary files.  mgarepo
+knows how to access these binrepos by checking which URL defined in the
+"[binrepo]" section of the configuration file matches the path-part of the
+repository being accessed.  (see open issues)
+
+The package changelogs will be generated from SVN commit logs in the main
+"plaintext" repository ("txtrepo" for short) only.  Old changelogs will be
+preserved, as even empty revisions are preserved in the binaries-filtering
+conversion.
+
+
+Mapping repositories states
+---------------------------
+
+In order to allow the use of `mgarepo {getsrpm,co} -r REV`, mgarepo will have
+to use a reference in the text repo which will be used to know in what
+state was the binrepo when a binary was uploaded.
+
+We cannot use direct revision number mapping through properties/files/etc
+mainly because we may have multiple binaries repositories, and eventually
+they can be filtered for reducing space, thus can't ensure revisions will
+survive.  Thus another mechanism which relies on dates instead of revisions
+numbers is needed.
+
+When a binary is uploaded to the binrepo, the file `sha1.lst` is updated to
+have the files's hash and commited in the main text repo. This file will be
+used as the reference when the user uses -r REV on mgarepo. mgarepo will
+checkout the package in the main text repo with -r REV and then will use
+the "Last Changed Date" of `sha1.lst` to checkout the binrepo part. Thus,
+`sha1.lst` should be always commited to the main text repository *after* the
+corresponding binary files have been commited to the binrepo. Hooks in the
+main repository may be used to try to enforce this, by checking if the files
+changed in `sha1.lst` are already commited in the corresponding binrepo.
+
+Computation of `sha1.lst` is unlikely to be an issue:
+
+- it should not happen too often for any given package
+- it takes[0] less than 10s to sha1sum all SOURCES of openoffice.org-3.1-1mdv2010.0.src.rpm
+- it probably takes way less than the time to upload the file into the repository
+- it can be computed in parallel to the binrepo commit, and probably finish
+  before that, thus ready by the time `sha1.lst` should be commited
+- users don't need to verify the SHA1s "everytime", but the build system
+  does, thus Repsys can default to not verify and avoid wasting users' time
+
+The use of `sha1.lst` has the valuable property of tying the state of the main
+repository and the binrepo.  With it, at getsrpm time of a package
+submission we can verify the SHA1 of the SOURCES-bin, and be sure that
+either the package will be built with the expected state, or early fail the
+build. It also allows for verifying binaries without trusting the binrepo,
+which may be useful if we consider using an unversioned plain filesystem
+storage in the future (for old distros or whatever), or at "client side",
+which maintainers may find useful.
+
+[0]: In a single core AMD Athlon(tm) 3800+ (2400Mhz)
+
+Mapping of revisions using SVN properties
+-----------------------------------------
+
+Alternatively to using the above "sha1.lst scheme", the revision mapping
+between the main repository and a binrepo could be done using subversion
+properties.  This could be done by making every commit to binrepos also
+cause a corresponding commit in the main text repository to happen, which
+would update a property recording the current date.  That is, a subversion
+property in the main text repository would be kept, such that for any given
+main repository revision, the corresponding state of the binrepos is
+obtainable (using the registered date).
+
+This would be "more transparent", as it can be maintened simply by using
+subversion hooks, without user intervention.  OTOH, as every time the user
+commits to a binrepo this would result in a commit in the main repository,
+it would require the user to "svn up" the directories from there before
+commiting, after every binrepo commit.  Also, this might result in a big
+number of "bogus" commits to the main repository, which could be seen as log
+pollution, and may potentially increase space usage etc..
+
+Why a new repository without the tarballs
+==========================================
+
+- the current svn repository is too large, hard to manage
+- big binary files (in general, "tarballs") history is of little value in
+  the distro development, we care much more about our specs, patches,
+  configurations, etc.; nonetheless, those big files we don't care much for
+  take the most resources and make backups and restoration in case of
+  failure very expensive, much more so than the more valuable data
+- there is no easy way to strip undesired tarballs without recreating the
+  whole repository
+- fedora and ubuntu have separated repositories, so we must have it too!
+
+Numbers
+-------
+
+Current repository is +390000 revisions and ~340Gb big, while the bzip2ed
+dumps backup for it takes about a bit more than half that size (FIXME:
+estimative, can't check in the backup server right now).  Current txtrepo
+with the same number of revisions is ~180Gb big, takes about 2-3 days to be
+imported, while the gzipped full dump backup for it currently takes ~1.2Gb. 
+Initial binrepo for Cooker (only `current/` packages' branches) took ~28Gb
+in disk, gzipped full dump for it takes ~25Gb, took about 5h30m to be
+populated from the current in use repository ("oldrepo").
+
+
+Drawbacks of this layout
+=========================
+
+- (always) everything that changes the single-repository usage increases the chance
+  of failure and make things more complicated.
+- subversion can't be used alone as easily as the current scheme allows
+- copying binaries between distro branches may not be "svn-cheap" anymore
+  (unless they're in the same binrepo)
+- ...
+
+
+Open issues
+============
+
+Multiple binrepos dont allow us to have one permanent URL
+---------------------------------------------------------
+
+We would have to update the configuration files from all the users in order
+to add a new stable repository. spuk suggests to use properties in the main
+text repo that would point to the right repository locations.
+
+How to handle failures when operating on more repositores?
+----------------------------------------------------------
+
+binrepos should replicate the structure of the main text repo. What we
+should do if the markrelease succeeds in the binrepo, but fails in the main
+text repo?
+
+R: Markrelease must be done first in the txtrepo. If it fails there "we're
+in trouble" (though currently, we just miss it[0]).  When the markrelease is
+done in the txtrepo, we can do markrelease in the binrepo using '-r {DATE}',
+using the markrelease date in the txtrepo as '{DATE}'.
+
+[0] We should add transaction support for markrelease. The transaction could
+be stored out of the packages SVN (another SVN, a DB, a txt file, etc.), and
+would work like:
+
+0. mark beginning of markrelease, early failing the package build if it fails
+1. do markrelease
+2. mark sucessful end of markrelease
+   or mark failed markrelease, so we can replay it later
+
+
+Interesting use cases (first phase)
+===================================
+
+mgarepo co 1/mutt
+---------------------
+
+- mgarepo checkouts
+  http://svn.mageia.org/svn/packages/updates/1/mutt/current to the
+  mutt directory
+
+- mgarepo checkouts
+  http://svn.mageia.org/svn/binrepo/updates/1/mutt/current/SOURCES
+  into mutt/SOURCES-bin
+
+- creates symlinks for all files found in SOURCES-bin/ into ../SOURCES/
+
+  (rpm doesn't handle symlinks, this allows us to have explicit links and
+   proper src.rpm generates by rpmbuild)
+
+In case the path doesn't exist in the binrepo it will not fail, as we may
+have not imported all packages or the repository is not prepared to work on
+this model, etc.
+
+markrelease of a package
+------------------------
+
+::
+
+   $ mgarepo markrelease 
+
+- will copy current/ to releases/VERSION/RELEASE, as usual
+
+- will copy current/ to releases/, on the binrepo too
+
+Optionally, markrelease could create revprops indicating which is the
+revision of current/ on the binrepo that represents the tarballs that are
+being tagged.
+
+
+Use cases to be implemented after the first phase
+=================================================
+
+upgrading to a newer version of the package
+-------------------------------------------
+
+::
+
+  $ cd bla/SOURCES/
+  $ wget http://prdownloads.sourceforge.net/bla/bla-1.6.tar.bz2
+  $ mgarepo upload bla-1.6.0.tar.bz2
+
+- mgarepo notices this is a tarball (checking filename and/or file size)
+
+- mgarepo will move the file to SOURCES-bin/, create the symlink, and svn-add
+  it to the working copy
+
+  $ # the user updates the spec
+
+  $ mgarepo rm SOURCES/bla-1.5.1.tar.bz2
+
+- it will remove the symlink and run svn rm on
+  SOURCES-bin/bla-1.6.0.tar.bz2::
+
+  $ cd ../ # package top dir
+  $ mgarepo ci
+
+- mgarepo will commit the new tarball on SOURCES-bin/ and then on the rest
+  of the working copy
+
+mgarepo sync would perform these steps too.
+
+importing a package
+-------------------
+
+  $ mgarepo putsrpm mypkg.src.rpm
+
+- mgarepo will open the src.rpm
+
+- will look for tarballs inside SOURCES/ and import them to
+  http://svn.mageia.org/svn/binrepo/cauldron/mypkg/current/SOURCES/
+
+- will move the tarballs out of SOURCES and import the remaining files to
+  http://svn.mageia.org/svn/packages/cauldron/mypkg/current/
+
+- will do whatever else putsrpm already does
+
+TODO
+=====
+
+First phase
+-----------
+
+- upload
+- markrelease
+- putsrpm
+- getsrpm
+
+
+Second phase
+------------
+
+- up
+- sync
+
+Rejected or postponed ideas
+===========================
+
+Use of a plain filesystem storage for the tarballs
+--------------------------------------------------
+
+This was planned, then rejected. It becomes too complicated when thinking
+about markrelease, and mapping SVN revisions in the main repository to
+binaries versions in the "tarballs storage", basically requiring
+implementing VCS-like features on top of filesystem.  Would also require
+implementing another authentication and access scheme.  The main feature
+would be ease of removing old binaries, which isn't much of a point because
+we don't know precisely what and when we want to remove, so may end up not
+removing much files anyway.
+
+Use of a plain unversioned filesystem storage for the tarballs
+--------------------------------------------------------------
+
+Different than the previous one, this would mean not relying at all on
+binary files history keeping.  Structure could be something simple like::
+
+  packages/${pkg:0:1}/$pkg/$tarball
+
+This alternative does not suffice for Cooker, nor for supported distros, for
+which we want history.  It could, however, at some point be used for "very
+old" distros, for which we may have lost interest in keeping *binaries*
+history (package history will kept "forever" in the main SVN repository). 
+Alternatively, "resetting" an SVN binrepo (i.e.  recreate the repository) to
+contain only the latest tarballs would probably take about the same amount
+of space, anyway...
+
+Open tarballs repository
+------------------------
+
+This idea is not really rejected. It does not go against splitting txtrepo
+and binrepo, but rather complement this idea, where the
+open-tarballs-repository would take the place of the binrepo.  The txtrepo
+would still be used +- the same way.  This repository could be used
+selectively, for packages where it makes sense, while most packages could be
+kept "closed", still as tarballs.
+
+Use of externals for more seamless Subversion usage
+---------------------------------------------------
+
+This idea is not discarded, but it just provides easiness. OTOH, it makes
+things more complicated:
+
+- markrelease: externals would have to be updated in order to make it point
+  to the tagged version in the binrepo, otherwise changes in
+  current@binrepo would change older releases;
+- branching whole distro: even though subversion now supports "relative
+  externals", we would have to update the URLs for *every* package on the
+  distro, as the path to reach the binrepo spans the local distribution
+  directory;
+- keeping externals up-to-date (as stated above and below)
+- authentication and access control: only markrelease action done by the
+  build system should be allowed to change externals (so what about importing
+  new packages?)
+- just a convenience, we don't need and shouldn't rely on externals for
+  running the build system, while most people will use the repositories via
+  Repsys, so why spend time to implement and keep it?
+- "svn co" works transparently, cool, but "svn co -r N" does not, otherwise
+  every change in the binrepo would require svn:externals to be updated in
+  the respective package;
+- it does not solve the problem of creating and handling symlinks between
+  SOURCES and SOURCES-bin.
+
+Keeping svn:externals updated for every package has almost the same cost of
+keeping the `sha1.lst` updated, with the difference that in the latter we
+would not have to update every package when creating distro branches.
+
+Use of "external" xdelta to save space on binaries
+--------------------------------------------------
+
+But how? First idea is this could be done by defining a protocol and
+assuming repository manipulation with mgarepo (for ease).  Repsys could
+xdelta tarballs and add it to SVN with a special filename, then use it when
+checking out.  Would require a policy/algorithm on when to ditch old whole
+binaries, too (i.e.  hopefully wouldn't need to be handled manually by the
+maintainer).  Also, this is something complemental to splitting the
+repository, so we may do it later, for binrepos.
+
+
+The Future
+==========
+
+- Open tarballs repositories
+
+  - suited for GIT, maybe multi-VCS
+  - incremental move
+  - not everything will be suited for this, must handle all cases or be
+    optional
+
+- Xdelta
+