end of the move

1 files changed, 594 insertions, 0 deletions

diff --git a/doc/howto_bcd.tex b/doc/howto_bcd.tex
new file mode 100644
index 0000000..5cff1dc
--- /dev/null
+++ b/doc/howto_bcd.tex
@@ -0,0 +1,594 @@
+\documentclass[oneside,a4paper,10pt]{article}
+\usepackage{times}
+\usepackage{graphicx}
+\usepackage[T1]{fontenc}
+\usepackage[english]{babel}
+\usepackage{fullpage}
+
+\begin{document}
+
+
+\title{HOW TO use BCD}
+\author{Antoine Ginies}
+\begin{center}
+ \includegraphics{./mandriva-logo.jpg}
+ % mandriva-logo.jpg: 950x239 pixel, 72dpi, 33.51x8.43 cm, bb=0 0 950 239
+\end{center}
+\maketitle
+
+\tableofcontents
+
+\section{Summary}
+
+BCD try to be a simple tool to build ISO using \textbf{urpmi} to create a mini 
+chroot of the future system, and \textbf{urpmq} command to calculate dependencies
+of packages on the future DVD/CD.
+
+\section{The environment}
+
+\begin{flushleft}
+BCD is available in Mandriva subversion repository:
+http://svn.mandriva.com/svn/soft/build\_system/bcd
+\end{flushleft}
+
+\begin{flushleft}
+Some release of BCD has been packaged, but you should use the svn version, because this tool is under development, and really unstable....
+All previous XML configuration's file are compatibles with previous version of BCD. 
+\end{flushleft}
+
+\begin{flushleft}
+  Create a user on the system:
+  \begin{verbatim}
+adduser plop
+  \end{verbatim}  
+\end{flushleft}
+
+\begin{flushleft}
+  You must create a directory to store all \textbf{urpmq} queries. All queries done by \textbf{urpmq} per package
+are stored in this directory + product-name-release-arch. To speed up the process of creating ISO,
+next time you run BCD, it will check for each package in input list if there is a file wich contains the result
+of the urpmq queries, if BCD found one then it will use it, else it will redo an
+\textbf{urpmq --requires-recursives --sources package\_name} and store the result in a new file.
+So if you have update your repository, you should clean this directory. If you have update some packages,
+you should remove file relative to those packages, to redo the \textbf{urpmq} queries otherwise BCD will use
+the old result of the \textbf{urpmq} queries, and this can lead to error if deps are no more the same. Moreover
+if you change the tag \textbf{<nosuggests></nosuggests>} to yes or no, you must erase all
+your previous queries.
+
+  \begin{verbatim}
+mkdir -p /var/lib/bcd
+  \end{verbatim} 
+\end{flushleft}
+
+\begin{flushleft}
+  Give you user RW on this directory:
+  \begin{verbatim}
+chown plop.users /var/lib/bcd
+  \end{verbatim} 
+\end{flushleft}
+
+\begin{flushleft}
+  Give some \textbf{sudoers} right to your user:
+  \begin{verbatim}
+vi /etc/sudoers
+plop ALL=(root) NOPASSWD:/bin/mount, /bin/umount, \
+        /usr/sbin/chroot, \
+        /usr/sbin/urpmi, \
+        /usr/sbin/urpmi.addmedia, \
+        /usr/sbin/urpmi.removemedia, \
+        /bin/rm
+  \end{verbatim}
+\end{flushleft}
+
+\begin{flushleft}
+  Checkout the source of BCD:
+  \begin{verbatim}
+svn co http://svn.mandriva.com/svn/soft/build_system/bcd
+  \end{verbatim} 
+\end{flushleft}
+
+\begin{flushleft}
+  Prepare your distribution repository, you should have
+  an access to it, or a local copy.
+  \begin{verbatim}
+[plop@isobel /mnt/BIG/dis/official] 04:04:28pm
+$ ls
+2008.0/  2008.1/  2009.0/  2009.1/  2010.0/  iso/
+$ ls -r 2010.0
+x86_64/  SRPMS/  i586/
+  \end{verbatim} 
+\end{flushleft}
+
+\begin{flushleft}
+  You need to install some tools and deps to be able to use bcd:
+  \begin{verbatim}
+urpmi perl-File-Copy-Recursive sudo urpmi perl-Parallel-ForkManager \
+ cdrkit-genisoimage cdrkit-isotools syslinux \
+ gfxboot mandriva-gfxboot-theme drakxtools-backend \
+ mandriva-Free-theme rpmtools
+  \end{verbatim} 
+\end{flushleft}
+
+\begin{flushleft}
+  Now your system is ready to build ISO.
+\end{flushleft}
+
+\newpage
+\section{configuration file}
+
+You need to have an XML configuration file. You should use
+an example provided in this package and adapt it to your configuration file.
+
+\input{conf.xml.tex}
+
+\newpage
+\section{architecture of a Mandriva DVD}
+
+A Mandriva CD/DVD contains:
+\begin{itemize}
+ \item \textbf{autorun.inf} file and \textbf{dosutils} directory used under Windows OS to popup a dialog box
+ \item \textbf{i586} or \textbf{x86\_64} directory wich contains the installer and packages and other stuff
+\end{itemize}
+
+In the \textbf{i586} or the \textbf{x86\_64} directory:
+\begin{itemize}
+ \item \textbf{doc}: full doc of howto install a Mandriva linux system
+ \item \textbf{install}
+  \begin{itemize}
+    \item \textbf{extra/advertising}: all images used in the slideshow while installing
+    \item \textbf{images}: file used to boot the system
+    \item \textbf{stage2}: mdkinst.sqfs is the stage2 of the installer, rescue.sqfs is the rescue system. Those
+    two files are compressed in squashfs.
+  \end{itemize}
+ \item \textbf{isolinux}: contains file used to boot the CD/DVD, and all gfxboot stuff
+ \item \textbf{media}
+ \item \textbf{main}: a media with packages
+ \item \textbf{contrib}: another one with packages
+ \item \textbf{media\_info}
+    \begin{itemize}
+  \item \textbf{compssUsers.pl}: describe category to display in "select package to install" step
+  \item \textbf{file-deps}: Use the given file to know which file dependencies are used by other media (man \textbf{genhdlist2})
+  \item \textbf{hdlists}: old wayt to describe media on the DVD/CD
+  \item \textbf{media.cfg}: describe the repository of the DVD/CD
+  \item \textbf{rpmsrate}: a package is selected if its category is selected and depending of its priority (5=high, 1=very low)
+    \end{itemize}
+\end{itemize}
+
+\newpage
+\section{Step by step process}
+
+\begin{itemize}
+  \item clean all previous build
+  \item create all needed directory
+  \item isolinux step
+    \begin{itemize}
+      \item erase all previous isolinux directory in the build dir
+      \item copy the \textbf{isolinux} dir in the build dir
+      \item add entry if specidifed in the configuration file (like HDT)
+      \item switch theme for all images in \textbf{isolinux/altX} directory
+    \end{itemize}
+  \item stage1 and stage2
+    \begin{itemize}
+      \item copy all files from the \textit{install} dir to the build dir
+      \item update all \textbf{advertising}
+      \item copy \textbf{dosutils} tools in the build dir
+      \item copy the doc
+      \item copy the patch file if present
+    \end{itemize}
+  \item media step
+    \begin{itemize}
+      \item list all media configured, and report missing list file
+      \item create media directory in the build dir
+      \item copy all pubkey for each media and report missing one
+      \item create the \textbf{media/media\_info/media.cfg} file
+      \item copy the file-deps
+      \item copy the \textbf{rpmsrate}
+      \item copy \textbf{compssUsers.pl}
+      \item create a basic chroot of the futur Distribution (to be able to run urpmq)
+      \item mount loop distribution repository and external media in chroot
+      \item generate a list of rpm to copy and dependencies based on list files
+      \item concatenate all files
+      \item try to define the size of the DVD/CD (not yet implemented)
+      \item link or copy packages on the build dir
+      \item try to solve all doble issue of packages
+      \item remove all packages listed in exclude list
+      \item use \textbf{gendistrib} to create hdlists and synthesys
+      \item launch \textbf{rpmcheck} to check missing deps conflicts and other failure
+      \item use the rpmcheck's package list to solve missing deps using urpmq
+      \item re-link or copy packages on the build dir
+      \item re-try to solve all doble issue of packages
+      \item re-remove all packages listed in exclude list
+      \item re-launch \textbf{gendistrib} to create hdlists and synthesys
+      \item re-launch \textbf{rpmcheck} to check missing deps conflicts and other failure
+      \item umount all mountloop disrectory (distribution repository and extra media)
+      \item remove hdlist if no more needed (Mandriva version > 2010.0)
+      \item show the size of the DVD (not yet implemented)
+    \end{itemize}
+  \item ISO step
+    \begin{itemize}
+      \item create the \textbf{product.id} file
+      \item copy text files (INSTALL, README etc...)
+      \item create the idx
+      \item create the ISO
+      \item \textbf{isohybrid} the ISO
+      \item create the md5 and sha1
+      \item create an html report web page
+    \end{itemize}
+\end{itemize}
+
+\newpage
+\section{list files}
+
+\begin{itemize}
+ \item input list: list of rpm needed on the CD. Those list should contains the name
+or the provides of a package. You may also use a CAT\_XXX from the rpmsrate file, with the weight parameter.
+\end{itemize}
+
+\begin{verbatim}
+ie: of lists/input_main
+ INSTALL 1
+ CAT_X 1
+ CAT_GNOME 1
+ CAT_AUDIO 1
+ aumix
+ task-lxde
+\end{verbatim} 
+
+\begin{itemize}
+ \item exclude list: list of unwanted packages on the CD. Be carreful to do not exclude mandatory packages
+or you DVD/CD will be not usable. Rpmcheck will report any bug in hdlists at the end of
+the 'media' step process. Packages listed in exclude list will be removed from the
+DVD/CD at the end of the build process.
+\end{itemize}
+
+\begin{itemize}
+ \item extra-filter: list of RPM you want to compare to keep the latest one. Usefull for kernel and dkms release,
+or if you are mixing release and udpates media.
+\end{itemize}
+
+\newpage
+\section{BCD options}
+
+\begin{itemize}
+ \item First arg must be the XML configuration file
+  \item Second arg must be
+  \begin{itemize}
+    \item stagex: update the stage1/stage2 in builddir
+    \item isolinux: update the isolinux in builddir
+    \item clean: clean the builddir and isodir
+    \item media: create the list of wanted packages, create all media in builddir, link/copy packages in builddir/media,
+create the media.cfg, and generate the hdtlist/synthesis
+    \item gendistrib: create the hdlists/synthesis in builddir (depends on the based\_on tag)
+    \item rpmcheck: check the hdlist in the buildir
+    \item list: list all media and list of input/exclude (from the xml conf file)
+    \item mediarepo: list all medias availables in the repositery
+    \item checkrepo: rpmcheck main hdlists in the repositery
+    \item all: run all stage in order to get an ISO
+    \item md5: create MD5 sum of the ISO
+    \item resign: resign all rpms with a password and a gpg file
+    \item doble: check doble rpm in the media build\_dir
+    \item kernel: compare kernel and keep the latest one
+    \item nodoble: don't check doble rpm in the media build\_dir
+    \item info: parse the configuration file (usefull to check the XML syntax)
+    \item clean: clean previous urpmq log file
+    \item html: generate a log report web page
+    \item iso: create the final ISO from the builddir
+    \item verbose: be more verbose
+  \end{itemize}
+  \item extra arg:
+    \begin{itemize}
+    \item noiso: don't build the iso (md5, isohybrid) in case of 'all' process
+    \item copy: copy rpm instead of link them from the repositery
+    \item verbose: verbose mode to report more info in the output
+    \end{itemize}
+\end{itemize}
+
+\section{Examples of use}
+
+\begin{flushleft}
+the folowing example will do all step to create an ISO:
+  \begin{verbatim}
+bcd mdv-20091.xml all
+  \end{verbatim}
+\end{flushleft}
+
+\begin{flushleft}
+the following example will create the build dir without creating ISO:
+  \begin{verbatim}
+bcd.pl mdv-20091.xml all noiso
+  \end{verbatim}
+\end{flushleft}
+
+\begin{flushleft}
+same as above but copy packages instead of link them from the repositery:
+  \begin{verbatim}
+bcd.pl mdv-20091.xml all noiso copy
+  \end{verbatim}
+\end{flushleft}
+
+\begin{flushleft}
+the following will update the isolinux in the build directory:
+  \begin{verbatim}
+bcd.pl mdv-20091.xml isolinux
+  \end{verbatim}
+\end{flushleft}
+
+\begin{flushleft}
+to check the main hdlist in the repositery with rpmcheck
+  \begin{verbatim}
+bcd.pl mdv-20091.xml checkrepo
+  \end{verbatim}
+\end{flushleft}
+
+\begin{flushleft}
+clean the builddir and isodir
+  \begin{verbatim}
+bcd.pl mdv-20091.xml clean
+  \end{verbatim}
+\end{flushleft}
+
+\newpage
+\section{XtreemOS example}
+
+\subsection{introduction}
+
+XtreemOS is an European project (http://xtreemos.eu/) to build and promote a Linux-based Operating System
+to Support Virtual Organizations for Next Generation Grids. This project was one of the platform used
+to develop and test BCD.
+
+XtreemOS GNU/Linux distribution is based on a Mandriva 2009.0 with updates, and and
+external repository xtreemos, wich contains sub-media release, updates and testing.
+We will explain the full configuration file for x864\_64 arch to help you understand the way 
+to create one for your need.
+
+\subsection{the xos64.xml file}
+
+The name of the product will be XtreemOS, we want to do only 5 fork of \textbf{urpmq} queries, this product is based on
+a Mandriva 2009.0 product, and we want to build a DVD for x86\_64 architecture.
+The repository of the distribution is available at /mnt/BIG/dis directory, and i want to
+create my build dir and iso dir in the \textbf{/home/xtreemos/build\_bcd} directory.
+
+\begin{verbatim}
+<?xml version='1.0'?>
+<productname desc="The Linux product">
+        <nameid>XtreemOS</nameid>
+        <nb_fork>5</nb_fork>
+        <based_on>2009.0</based_on>
+        <arch>x86_64</arch>
+        <repo desc="Where is the repositery of the distribution">
+                <path>/mnt/BIG/dis</path>
+        </repo>
+        <workdir desc="Where is my working dir">
+                <path>/home/xtreemos/build_bcd</path>
+        </workdir>
+\end{verbatim}
+
+The prefix for log will be \textbf{tocopy\_plop}. BCD will use the installer in the \textbf{/home/xtreemos/export\_x86\_64}
+directory, and use the \textbf{/home/xtreemos/Build/pieces/patch-oem.pl} patch. New advertising will be taken from
+\textbf{/home/xtreemos/Build/pieces/advertising} directory.
+if you want to use proprietary firmware in stage1 (from the non-free/initrd-firmware package) set the firmware option to yes.
+
+
+\begin{verbatim}
+<tocopy_file>tocopy_plop</tocopy_file>
+   <isolinux desc="isolinux info">
+         <defaultpath>isolinux</defaultpath>
+	 <firmware>yes</firmware>
+         <fullpath>/home/xtreemos/export_x86_64/isolinux</fullpath>
+           <entry>
+           </entry>
+           <!-- keep this entry to create an array ref -->
+           <entry></entry>
+    </isolinux>
+    <installer desc="path to the Mandriva installer (stage1/stage2)">
+           <defaultpath>install</defaultpath>
+           <fullpath>/home/xtreemos/export_x86_64</fullpath>
+           <patch>/home/xtreemos/Build/pieces/patch-oem.pl</patch>
+           <advertising>
+                 <defaultpath>advertising</defaultpath>
+                 <fullpath>/home/xtreemos/Build/pieces/advertising</fullpath>
+           </advertising>
+    </installer>
+\end{verbatim}
+
+Theme is the default one
+\begin{verbatim}
+<!-- theme must be available to path altx/all.rdz files -->
+     <theme name="Mandriva-Free">
+            <bootsplash>Mandriva-Free</bootsplash>
+            <bootsplash_path>/usr/share/bootsplash</bootsplash_path>
+            <gfxboot_path>/usr/share/gfxboot</gfxboot_path>
+     </theme>
+\end{verbatim}
+
+The first list is the Main one, we want to take into account all updates from this media, but
+we don't want to take into account suggests for packages. Package choosen from the list, and all
+dependencies will be stored into \textbf{media/main} directory of the DVD.
+\begin{verbatim}
+<media desc="media to take into account to build the ISO">
+            <list name="main" desc="Main_media">
+                 <input name="base">
+                    <file>lists/input_main</file>
+                 </input>
+                 <!-- to create an array you must keep this even if it is empty -->
+                 <input>
+                    <file>lists/to_add</file>
+                 </input>
+                 <!-- MANDATORY list of unwanted packages on the iso -->
+                 <exclude name="exclude">
+                     <file>lists/exclude_main</file>
+                 </exclude>
+                 <!-- to create an array you must keep this even if it is empty -->
+                 <exclude>
+                 </exclude>
+                 <!-- nosuggests options should be yes or no -->
+                 <nosuggests>yes</nosuggests>
+                 <!-- will be taken in {repo}{version}{arch}/media -->
+                 <mediadir>main</mediadir>
+                 <destmedia>main</destmedia>
+                 <pubkey>release/media_info/pubkey</pubkey>
+                 <release>yes</release>
+                 <updates>yes</updates>
+                 <backport>no</backport>
+                 <testing>no</testing>
+             </list>
+\end{verbatim}
+
+We create the media \textbf{contrib}, taken from the Mandriva 2009.0 repository to be able to
+get dependencies from packages for other media (here the XtreemOS media). We don't use list,
+because we dont want to take packages from contrib except dependencies for packages in the XtreemOS media.
+\begin{verbatim}
+                <list name="contrib" desc="Contrib_media">
+                        <!-- to create an array you must keep this even if it is empty -->
+                        <input>
+                        </input>
+                        <input>
+                        </input>
+                        <!-- MANDATORY list of unwanted packages on the iso -->
+                        <exclude name="exclude">
+                                <file>lists/exclude_contrib</file>
+                        </exclude>
+                        <!-- to create an array you must keep this even if it is empty -->
+                        <exclude>
+                        </exclude>
+                        <!-- nosuggests options should be yes or no -->
+                        <nosuggests>yes</nosuggests>
+                        <!-- will be taken in {repo}{version}{arch}/media -->
+                        <mediadir>contrib</mediadir>
+                        <destmedia>main</destmedia>
+                        <pubkey>release/media_info/pubkey</pubkey>
+                        <release>yes</release>
+                        <updates>yes</updates>
+                        <backport>no</backport>
+                        <testing>no</testing>
+                </list>
+\end{verbatim}
+
+The \textbf{r\_xos} media is the XtreemOS release media. We need to use the previously defined Contrib media
+to be able to solve dependencies for packages in this media. We don't want ot use \textbf{suggests} for packages,
+and this media is not available in the Mandriva 2009.0 repository, but in an external directory.
+We set the updates flag to yes to use all packages available in updates.
+\begin{verbatim}
+                <list name="r_xos" desc="r_xos_media">
+                        <media_supp>Contrib</media_supp>
+                        <input>
+                        </input>
+                        <!-- to create an array you must keep this even if it is empty -->
+                        <input></input>
+                        <!-- MANDATORY list of unwanted packages on the iso -->
+                        <exclude name="exclude">
+                                <file>lists/exclude_main</file>
+                        </exclude>
+                        <exclude name="exclude">
+                                <file>lists/exclude_contrib</file>
+                        </exclude>
+                        <!-- to create an array you must keep this even if it is empty -->
+                        <exclude>
+                        </exclude>
+                        <!-- nosuggests options should be yes or no -->
+                        <nosuggests>yes</nosuggests>
+                        <!-- will be taken in {repo}{version}{arch}/media -->
+                        <mediadir>xtreemos</mediadir>
+                        <destmedia>main</destmedia>
+                        <external name="r_xos">
+                            <fullpath_mediadir>/home/xtreemos/Build/pieces/x86_64_xos_release</fullpath_mediadir>
+                        </external>
+                        <!--
+                        <fullpath_mediadir>/home/plop/iso/final/t/x86_64/media/main</fullpath_mediadir>
+                        -->
+                        <pubkey>release/media_info/pubkey</pubkey>
+                        <release>yes</release>
+                        <updates>yes</updates>
+                        <backport>no</backport>
+                        <testing>no</testing>
+                </list>
+\end{verbatim}
+
+The product version is 2.1.1, and we use custom \textbf{compssUsers.pl} and \textbf{rpmsrate} file available in the
+\textbf{/home/xtreemos/Build/pieces} directory.
+\begin{verbatim}
+<mediainfo desc="All files relatives to media_info dir">
+        <version>2.1.1</version>
+           <!-- set fullpath if you want to use an files in a non MDV repositery -->
+           <fullpath>/home/xtreemos/Build/pieces</fullpath>
+           <mediainfo_dir>media/media_info</mediainfo_dir>
+           <rpmsrate>rpmsrate</rpmsrate>
+           <compssusers>compssUsers.pl</compssusers>
+           <filedeps>file-deps</filedeps>
+           <filter>lists/extra-filter</filter>
+           <urpmi_option> --wget </urpmi_option>
+           <askmedia desc="add for other media ?">
+                  <todo>yes</todo>
+           </askmedia>
+           <suppl desc="add an external media ?">
+                  <todo>yes</todo>
+           </suppl>
+           <xmlinfo>
+                  <todo>no</todo>
+           </xmlinfo>
+</mediainfo>
+\end{verbatim}
+
+Tag will be beta1, and we have definied some custom ISO header. All other option are standard one.
+\begin{verbatim}
+<iso>
+          <tag>beta1</tag>
+          <minor>1</minor>
+          <subversion>1</subversion>
+          <product>Download</product>
+          <branch>Devel</branch>
+          <type>basic</type>
+          <!-- dual arch not yet supported -->
+          <dualarch>no</dualarch>
+          <!-- SIZE is not YET USED !!! -->
+          <size>700</size>
+          <!-- media.cfg will be stored on genisoimage->{builddir}/{nameid}/{arch} -->
+          <mediacfg>media/media_info/media.cfg</mediacfg>
+          <hdlist>yes</hdlist>
+          <synthesis>yes</synthesis>
+          <header desc="Iso header info">
+               <systemid>XtreemOS Linux</systemid>
+               <volumeid>XOS-2.1.1-x86_64</volumeid>
+               <volumesetid>XtreemOS Linux - 2.1.1 - x86_64 DVD</volumesetid>
+               <publisherid>XtreemOS</publisherid>
+               <datapreparer>Mandriva BCD</datapreparer>
+               <applicationid>XtreemOS Linux - 2.1.1 </applicationid>
+               <copyrightid>XtreemOS Linux</copyrightid>
+           </header>
+           <genisoimage desc="iso generation tool">
+               <options>-f -r -J -hide-rr-moved -nobak -cache-inodes -no-emul-boot -boot-load-size 4 </options>
+               <bootcat>isolinux/boot.cat</bootcat>
+           <!-- builddir and destdir will always use the workdir -->
+               <builddir>build</builddir>
+                <isodir>iso</isodir>
+           </genisoimage>
+</iso>
+</productname>
+\end{verbatim}
+
+Resume of the XtreemOS example:
+
+\begin{itemize}
+\item the build directory used to create the ISO will be:\textbf{/home/xtreemos/build\_bcd/build/XtreemOS-2.1.1-x86\_64}
+\item the iso directory will be: \textbf{/home/xtreemos/build\_bcd/iso/XtreemOS-2.1.1-x86\_64}
+\item ISO name will be: \textbf{XtreemOS-2.1.1-x86\_64.iso}
+\item The chroot and logs are in \textbf{/tmp/XtreemOS-2.1.1-x86\_64} directory
+\item all urpmq queris are stored in \textbf{/tmp/XtreemOS-2.1.1-x86\_64/urpmq} directory
+\item previous urpmq queries were in \textbf{/var/lib/bcd/XtreemOS-2.1.1-x86\_64} directory
+\end{itemize}
+
+
+\subsection{running output}
+
+
+\newpage
+\section{ISSUES}
+
+\begin{itemize}
+\item if you want to build ISO for x86\_64 you must use an x86\_64 system (due to \textbf{urpmi})
+\item if hdlist/synthesis are corrupted in your repositery, you can't build ISO
+\item don't check for bad rpm
+\item BCD do an ``urpmq --updates'' wich can be removed to speed up the process
+\end{itemize}
+
+\end{document}