path: root/doc/howto_bcd.tex

\documentclass[oneside,a4paper,10pt]{article}
\usepackage{times}
\usepackage{graphicx}
\usepackage[T1]{fontenc}
\usepackage[english]{babel}
\usepackage{fullpage}

\begin{document}


\title{HOW TO use BCD}
\author{Antoine Ginies}
\begin{center}
 \includegraphics{./mandriva-logo.jpg}
 % mandriva-logo.jpg: 950x239 pixel, 72dpi, 33.51x8.43 cm, bb=0 0 950 239
\end{center}
\maketitle

\tableofcontents

\section{Summary}

BCD is a simple tool to build ISO using \textbf{urpmi} to create a mini 
chroot of the future system, and \textbf{urpmq} command to calculate dependencies
of packages on the future DVD/CD.

\section{The environment}

\begin{flushleft}
BCD is available in Mandriva subversion repository:
http://svn.mandriva.com/svn/soft/build\_system/bcd
\end{flushleft}

\begin{flushleft}
Some release of BCD has been packaged, but you should use the svn version, because this tool is under development, and really unstable....
All previous XML configuration's file are compatibles with previous version of BCD. 
\end{flushleft}

\begin{flushleft}
  Create a user on the system:
  \begin{verbatim}
adduser plop
  \end{verbatim}  
\end{flushleft}

\begin{flushleft}
  You must create a directory to store all \textbf{urpmq} queries. All queries done by \textbf{urpmq} per package
are stored in this directory + product-name-release-arch. To speed up the process of creating ISO,
next time you run BCD, it will check for each package in input list if there is a file wich contains the result
of the urpmq queries, if BCD found one then it will use it, else it will redo an
\textbf{urpmq --requires-recursives --sources package\_name} and store the result in a new file.
So if you have update your repository, you should clean this directory. If you have update some packages,
you should remove file relative to those packages, to redo the \textbf{urpmq} queries otherwise BCD will use
the old result of the \textbf{urpmq} queries, and this can lead to error if deps are no more the same. Moreover
if you change the tag \textbf{<nosuggests></nosuggests>} to yes or no, you must erase all your previous queries.
If your media contains SRPMS, the result of previous \textbf{urpmq} queries won't be used, because we want to calculate \textbf{buildrequires} in this case.

  \begin{verbatim}
mkdir -p /var/lib/bcd
  \end{verbatim} 
\end{flushleft}

\begin{flushleft}
  Give you user RW on this directory:
  \begin{verbatim}
chown plop.users /var/lib/bcd
  \end{verbatim} 
\end{flushleft}

\begin{flushleft}
  Give some \textbf{sudoers} right to your user:
  \begin{verbatim}
vi /etc/sudoers
plop ALL=(root) NOPASSWD:/bin/mount, /bin/umount, \
        /usr/sbin/chroot, \
        /usr/sbin/urpmi, \
        /usr/sbin/urpmi.addmedia, \
        /usr/sbin/urpmi.removemedia, \
        /bin/rm
  \end{verbatim}
\end{flushleft}

\begin{flushleft}
  Checkout the source of BCD:
  \begin{verbatim}
svn co http://svn.mandriva.com/svn/soft/build_system/bcd
  \end{verbatim} 
\end{flushleft}

\begin{flushleft}
  Prepare your distribution repository, you should have
  an access to it, or a local copy.
  \begin{verbatim}
[plop@isobel /mnt/BIG/dis/official] 04:04:28pm
$ ls
2008.0/  2008.1/  2009.0/  2009.1/  2010.0/  iso/
$ ls -r 2010.0
x86_64/  SRPMS/  i586/
  \end{verbatim} 
\end{flushleft}

\begin{flushleft}
  You need to install some tools and deps to be able to use bcd:
  \begin{verbatim}
urpmi perl-File-Copy-Recursive sudo urpmi perl-Parallel-ForkManager \
 cdrkit-genisoimage cdrkit-isotools syslinux \
 gfxboot mandriva-gfxboot-theme drakxtools-backend \
 mandriva-Free-theme rpmtools rpmcheck
  \end{verbatim} 
\end{flushleft}

\begin{flushleft}
  Now your system should be ready to build ISO.
\end{flushleft}

\newpage
\section{configuration file}

You need to have an XML configuration file. You should use
an example provided and adapt it to your configuration file.

\input{conf.xml.tex}

\newpage
\section{architecture of a Mandriva DVD}

A Mandriva CD/DVD contains:
\begin{itemize}
 \item \textbf{autorun.inf} file and \textbf{dosutils} directory used under Windows OS to popup a dialog box
 \item \textbf{i586} or \textbf{x86\_64} directory wich contains the installer and packages and other stuff
\end{itemize}

In the \textbf{i586} or the \textbf{x86\_64} directory:
\begin{itemize}
 \item \textbf{doc}: full doc of howto install a Mandriva linux system
 \item \textbf{install}
  \begin{itemize}
    \item \textbf{extra/advertising}: all images used in the slideshow while installing
    \item \textbf{images}: file used to boot the system
    \item \textbf{stage2}: mdkinst.sqfs is the stage2 of the installer, rescue.sqfs is the rescue system. Those
    two files are compressed in squashfs.
  \end{itemize}
 \item \textbf{isolinux}: contains file used to boot the CD/DVD, and all gfxboot stuff
 \item \textbf{media}
 \item \textbf{main}: a media with packages
 \item \textbf{contrib}: another one with packages
 \item \textbf{media\_info}
    \begin{itemize}
  \item \textbf{compssUsers.pl}: describe category to display in "select package to install" step
  \item \textbf{file-deps}: Use the given file to know which file dependencies are used by other media (man \textbf{genhdlist2})
  \item \textbf{hdlists}: old way to describe media on the DVD/CD
  \item \textbf{media.cfg}: describe the repository of the DVD/CD
  \item \textbf{rpmsrate}: a package is selected if its category is selected and depending of its priority (5=high, 1=very low)
    \end{itemize}
\end{itemize}

\newpage
\section{Step by step process}

\begin{itemize}
  \item clean all previous build
  \item create all needed directory
  \item isolinux step
    \begin{itemize}
      \item erase all previous isolinux directory in the build dir
      \item copy the \textbf{isolinux} dir in the build dir
      \item add entry if specidifed in the configuration file (like HDT)
      \item switch theme for all images in \textbf{isolinux/altX} directory
    \end{itemize}
  \item stage1 and stage2
    \begin{itemize}
      \item copy all files from the \textit{install} dir to the build dir
      \item update all \textbf{advertising}
      \item copy \textbf{dosutils} tools in the build dir
      \item copy the doc
      \item copy the patch file if present
    \end{itemize}
  \item media step
    \begin{itemize}
      \item list all media configured, and report missing list file
      \item create media directory in the build dir
      \item copy all pubkey for each media and report missing one
      \item create the \textbf{media/media\_info/media.cfg} file
      \item copy the file-deps
      \item copy the \textbf{rpmsrate}
      \item copy \textbf{compssUsers.pl}
      \item create a basic chroot of the futur Distribution (to be able to run urpmq)
      \item mount loop distribution repository and external media in chroot
      \item generate a list of rpm to copy and dependencies based on list files
      \item concatenate all files
      \item try to define the size of the DVD/CD (not yet implemented)
      \item link or copy packages on the build dir
      \item try to solve all doble issue of packages
      \item remove all packages listed in exclude list
      \item use \textbf{gendistrib} to create hdlists and synthesys
      \item launch \textbf{rpmcheck} to check missing deps conflicts and other failure
      \item use the rpmcheck's package list to solve missing deps using urpmq
      \item re-link or copy packages on the build dir
      \item re-try to solve all doble issue of packages
      \item re-remove all packages listed in exclude list
      \item re-launch \textbf{gendistrib} to create hdlists and synthesys
      \item re-launch \textbf{rpmcheck} to check missing deps conflicts and other failure
      \item umount all mountloop disrectory (distribution repository and extra media)
      \item remove hdlist if no more needed (Mandriva version > 2010.0)
      \item show the size of the DVD (not yet implemented)
    \end{itemize}
  \item ISO step
    \begin{itemize}
      \item create the \textbf{product.id} file
      \item copy text files (INSTALL, README etc...)
      \item create the idx
      \item create the ISO
      \item \textbf{isohybrid} the ISO
      \item create the md5 and sha1
      \item create an html report web page
    \end{itemize}
\end{itemize}

\newpage
\section{list files}

\begin{itemize}
 \item input list: list of rpm needed on the CD. Those list should contains the name
or the provides of a package. You may also use a CAT\_XXX from the rpmsrate file, with the weight parameter.
\end{itemize}

\begin{verbatim}
ie: of lists/input_main
 INSTALL 1
 CAT_X 1
 CAT_GNOME 1
 CAT_AUDIO 1
 aumix
 task-lxde
\end{verbatim} 

\begin{itemize}
 \item exclude list: list of unwanted packages on the CD. Be carreful to do not exclude mandatory packages
or you DVD/CD will be not usable. Rpmcheck will report any bug in hdlists at the end of
the 'media' step process. Packages listed in exclude list will be removed from the
DVD/CD at the end of the build process.
\end{itemize}

\begin{itemize}
 \item no-filter: list of RPM you don't want to compare. Usefull to keep more than one version of a package, or use an old one.
\end{itemize}

\newpage
\section{BCD options}

\begin{itemize}
 \item First arg must be the XML configuration file
  \item Second arg must be
  \begin{itemize}
    \item stagex: update the stage1/stage2 in builddir
    \item isolinux: update the isolinux in builddir
    \item clean: clean the builddir and isodir
    \item media: create the list of wanted packages, create all media in builddir, link/copy packages in builddir/media,
create the media.cfg, and generate the hdtlist/synthesis
    \item gendistrib: create the hdlists/synthesis in builddir (depends on the based\_on tag)
    \item rpmcheck: check the hdlist in the buildir
    \item list: list all media and list of input/exclude (from the xml conf file)
    \item mediarepo: list all medias availables in the repositery
    \item checkrepo: rpmcheck main hdlists in the repositery
    \item all: run all stage in order to get an ISO
    \item md5: create MD5 sum of the ISO
    \item resign: resign all rpms with a password and a gpg file
    \item doble: check doble rpm in the media build\_dir
    \item kernel: compare kernel and keep the latest one
    \item nodoble: don't check doble rpm in the media build\_dir
    \item info: parse the configuration file (usefull to check the XML syntax)
    \item clean: clean previous urpmq log file
    \item html: generate a log report web page
    \item iso: create the final ISO from the builddir
    \item verbose: be more verbose
  \end{itemize}
  \item extra arg:
    \begin{itemize}
    \item noiso: don't build the iso (md5, isohybrid) in case of 'all' process
    \item copy: copy rpm instead of link them from the repositery
    \item verbose: verbose mode to report more info in the output
    \end{itemize}
\end{itemize}

\section{Examples of use}

\begin{flushleft}
the folowing example will do all step to create an ISO:
  \begin{verbatim}
bcd mdv-20091.xml all
  \end{verbatim}
\end{flushleft}

\begin{flushleft}
the following example will create the build dir without creating ISO:
  \begin{verbatim}
bcd.pl mdv-20091.xml all noiso
  \end{verbatim}
\end{flushleft}

\begin{flushleft}
same as above but copy packages instead of link them from the repositery:
  \begin{verbatim}
bcd.pl mdv-20091.xml all noiso copy
  \end{verbatim}
\end{flushleft}

\begin{flushleft}
the following will update the isolinux in the build directory:
  \begin{verbatim}
bcd.pl mdv-20091.xml isolinux
  \end{verbatim}
\end{flushleft}

\begin{flushleft}
to check the main hdlist in the repositery with rpmcheck
  \begin{verbatim}
bcd.pl mdv-20091.xml checkrepo
  \end{verbatim}
\end{flushleft}

\begin{flushleft}
clean the builddir and isodir
  \begin{verbatim}
bcd.pl mdv-20091.xml clean
  \end{verbatim}
\end{flushleft}

\newpage
\section{XtreemOS example}

\subsection{introduction}

XtreemOS is an European project (http://xtreemos.eu/) to build and promote a Linux-based Operating System
to Support Virtual Organizations for Next Generation Grids. This project was one of the platform used
to develop and test BCD.

XtreemOS GNU/Linux distribution is based on a Mandriva 2009.0 with updates, and and
external repository xtreemos, wich contains sub-media release, updates and testing.
We will explain the full configuration file for x864\_64 arch to help you understand the way 
to create one for your need.

\subsection{the xos64.xml file}

The name of the product will be XtreemOS, we want to do only 5 fork of \textbf{urpmq} queries, this product is based on
a Mandriva 2009.0 product, and we want to build a DVD for x86\_64 architecture.
The repository of the distribution is available at /mnt/BIG/dis directory, and i want to
create my build dir and iso dir in the \textbf{/home/xtreemos/build\_bcd} directory.

\begin{verbatim}
<?xml version='1.0'?>
<productname desc="The Linux product">
        <nameid>XtreemOS</nameid>
        <nb_fork>5</nb_fork>
        <based_on>2009.0</based_on>
        <arch>x86_64</arch>
        <repo desc="Where is the repositery of the distribution">
                <path>/mnt/BIG/dis</path>
        </repo>
        <workdir desc="Where is my working dir">
                <path>/home/xtreemos/build_bcd</path>
        </workdir>
\end{verbatim}

The prefix for log will be \textbf{tocopy\_plop}. BCD will use the installer in the \textbf{/home/xtreemos/export\_x86\_64}
directory, and use the \textbf{/home/xtreemos/Build/pieces/patch-oem.pl} patch. New advertising will be taken from
\textbf{/home/xtreemos/Build/pieces/advertising} directory.
if you want to use proprietary firmware in stage1 (from the non-free/initrd-firmware package) set the firmware option to yes.


\begin{verbatim}
<tocopy_file>tocopy_plop</tocopy_file>
   <isolinux desc="isolinux info">
         <defaultpath>isolinux</defaultpath>
	 <firmware>yes</firmware>
         <fullpath>/home/xtreemos/export_x86_64/isolinux</fullpath>
           <entry>
           </entry>
           <!-- keep this entry to create an array ref -->
           <entry></entry>
    </isolinux>
    <installer desc="path to the Mandriva installer (stage1/stage2)">
           <defaultpath>install</defaultpath>
           <fullpath>/home/xtreemos/export_x86_64</fullpath>
           <patch>/home/xtreemos/Build/pieces/patch-oem.pl</patch>
           <advertising>
                 <defaultpath>advertising</defaultpath>
                 <fullpath>/home/xtreemos/Build/pieces/advertising</fullpath>
           </advertising>
    </installer>
\end{verbatim}

Theme is the default one
\begin{verbatim}
<!-- theme must be available to path altx/all.rdz files -->
     <theme name="Mandriva-Free">
            <bootsplash>Mandriva-Free</bootsplash>
            <bootsplash_path>/usr/share/bootsplash</bootsplash_path>
            <gfxboot_path>/usr/share/gfxboot</gfxboot_path>
     </theme>
\end{verbatim}

The first list is the Main one, we want to take into account all updates from this media, but
we don't want to take into account suggests for packages. Package choosen from the list, and all
dependencies will be stored into \textbf{media/main} directory of the DVD.
\begin{verbatim}
<media desc="media to take into account to build the ISO">
            <list name="main" desc="Main_media">
                 <input name="base">
                    <file>lists/input_main</file>
                 </input>
                 <!-- to create an array you must keep this even if it is empty -->
                 <input>
                    <file>lists/to_add</file>
                 </input>
                 <!-- MANDATORY list of unwanted packages on the iso -->
                 <exclude name="exclude">
                     <file>lists/exclude_main</file>
                 </exclude>
                 <!-- to create an array you must keep this even if it is empty -->
                 <exclude>
                 </exclude>
                 <!-- nosuggests options should be yes or no -->
                 <nosuggests>yes</nosuggests>
                 <!-- will be taken in {repo}{version}{arch}/media -->
                 <mediadir>main</mediadir>
                 <destmedia>main</destmedia>
                 <pubkey>release/media_info/pubkey</pubkey>
                 <release>yes</release>
                 <updates>yes</updates>
                 <backport>no</backport>
                 <testing>no</testing>
             </list>
\end{verbatim}

We create the media \textbf{contrib}, taken from the Mandriva 2009.0 repository to be able to
get dependencies from packages for other media (here the XtreemOS media). We don't use list,
because we dont want to take packages from contrib except dependencies for packages in the XtreemOS media.
\begin{verbatim}
                <list name="contrib" desc="Contrib_media">
                        <!-- to create an array you must keep this even if it is empty -->
                        <input>
                        </input>
                        <input>
                        </input>
                        <!-- MANDATORY list of unwanted packages on the iso -->
                        <exclude name="exclude">
                                <file>lists/exclude_contrib</file>
                        </exclude>
                        <!-- to create an array you must keep this even if it is empty -->
                        <exclude>
                        </exclude>
                        <!-- nosuggests options should be yes or no -->
                        <nosuggests>yes</nosuggests>
                        <!-- will be taken in {repo}{version}{arch}/media -->
                        <mediadir>contrib</mediadir>
                        <destmedia>main</destmedia>
                        <pubkey>release/media_info/pubkey</pubkey>
                        <release>yes</release>
                        <updates>yes</updates>
                        <backport>no</backport>
                        <testing>no</testing>
                </list>
\end{verbatim}

The \textbf{r\_xos} media is the XtreemOS release media. We need to use the previously defined Contrib media
to be able to solve dependencies for packages in this media. We don't want ot use \textbf{suggests} for packages,
and this media is not available in the Mandriva 2009.0 repository, but in an external directory.
We set the updates flag to yes to use all packages available in updates.
\begin{verbatim}
                <list name="r_xos" desc="r_xos_media">
                        <media_supp>Contrib</media_supp>
                        <input>
                        </input>
                        <!-- to create an array you must keep this even if it is empty -->
                        <input></input>
                        <!-- MANDATORY list of unwanted packages on the iso -->
                        <exclude name="exclude">
                                <file>lists/exclude_main</file>
                        </exclude>
                        <exclude name="exclude">
                                <file>lists/exclude_contrib</file>
                        </exclude>
                        <!-- to create an array you must keep this even if it is empty -->
                        <exclude>
                        </exclude>
                        <!-- nosuggests options should be yes or no -->
                        <nosuggests>yes</nosuggests>
                        <!-- will be taken in {repo}{version}{arch}/media -->
                        <mediadir>xtreemos</mediadir>
                        <destmedia>main</destmedia>
                        <external name="r_xos">
                            <fullpath_mediadir>/home/xtreemos/Build/pieces/x86_64_xos_release</fullpath_mediadir>
                        </external>
                        <!--
                        <fullpath_mediadir>/home/plop/iso/final/t/x86_64/media/main</fullpath_mediadir>
                        -->
                        <pubkey>release/media_info/pubkey</pubkey>
                        <release>yes</release>
                        <updates>yes</updates>
                        <backport>no</backport>
                        <testing>no</testing>
                </list>
\end{verbatim}

The product version is 2.1.1, and we use custom \textbf{compssUsers.pl} and \textbf{rpmsrate} file available in the
\textbf{/home/xtreemos/Build/pieces} directory.
\begin{verbatim}
<mediainfo desc="All files relatives to media_info dir">
        <version>2.1.1</version>
           <!-- set fullpath if you want to use an files in a non MDV repositery -->
           <fullpath>/home/xtreemos/Build/pieces</fullpath>
           <mediainfo_dir>media/media_info</mediainfo_dir>
           <rpmsrate>rpmsrate</rpmsrate>
           <compssusers>compssUsers.pl</compssusers>
           <filedeps>file-deps</filedeps>
           <filter>lists/no-filter</filter>
           <urpmi_option> --wget </urpmi_option>
           <askmedia desc="add for other media ?">
                  <todo>yes</todo>
           </askmedia>
           <suppl desc="add an external media ?">
                  <todo>yes</todo>
           </suppl>
           <xmlinfo>
                  <todo>no</todo>
           </xmlinfo>
</mediainfo>
\end{verbatim}

Tag will be beta1, and we have definied some custom ISO header. All other option are standard one.
\begin{verbatim}
<iso>
          <tag>beta1</tag>
          <minor>1</minor>
          <subversion>1</subversion>
          <product>Download</product>
          <branch>Devel</branch>
          <type>basic</type>
          <!-- dual arch not yet supported -->
          <dualarch>no</dualarch>
          <!-- SIZE is not YET USED !!! -->
          <size>700</size>
          <!-- media.cfg will be stored on genisoimage->{builddir}/{nameid}/{arch} -->
          <mediacfg>media/media_info/media.cfg</mediacfg>
          <hdlist>yes</hdlist>
          <synthesis>yes</synthesis>
          <header desc="Iso header info">
               <systemid>XtreemOS Linux</systemid>
               <volumeid>XOS-2.1.1-x86_64</volumeid>
               <volumesetid>XtreemOS Linux - 2.1.1 - x86_64 DVD</volumesetid>
               <publisherid>XtreemOS</publisherid>
               <datapreparer>Mandriva BCD</datapreparer>
               <applicationid>XtreemOS Linux - 2.1.1 </applicationid>
               <copyrightid>XtreemOS Linux</copyrightid>
           </header>
           <genisoimage desc="iso generation tool">
               <options>-f -r -J -hide-rr-moved -nobak -cache-inodes -no-emul-boot -boot-load-size 4 </options>
               <bootcat>isolinux/boot.cat</bootcat>
           <!-- builddir and destdir will always use the workdir -->
               <builddir>build</builddir>
                <isodir>iso</isodir>
           </genisoimage>
</iso>
</productname>
\end{verbatim}

Resume of the XtreemOS example:

\begin{itemize}
\item the build directory used to create the ISO will be:\textbf{/home/xtreemos/build\_bcd/build/XtreemOS-2.1.1-x86\_64}
\item the iso directory will be: \textbf{/home/xtreemos/build\_bcd/iso/XtreemOS-2.1.1-x86\_64}
\item ISO name will be: \textbf{XtreemOS-2.1.1-x86\_64.iso}
\item The chroot and logs are in \textbf{/tmp/XtreemOS-2.1.1-x86\_64} directory
\item all urpmq queris are stored in \textbf{/tmp/XtreemOS-2.1.1-x86\_64/urpmq} directory
\item previous urpmq queries were in \textbf{/var/lib/bcd/XtreemOS-2.1.1-x86\_64} directory
\end{itemize}


\subsection{running output}


\newpage
\section{ISSUES}

\begin{itemize}
\item if you want to build ISO for x86\_64 you must use an x86\_64 system (due to \textbf{urpmi})
\item if hdlist/synthesis are corrupted in your repositery, you can't build ISO
\item don't check for bad rpm
\end{itemize}

\end{document}