[Mageia-sysadm] [241] Add a module to construct files from fragments.

Fri Nov 12 18:42:23 CET 2010

Revision: 241
+Author:   boklm
+Date:     2010-11-12 18:42:22 +0100 (Fri, 12 Nov 2010)
+Log Message:
+-----------
+Add a module to construct files from fragments.
+
+Downloaded from https://github.com/ripienaar/puppet-concat/
+
+Added Paths:
+-----------
+    puppet/modules/concat/
+    puppet/modules/concat/CHANGELOG
+    puppet/modules/concat/README.markdown
+    puppet/modules/concat/files/
+    puppet/modules/concat/files/concatfragments.sh
+    puppet/modules/concat/files/null/
+    puppet/modules/concat/files/null/.gitignore
+    puppet/modules/concat/manifests/
+    puppet/modules/concat/manifests/fragment.pp
+    puppet/modules/concat/manifests/init.pp
+    puppet/modules/concat/manifests/setup.pp
+
+Added: puppet/modules/concat/CHANGELOG
+===================================================================
+--- puppet/modules/concat/CHANGELOG	                        (rev 0)
++++ puppet/modules/concat/CHANGELOG	2010-11-12 17:42:22 UTC (rev 241)
+@@ -0,0 +1,27 @@
++KNOWN ISSUES:
++- In 0.24.8 you will see inintended notifies, if you build a file
++  in a run, the next run will also see it as changed.  This is due
++  to how 0.24.8 does the purging of unhandled files, this is improved
++  in 0.25.x and we cannot work around it in our code.
++
++CHANGELOG:
++- 2010/02/19 - initial release
++- 2010/03/12 - add support for 0.24.8 and newer
++             - make the location of sort configurable
++             - add the ability to add shell comment based warnings to
++               top of files
++             - add the ablity to create empty files
++- 2010/04/05 - fix parsing of WARN and change code style to match rest
++               of the code
++             - Better and safer boolean handling for warn and force
++             - Don't use hard coded paths in the shell script, set PATH
++               top of the script
++             - Use file{} to copy the result and make all fragments owned
++               by root.  This means we can chnage the ownership/group of the
++               resulting file at any time.
++             - You can specify ensure => "/some/other/file" in concat::fragment
++               to include the contents of a symlink into the final file.
++- 2010/04/16 - Add more cleaning of the fragment name - removing / from the $name
++- 2010/05/22 - Improve documentation and show the use of ensure =>
++- 2010/07/14 - Add support for setting the filebucket behavior of files
++- 2010/10/04 - Make the warning message configurable
+
+Added: puppet/modules/concat/README.markdown
+===================================================================
+--- puppet/modules/concat/README.markdown	                        (rev 0)
++++ puppet/modules/concat/README.markdown	2010-11-12 17:42:22 UTC (rev 241)
+@@ -0,0 +1,103 @@
++What is it?
++===========
++
++A Puppet module that can construct files from fragments.
++
++Please see the comments in the various .pp files for details
++as well as posts on my blog at http://www.devco.net/
++
++Released under the Apache 2.0 licence
++
++Usage:
++------
++
++If you wanted a /etc/motd file that listed all the major modules
++on the machine.  And that would be maintained automatically even
++if you just remove the include lines for other modules you could
++use code like below, a sample /etc/motd would be:
++
++<pre>
++Puppet modules on this server:
++
++    -- Apache
++    -- MySQL
++</pre>
++
++Local sysadmins can also append to the file by just editing /etc/motd.local
++their changes will be incorporated into the puppet managed motd.
++
++<pre>
++# class to setup basic motd, include on all nodes
++class motd {
++   include concat::setup
++   $motd = "/etc/motd"
++
++   concat{$motd,
++      owner => root,
++      group => root,
++      mode  => 644
++   }
++
++   concat::fragment{"motd_header":
++      target => $motd,
++      content => "\nPuppet modules on this server:\n\n",
++      order   => 01,
++   }
++
++   # local users on the machine can append to motd by just creating
++   # /etc/motd.local
++   concat::fragment{"motd_local":
++      target => $motd,
++      ensure  => "/etc/motd.local",
++      order   => 15
++   }
++}
++
++# used by other modules to register themselves in the motd
++define motd::register($content="", $order=10) {
++   if $content == "" {
++      $body = $name
++   } else {
++      $body = $content
++   }
++
++   concat::fragment{"motd_fragment_$name":
++      target  => "/etc/motd",
++      content => "    -- $body\n"
++   }
++}
++
++# a sample apache module
++class apache {
++   include apache::install, apache::config, apache::service
++
++   motd::register{"Apache": }
++}
++</pre>
++
++Known Issues:
++-------------
++* In 0.24.8 you will see inintended notifies, if you build a file
++  in a run, the next run will also see it as changed.  This is due
++  to how 0.24.8 does the purging of unhandled files, this is improved
++  in 0.25.x and we cannot work around it in our code.
++
++Contributors:
++-------------
++**Paul Elliot**   
++
++ * Provided 0.24.8 support, shell warnings and empty file creation support.
++
++**Chad Netzer**
++
++ * Various patches to improve safety of file operations
++ * Symlink support
++
++**David Schmitt**
++
++ * Patch to remove hard coded paths relying on OS path
++ * Patch to use file{} to copy the resulting file to the final destination.  This means Puppet client will show diffs and that hopefully we can change file ownerships now
++
++Contact:
++--------
++You can contact me on rip at devco.net or follow my blog at http://www.devco.net I am also on twitter as ripienaar
+
+Added: puppet/modules/concat/files/concatfragments.sh
+===================================================================
+--- puppet/modules/concat/files/concatfragments.sh	                        (rev 0)
++++ puppet/modules/concat/files/concatfragments.sh	2010-11-12 17:42:22 UTC (rev 241)
+@@ -0,0 +1,123 @@
++#!/bin/bash
++
++# Script to concat files to a config file.
++#
++# Given a directory like this:
++# /path/to/conf.d
++# |-- fragments
++# |   |-- 00_named.conf
++# |   |-- 10_domain.net
++# |   `-- zz_footer
++#
++# The script supports a test option that will build the concat file to a temp location and 
++# use /usr/bin/cmp to verify if it should be run or not.  This would result in the concat happening
++# twice on each run but gives you the option to have an unless option in your execs to inhibit rebuilds.
++# 
++# Without the test option and the unless combo your services that depend on the final file would end up 
++# restarting on each run, or in other manifest models some changes might get missed.
++#
++# OPTIONS:
++#  -o	The file to create from the sources
++#  -d	The directory where the fragments are kept
++#  -t	Test to find out if a build is needed, basically concats the files to a temp
++#       location and compare with what's in the final location, return codes are designed
++#       for use with unless on an exec resource
++#  -w   Add a shell style comment at the top of the created file to warn users that it 
++#       is generated by puppet
++#  -f   Enables the creation of empty output files when no fragments are found
++#  -n	Sort the output numerically rather than the default alpha sort
++#
++# the command: 
++#
++#   concatfragments.sh -o /path/to/conffile.cfg -d /path/to/conf.d
++#
++# creates /path/to/conf.d/fragments.concat and copies the resulting 
++# file to /path/to/conffile.cfg.  The files will be sorted alphabetically
++# pass the -n switch to sort numerically.
++# 
++# The script does error checking on the various dirs and files to make
++# sure things don't fail.
++
++OUTFILE=""
++WORKDIR=""
++TEST=""
++FORCE=""
++WARN=""
++SORTARG="-z"
++
++PATH=/sbin:/usr/sbin:/bin:/usr/bin
++
++while getopts "o:s:d:tnw:f" options; do
++	case $options in
++		o ) OUTFILE=$OPTARG;;
++		d ) WORKDIR=$OPTARG;;
++		n ) SORTARG="-zn";;
++		w ) WARNMSG="$OPTARG";;
++		f ) FORCE="true";;
++		t ) TEST="true";;
++		* ) echo "Specify output file with -o and fragments directory with -d"
++		    exit 1;;
++	esac
++done
++
++# do we have -o?
++if [ x${OUTFILE} = "x" ]; then
++	echo "Please specify an output file with -o"
++	exit 1
++fi
++
++# do we have -d?
++if [ x${WORKDIR} = "x" ]; then
++	echo "Please fragments directory with -d"
++	exit 1
++fi
++
++# can we write to -o?
++if [ -a ${OUTFILE} ]; then
++	if [ ! -w ${OUTFILE} ]; then
++		echo "Cannot write to ${OUTFILE}"
++		exit 1
++	fi
++else
++	if [ ! -w `dirname ${OUTFILE}` ]; then
++		echo "Cannot write to `dirname ${OUTFILE}` to create ${OUTFILE}"
++		exit 1
++	fi
++fi
++
++# do we have a fragments subdir inside the work dir?
++if [ ! -d "${WORKDIR}/fragments" ]  && [ ! -x "${WORKDIR}/fragments" ]; then
++	echo "Cannot access the fragments directory"
++	exit 1
++fi
++
++# are there actually any fragments?
++if [ ! "$(ls -A ${WORKDIR}/fragments)" ]; then
++	if [ x${FORCE} = "x" ]; then
++		echo "The fragments directory is empty, cowardly refusing to make empty config files"
++		exit 1
++	fi
++fi
++
++cd ${WORKDIR}
++
++if [ x${WARNMSG} = "x" ]; then
++	: > "fragments.concat"
++else
++	echo -e "$WARNMSG" > "fragments.concat"
++fi
++
++# find all the files in the fragments directory, sort them numerically and concat to fragments.concat in the working dir
++find fragments/ -type f -follow -print0 |sort ${SORTARG}|xargs -0 cat >>"fragments.concat"
++
++if [ x${TEST} = "x" ]; then
++	# This is a real run, copy the file to outfile
++	cp fragments.concat ${OUTFILE}
++	RETVAL=$?
++else
++	# Just compare the result to outfile to help the exec decide
++	cmp ${OUTFILE} fragments.concat
++	RETVAL=$?
++fi
++
++exit $RETVAL
+
+
+Property changes on: puppet/modules/concat/files/concatfragments.sh
+___________________________________________________________________
+Added: svn:executable
+   + *
+Added: svn:eol-style
+   + native
+
+Added: puppet/modules/concat/files/null/.gitignore
+===================================================================
+Added: puppet/modules/concat/manifests/fragment.pp
+===================================================================
+--- puppet/modules/concat/manifests/fragment.pp	                        (rev 0)
++++ puppet/modules/concat/manifests/fragment.pp	2010-11-12 17:42:22 UTC (rev 241)
+@@ -0,0 +1,51 @@
++# Puts a file fragment into a directory previous setup using concat
++# 
++# OPTIONS:
++#   - target    The file that these fragments belong to
++#   - content   If present puts the content into the file
++#   - source    If content was not specified, use the source
++#   - order     By default all files gets a 10_ prefix in the directory
++#               you can set it to anything else using this to influence the
++#               order of the content in the file
++#   - ensure    Present/Absent or destination to a file to include another file
++#   - mode      Mode for the file
++#   - owner     Owner of the file
++#   - group     Owner of the file
++#   - backup    Controls the filebucketing behavior of the final file and
++#               see File type reference for its use.  Defaults to 'puppet'
++define concat::fragment($target, $content='', $source='', $order=10, $ensure = "present", $mode = 0644, $owner = root, $group = root, $backup = "puppet") {
++    $safe_name = regsubst($name, '/', '_', 'G')
++    $safe_target_name = regsubst($target, '/', '_', 'G')
++    $concatdir = $concat::setup::concatdir
++    $fragdir = "${concatdir}/${safe_target_name}"
++
++    # if content is passed, use that, else if source is passed use that
++    # if neither passed, but $ensure is in symlink form, make a symlink
++    case $content {
++        "": {
++                case $source {
++                        "": {
++                                case $ensure {
++                                    "", "absent", "present", "file", "directory": {
++                                        crit("No content, source or symlink specified")
++                                    }
++                                }
++                            }
++                   default: { File{ source => $source } }
++                }
++            }
++        default: { File{ content => $content } }
++    }
++
++    file{"${fragdir}/fragments/${order}_${safe_name}":
++        mode   => $mode,
++        owner  => $owner,
++        group  => $group,
++        ensure => $ensure,
++        backup => $backup,
++        alias  => "concat_fragment_${name}",
++        notify => Exec["concat_${target}"]
++    }
++}
++
++# vi:tabstop=4:expandtab:ai
+
+Added: puppet/modules/concat/manifests/init.pp
+===================================================================
+--- puppet/modules/concat/manifests/init.pp	                        (rev 0)
++++ puppet/modules/concat/manifests/init.pp	2010-11-12 17:42:22 UTC (rev 241)
+@@ -0,0 +1,164 @@
++# A system to construct files using fragments from other files or templates.
++#
++# This requires at least puppet 0.25 to work correctly as we use some 
++# enhancements in recursive directory management and regular expressions
++# to do the work here.
++#
++# USAGE:
++# The basic use case is as below:
++#
++# concat{"/etc/named.conf": 
++#    notify => Service["named"]
++# }
++#
++# concat::fragment{"foo.com_config":
++#    target  => "/etc/named.conf",
++#    order   => 10,
++#    content => template("named_conf_zone.erb")
++# }
++#
++# # add a fragment not managed by puppet so local users 
++# # can add content to managed file
++# concat::fragment{"foo.com_user_config":
++#    target  => "/etc/named.conf",
++#    order   => 12,
++#    ensure  => "/etc/named.conf.local"
++# }
++#
++# This will use the template named_conf_zone.erb to build a single 
++# bit of config up and put it into the fragments dir.  The file
++# will have an number prefix of 10, you can use the order option
++# to control that and thus control the order the final file gets built in.
++#
++# SETUP:
++# The class concat::setup defines a variable $concatdir - you should set this
++# to a directory where you want all the temporary files and fragments to be
++# stored.  Avoid placing this somewhere like /tmp since you should never
++# delete files here, puppet will manage them.
++#
++# There's some regular expression magic to figure out the puppet version but
++# if you're on an older 0.24 version just set $puppetversion = 24
++#
++# Before you can use any of the concat features you should include the 
++# class concat::setup somewhere on your node first.
++#
++# DETAIL:
++# We use a helper shell script called concatfragments.sh that gets placed
++# in /usr/local/bin to do the concatenation.  While this might seem more 
++# complex than some of the one-liner alternatives you might find on the net
++# we do a lot of error checking and safety checks in the script to avoid 
++# problems that might be caused by complex escaping errors etc.
++# 
++# LICENSE:
++# Apache Version 2
++#
++# LATEST:
++# http://github.com/ripienaar/puppet-concat/
++#
++# CONTACT:
++# R.I.Pienaar <rip at devco.net> 
++# Volcane on freenode
++# @ripienaar on twitter
++# www.devco.net
++
++
++# Sets up so that you can use fragments to build a final config file, 
++#
++# OPTIONS:
++#  - mode       The mode of the final file
++#  - owner      Who will own the file
++#  - group      Who will own the file
++#  - force      Enables creating empty files if no fragments are present
++#  - warn       Adds a normal shell style comment top of the file indicating
++#               that it is built by puppet
++#  - backup     Controls the filebucketing behavior of the final file and
++#               see File type reference for its use.  Defaults to 'puppet'
++#
++# ACTIONS:
++#  - Creates fragment directories if it didn't exist already
++#  - Executes the concatfragments.sh script to build the final file, this script will create
++#    directory/fragments.concat.   Execution happens only when:
++#    * The directory changes 
++#    * fragments.concat != final destination, this means rebuilds will happen whenever 
++#      someone changes or deletes the final file.  Checking is done using /usr/bin/cmp.
++#    * The Exec gets notified by something else - like the concat::fragment define
++#  - Copies the file over to the final destination using a file resource
++#
++# ALIASES:
++#  - The exec can notified using Exec["concat_/path/to/file"] or Exec["concat_/path/to/directory"]
++#  - The final file can be referened as File["/path/to/file"] or File["concat_/path/to/file"]  
++define concat($mode = 0644, $owner = "root", $group = "root", $warn = "false", $force = "false", $backup = "puppet") {
++    $safe_name   = regsubst($name, '/', '_', 'G')
++    $concatdir   = $concat::setup::concatdir
++    $version     = $concat::setup::majorversion
++    $fragdir     = "${concatdir}/${safe_name}"
++    $concat_name = "fragments.concat.out"
++    $default_warn_message = '# This file is managed by Puppet. DO NOT EDIT.'
++
++    case $warn {
++        'true',true,yes,on:   { $warnmsg = "$default_warn_message" }
++        'false',false,no,off: { $warnmsg = "" }
++        default:              { $warnmsg = "$warn" }
++    }
++
++    $warnmsg_escaped = regsubst($warnmsg, "'", "'\\\\''", 'G')
++    $warnflag = $warnmsg_escaped ? {
++        ''      => '',
++        default => "-w '$warnmsg_escaped'"
++    }
++
++    case $force {
++        'true',true,yes,on: { $forceflag = "-f" }
++        'false',false,no,off: { $forceflag = "" }
++        default: { fail("Improper 'force' value given to concat: $force") }
++    }
++
++    File{
++        owner  => root,
++        group  => root,
++        mode   => $mode,
++        backup => $backup
++    }
++
++    file{$fragdir:
++            ensure   => directory;
++
++         "${fragdir}/fragments":
++            ensure   => directory,
++            recurse  => true,
++            purge    => true,
++            force    => true,
++            ignore   => [".svn", ".git", ".gitignore"],
++            source   => $version ? {
++                            24      => "puppet:///concat/null",
++                            default => undef,
++                        },
++            notify   => Exec["concat_${name}"];
++
++         "${fragdir}/fragments.concat":
++            ensure   => present;
++
++         "${fragdir}/${concat_name}":
++            ensure   => present;
++
++         $name:
++            source   => "${fragdir}/${concat_name}",
++            owner    => $owner,
++            group    => $group,
++            checksum => md5,
++            mode     => $mode,
++            ensure   => present,
++            alias    => "concat_${name}";
++    }
++
++    exec{"concat_${name}":
++        user      => root,
++        group     => root,
++        notify    => File[$name],
++        subscribe => File[$fragdir],
++        alias     => "concat_${fragdir}",
++        require   => [ File["/usr/local/bin/concatfragments.sh"], File[$fragdir], File["${fragdir}/fragments"], File["${fragdir}/fragments.concat"] ],
++        unless    => "/usr/local/bin/concatfragments.sh -o ${fragdir}/${concat_name} -d ${fragdir} -t ${warnflag} ${forceflag}",
++        command   => "/usr/local/bin/concatfragments.sh -o ${fragdir}/${concat_name} -d ${fragdir} ${warnflag} ${forceflag}",
++    }
++}
+
+Added: puppet/modules/concat/manifests/setup.pp
+===================================================================
+--- puppet/modules/concat/manifests/setup.pp	                        (rev 0)
++++ puppet/modules/concat/manifests/setup.pp	2010-11-12 17:42:22 UTC (rev 241)
+@@ -0,0 +1,36 @@
++# Sets up the concat system.
++#
++# $concatdir should point to a place where you wish the fragments to
++# live. This should not be somewhere like /tmp since ideally these files
++# should not be deleted ever, puppet should always manage them
++#
++# $puppetversion should be either 24 or 25 to enable a 24 compatible
++# mode, in 24 mode you might see phantom notifies this is a side effect
++# of the method we use to clear the fragments directory.
++# 
++# The regular expression below will try to figure out your puppet version
++# but this code will only work in 0.24.8 and newer.
++#
++# It also copies out the concatfragments.sh file to /usr/local/bin
++class concat::setup {
++    $concatdir = "/var/lib/puppet/concat"
++    $majorversion = regsubst($puppetversion, '^[0-9]+[.]([0-9]+)[.][0-9]+$', '\1')
++
++    file{"/usr/local/bin/concatfragments.sh": 
++            owner  => root,
++            group  => root,
++            mode   => 755,
++            source => $majorversion ? {
++                        24      => "puppet:///concat/concatfragments.sh",
++                        default => "puppet:///modules/concat/concatfragments.sh"
++                      };
++
++         $concatdir: 
++            ensure => directory,
++            owner  => root,
++            group  => root,
++            mode   => 755;
++    }
++}
++
++# vi:tabstop=4:expandtab:ai
+-------------- next part --------------
+An HTML attachment was scrubbed...
+URL: </pipermail/mageia-sysadm/attachments/20101112/c7579815/attachment-0001.html>
+