This article will use the GoCD terms "pipeline" and "stage" (a pipeline is somewhat like a "job" Jenkins, and a pipeline comprises one or more stages).
This article describes (at a very high level) the system my colleague Hilverd Reker and I have set up to automatically create pipelines. This has built on experience I gained doing something similar with Ran Fan at a previous client, and being the "customer" of an automated CI configuration system at another previous client.
Inception
We have a pipeline in our CI server to automatically create the pipelines we want. We have called this "inception", after the film - I think Ran Fan came up with the name.The inception pipeline looks for new things to build in new repositories, and sub directories within existing repositories, and creates pipelines as appropriate (using gomatic). (The inception process that Ran Fan and I wrote previously, looked for new things to build within "one large repo" (maybe the subject of a future blog article), and new branches of that repository).
The advantage of having this fully automated, compared to having to run a script to get the pipeline set up, is that it ensures that all pipelines get set up: none are forgotten and no effort is required.
Our inception job sets up a pipeline with only one stage, the bootstrap stage, which configures the rest of the pipeline. This keeps the inception job simple.
The bootstrap stage
Some of the configuration of a pipeline depends upon the files in the repository that the pipeline is for. By making the first stage in the pipeline the bootstrap stage, it can configure the pipeline accurately for the files as they exist when the pipeline runs. If a pipeline is configured by the inception job, or by running a script, rather than a bootstrap stage, then its configuration will not reflect the files in the repository when they change, but rather how they were at the time the inception job, or script, ran. This would result in pipelines failing because they are not set up correctly for the files they are trying to use; hence we have the bootstrap as part of the pipeline itself to solve that problem.Implementation notes
Our bootstrap stage only alters the configuration of the pipeline if it needs to: it runs very quickly if no changes are needed. GoCD handles changes to the configuration of pipeline well. After the bootstrap stage has run, the subsequent stages run in the newly configured, or reconfigured, pipeline as expected. GoCD also handles the history of a pipelined reasonably well (but not always getting it right), even when it's configuration changes over time.Example
What would help right now would be an example - but that'll take time to prepare; watch this space (patiently) ...Copyright ©2015 Ivan Moore
