1 <?xml version="1.0" encoding="UTF-8"?>
2 <chapter id="chapter_using_bitbake_and_oe">
3 <title>Using bitbake and OpenEmbedded</title>
5 <section id="usage_introduction" xreflabel="introduction">
6 <title>Introduction</title>
8 <para>If your reading this manual you probably already have some idea of
9 what OpenEmbedded is all about, which is taking a lot of software and
10 creating something that you can run on another device. This involves
11 downloading some source code, compiling it, creating packages (like .deb
12 or .rpm) and/or creating boot images that can written to flash on the
13 device. The difficulties of cross-compiling and the variety of devices
14 which can be supported lead to a lot more complexity in an OpenEmbedded
15 based distribution than you'd find in a typical desktop distribution
16 (which cross-compiling isn't needed).</para>
18 <para>A major part of OpenEmbedded deals with compiling source code for
19 various projects. For each project this generally requires the same basic
24 <para>Download the source code, and any supporting files (such as
29 <para>Extract the source code and apply any patches that might be
34 <para>Configure the software if needed (such as is done by running the
35 configure script);</para>
39 <para>Compile everything;</para>
43 <para>Package up all the files into some package format, like .deb or
44 .rpm or .ipk, ready for installation.</para>
48 <para>There's nothing particular unusual about this process when building
49 on the machine the package is to be installed on. What makes this
54 <para>Cross-compiling: cross-compiling is difficult, and lots of
55 software has no support for cross-compiling - all packages included in
56 OE are cross-compiled;</para>
60 <para>Target and host are different: This means you can't compile up a
61 program and then run it - it's compiled to run on the target system,
62 not on the system compiling it. Lots of software tries to build and
63 run little helper and/or test applications and this won't work when
64 cross-compiling.</para>
68 <para>Tool chains (compiler, linker etc) are often difficult to
69 compile. Cross tool chains are even more difficult. Typically you'd go
70 out and download a tool chain made by someone else - but not when your
71 using OE. In OE the entire toolchain is built as part of the process.
72 This may make things take longer initially and may make it more
73 difficult to get started but makes it easier to apply patches and test
74 out changes to the tool chain.</para>
78 <para>Of course there's a lot more to OE then just compiling packages
79 though. Some of the features that OE supports includes:</para>
83 <para>Support for both glibc and uclibc;</para>
87 <para>Support for building for multiple target devices from the one
92 <para>Automatically building anything that is required for the package
93 to compile and/or run (build and run time dependencies);</para>
97 <para>Creation of flash and disk images of any one of a number of
98 types (jffs2, ext2.gz, squashfs etc) for booting directly on the
103 <para>Support for various packaging formats;</para>
107 <para>Automatic building all of the cross-compiling tools you'll
112 <para>Support for "native" packages that are built for the host
113 computer and not for the target and used to help during the build
118 <para>The rest of this chapter assumes you have mastered the Getting Start
119 guides to OpenEmbedded (see the OpenEmbedded web site for details), and
120 therefore have an appropriately configured setup and that you have managed
121 to actually build the cross-compilers for your target. This section talks
122 you through some of the background on what is happening with the aim of
123 helping you understand how to debug and develop within
126 <para>You'll also not a lot of reference to variables that define specific
127 directories or change the behaviour of some part of the build process. You
128 should refer to <xref linkend="chapter_recipes" /> for full details on
129 these variables.</para>
132 <section id="usage_configuration" xreflabel="configuration">
133 <title>Configuration</title>
135 <para>Configuration covers basic items such as where the various files can
136 be found and where output should be placed to more specific items such as
137 which hardware is being targeted and what features you want to have
138 included in the final image. The main configuration areas in OE
143 <term>conf/machine</term>
146 <para>This directory contains machine configuration information. For
147 each physical device a configuration file is required in this
148 directory that describes various aspects of the device, such as
149 architecture of the device, hardware features of the device (does it
150 have usb? a keyboard? etc), the type of flash or disk images needed
151 for the device, the serial console settings (if any) etc. If you are
152 adding support for a new device you would need to create a machine
153 configuration in this directory for the device.</para>
158 <term>conf/distro</term>
161 <para>This directory contains distribution related files. A
162 distribution decides how various activities are handled in the final
163 image, such as how networking configured, if usb devices will be
164 supported, what packaging system is used, which libc is used
170 <term>conf/bitbake.conf</term>
173 <para>This is the main bitbake configuration file. This file is not
174 to be edited but it is useful to look at it since it declares a
175 larger number of the predefined variables used by OE and controls a
176 lot of the base functionality provided by OE.</para>
181 <term>conf/local.conf</term>
184 <para>This is the end-user specific configuration. This file needs
185 to be copied and edited and is used to specify the various working
186 directories, the machine to build for and the distribution to
193 <section id="usage_workspace" xreflabel="workspace">
194 <title>Work space</title>
196 <para>Let's start out by taking a look at a typically working area. Note
197 that this may not be exactly what see - there are a lot of options that
198 can effect exactly how things are done, but it gives us a pretty good idea
199 of whats going on. What we are looking at here is the tmp directory (as
200 specified by TMPDIR in your local.conf):<screen>~%> find tmp -maxdepth 2 -type d
212 tmp/work/busybox-1.2.1-r13
213 tmp/work/libice-1_1.0.3-r0
214 tmp/work/arpwatch-2.1a15-r2
233 tmp/staging/x86_64-linux
237 tmp/staging/sh4-linux
244 tmp/deploy/images</screen></para>
246 <para>The various top level directories under tmp include:</para>
253 <para>Nothing of interest to users in here. These time stamps are
254 used by bitbake to keep track of what tasks it has completed and
255 what tasks it still has outstanding. This is how it knows that
256 certain actions have been completed and it doesn't need to do them
265 <para>Contains the cross-compiler toolchain. That is the gcc and
266 binutils that run on the host system but produce output for the
267 target system.</para>
275 <para>Nothing of interest to users in here. This contains the
276 bitbake parse cache and is used to avoid the need to parse all of
277 the recipes each time bitbake is run. This makes bitbake a lot
278 faster on the 2nd and subsequent runs.</para>
286 <para>The work directory. This is the directory in which all
287 packages are built - this is where the source code is extract,
288 patches applied, software configure, compiled, installed and
289 package. This is where you'll spend most of you time looking when
290 working in OE.</para>
298 <para>The generated root filesystem image for your target device.
299 This is the contents of the root filesystem (NOTE: fakeroot means it
300 doesn't have the correct device special nodes and permissions to use
309 <para>Contains the staging area, which is used to stored natively
310 compiled tools and and libraries and headers for the target that are
311 required for building other software.</para>
319 <para>Contains the final output from OE. This includes the
320 installation packages (typically .ipkg packages) and flash and/or
321 disk images. This is where you go to get the final product.</para>
326 <para>When people refer to the <emphasis>"tmp directory"</emphasis> this
327 is the directory them are talking about.</para>
329 <para>To perform a complete rebuild from script you would usually rename
330 or delete tmp and then restart your build. I recommend keeping one old
331 version of tmp around to use for comparison if something goes wrong with
332 your new build. For example:<screen>%> rm -fr tmp.OLD
334 %> bitbake bootstrap-image</screen></para>
336 <section id="usage_workdir" xreflabel="work directory">
337 <title>work directory (tmp/work)</title>
339 <para>The work directory is where all source code is unpacked into,
340 where source is configured, compiled and packaged. In other words this
341 is where all the action happens. Each bitbake recipe will produce a
342 corresponding sub directory in the work directory. The sub directory
343 name will contain the recipe name, version and the release number (as
344 defined by the PR variable within the recipe).</para>
346 <para>Here's an example of a few of the subdirectories under the work
347 directory:<screen>~%> find tmp/work -maxdepth 1 -type d | head -4
349 tmp/work/busybox-1.2.1-r13
350 tmp/work/libice-1_1.0.3-r0
351 tmp/work/arpwatch-2.1a15-r2</screen>You can see that the first three (of
352 several hundred) recipes here and they are for release 13 of busybox
353 1.2.1, release 0 or libice 1.1.0.3 and release 2 of arpwatch 2.1a15.
354 It's also possible that you may just have a sub directory for your
355 targets architecture and operating system in which case these
356 directories will be in that additional subdirectory, as shown
357 here:<screen>~%> find tmp/work -maxdepth 2 -type d | head -4
360 tmp/work/sh4-linux/busybox-1.2.1-r13
361 tmp/work/sh4-linux/libice-1_1.0.3-r0
362 tmp/work/sh4-linux/arpwatch-2.1a15-r2</screen></para>
364 <para>The <emphasis role="bold">sh4-linux</emphasis> directory in the
365 above example is a combination of the target architecture (sh4) and
366 operating system (linux). This subdirectory has been added by the use of
367 one of OpenEmbedded's many features. In this case it's the
368 <emphasis>multimachine</emphasis> feature which is used to allow builds
369 for multiple targets within the one work directory and can be enabled on
370 a per distribution basis. This feature enables the sharing of native and
371 architecture neutral packages and building for multiple targets that
372 support the same architecture but require different linux kernels (for
373 example). We'll assume multimachine isn't being used for the rest of
374 this chapter, just remember to add the extra directory if your
375 distribution is using it.</para>
377 <para>Using lzo 1.08 as an example we'll examine the contents of the
378 working directory for a typical recipe:<screen>~%> find tmp/work/lzo-1.08-r14 -maxdepth 1
379 tmp/work/lzo-1.08-r14
380 tmp/work/lzo-1.08-r14/temp
381 tmp/work/lzo-1.08-r14/lzo-1.08
382 tmp/work/lzo-1.08-r14/install
383 tmp/work/lzo-1.08-r14/image</screen></para>
385 <para>The directory, <emphasis
386 role="bold">tmp/work/lzo-1.08-r14</emphasis>, is know as the
387 <emphasis>"working directory"</emphasis> for the recipe and is specified
388 via the <emphasis role="bold">WORKDIR</emphasis> variable in bitbake.
389 You'll sometimes see recipes refer directly to <emphasis
390 role="bold">WORKDIR</emphasis> and this is the directory they are
391 referencing. The <emphasis role="bold">1.08</emphasis> is the version of
392 lzo and <emphasis role="bold">r14</emphasis> is the release number, as
393 defined by the <emphasis role="bold">PR</emphasis> variable within the
396 <para>Under the working directory (<emphasis
397 role="bold">WORKDIR</emphasis>) there are four subdirectories:</para>
404 <para>The temp directories contains logs and in some cases scripts
405 that actually implement specific tasks (such as a script to
406 configure or compile the source).</para>
408 <para>You can look at the logs in this directory to get more
409 information into what happened (or didn't happen). This is usually
410 the first thing to look at when things are going wrong and these
411 usually need to be included when reporting bugs.</para>
413 <para>The scripts can be used to see what a particular task, such
414 as configure or compile, is trying to do.</para>
419 <term>lzo-1.08</term>
422 <para>This is the unpacked source code directory, which was
423 created when the lzo source code was extracted in this directory.
424 The name and format of this directory is therefore dependent on
425 the actual source code packaging. Within recipes this directory is
426 referred to as <emphasis role="bold">S</emphasis> and is usually
427 expected to be named like this, that is
428 <emphasis>"<name>-<version>"</emphasis>. If the source
429 code extracts to somewhere else then that would need to be
430 declared in the recipe by explicitly setting the value of the
431 variable <emphasis role="bold">S</emphasis> to the appropriate
440 <para>The image directory (or destination directory) is where the
441 software needs to be installed into in order to be packaged. This
442 directory is referred to as <emphasis role="bold">D</emphasis> in
443 recipes. So instead of installing binaries into <emphasis
444 role="bold">/usr/bin</emphasis> and libraries into <emphasis
445 role="bold">/usr/lib</emphasis> for example you would need to
446 install into <emphasis role="bold">${D}/usr/bin</emphasis> and
447 <emphasis role="bold">${D}/usr/lib</emphasis> instead. When
448 installed on the target the ${D} will be not be included so
449 they'll end up in the correct place. You definitely don't wont
450 files on your host system being replaced by cross-compiled
451 binaries for your target!</para>
459 <para>The install directory is used to split the installed files
460 into separate packages. One subdirectory is created per package to
461 be generated and the files are moved from the image directory
462 (<emphasis role="bold">D</emphasis>) over to this directory, and
463 into the appropriate package subdirectory, as each packaging
464 instruction is processed. Typically there will be separate
465 documentation (<emphasis>-doc</emphasis>), debugging
466 (<emphasis>-dbg</emphasis>) and development
467 (<emphasis>-dev</emphasis>) packages automatically created. There
468 are variables such as <emphasis role="bold">FILES_</emphasis> and
469 <emphasis role="bold">PACKAGES</emphasis> used in recipes which
470 control the separation of various files into individual
476 <para>So lets show some examples of the useful information you now have
479 <para>How about checking out what happened during the configuration of
480 lzo? Well that requires checking the log file for configure that is
481 generated in the temp directory:<screen>~%> less tmp/work/lzo-1.08-r14/temp/log.do_configure.*
483 checking whether ccache sh4-linux-gcc -ml -m4 suffers the -fschedule-insns bug... unknown
484 checking whether ccache sh4-linux-gcc -ml -m4 suffers the -fstrength-reduce bug... unknown
485 checking whether ccache sh4-linux-gcc -ml -m4 accepts -fstrict-aliasing... yes
486 checking the alignment of the assembler... 0
487 checking whether to build assembler versions... no
488 configure: creating ./config.status
489 config.status: creating Makefile
490 config.status: creating examples/Makefile
491 config.status: creating include/Makefile
492 config.status: creating ltest/Makefile
493 config.status: creating minilzo/Makefile
494 config.status: creating src/Makefile
495 config.status: creating tests/Makefile
496 config.status: creating config.h
497 config.status: executing depfiles commands</screen></para>
499 <para>Or perhaps you want to see how the files were distributed into
500 individual packages prior to packaging? The install directory is where
501 the files are split into separate packages and so that shows us which
502 files end up where:<screen>~%> find tmp/work/lzo-1.08-r14/install
503 tmp/work/lzo-1.08-r14/install
504 tmp/work/lzo-1.08-r14/install/lzo-doc
505 tmp/work/lzo-1.08-r14/install/lzo-dbg
506 tmp/work/lzo-1.08-r14/install/lzo-dbg/usr
507 tmp/work/lzo-1.08-r14/install/lzo-dbg/usr/lib
508 tmp/work/lzo-1.08-r14/install/lzo-dbg/usr/lib/.debug
509 tmp/work/lzo-1.08-r14/install/lzo-dbg/usr/lib/.debug/liblzo.so.1.0.0
510 tmp/work/lzo-1.08-r14/install/lzo-dev
511 tmp/work/lzo-1.08-r14/install/lzo-dev/usr
512 tmp/work/lzo-1.08-r14/install/lzo-dev/usr/include
513 tmp/work/lzo-1.08-r14/install/lzo-dev/usr/include/lzo2a.h
514 tmp/work/lzo-1.08-r14/install/lzo-dev/usr/include/lzo1y.h
515 tmp/work/lzo-1.08-r14/install/lzo-dev/usr/include/lzo1.h
516 tmp/work/lzo-1.08-r14/install/lzo-dev/usr/include/lzo1b.h
517 tmp/work/lzo-1.08-r14/install/lzo-dev/usr/include/lzo1f.h
518 tmp/work/lzo-1.08-r14/install/lzo-dev/usr/include/lzoconf.h
519 tmp/work/lzo-1.08-r14/install/lzo-dev/usr/include/lzo1x.h
520 tmp/work/lzo-1.08-r14/install/lzo-dev/usr/include/lzo16bit.h
521 tmp/work/lzo-1.08-r14/install/lzo-dev/usr/include/lzo1a.h
522 tmp/work/lzo-1.08-r14/install/lzo-dev/usr/include/lzo1z.h
523 tmp/work/lzo-1.08-r14/install/lzo-dev/usr/include/lzoutil.h
524 tmp/work/lzo-1.08-r14/install/lzo-dev/usr/include/lzo1c.h
525 tmp/work/lzo-1.08-r14/install/lzo-dev/usr/lib
526 tmp/work/lzo-1.08-r14/install/lzo-dev/usr/lib/liblzo.a
527 tmp/work/lzo-1.08-r14/install/lzo-dev/usr/lib/liblzo.so
528 tmp/work/lzo-1.08-r14/install/lzo-dev/usr/lib/liblzo.la
529 tmp/work/lzo-1.08-r14/install/lzo.shlibdeps
530 tmp/work/lzo-1.08-r14/install/lzo-locale
531 tmp/work/lzo-1.08-r14/install/lzo
532 tmp/work/lzo-1.08-r14/install/lzo/usr
533 tmp/work/lzo-1.08-r14/install/lzo/usr/lib
534 tmp/work/lzo-1.08-r14/install/lzo/usr/lib/liblzo.so.1
535 tmp/work/lzo-1.08-r14/install/lzo/usr/lib/liblzo.so.1.0.0</screen></para>
539 <section id="usage_tasks" xreflabel="tasks">
542 <para>When you go about building and installing a software package there
543 are a number of tasks that you generally follow with most software
544 packages. You probably need to start out by downloading the source code,
545 then unpacking the source code. Maye you need to apply some patches for
546 some reason. Then you might run the configure script of the package,
547 perhaps passing it some options to configure it to your liking. The you
548 might run "make install" to install the software. If your actually going
549 to make some packages, such as .deb or .rpm, then you'd have additional
550 tasks you'd perform to make them.</para>
552 <para>You find that building things in OpenEmbedded works in a similar way
553 - there are a number of tasks that are executed in a predefined order for
554 each recipe. Any many of the tasks correspond to those listed above like
555 <emphasis>"download the source"</emphasis>. In fact you've probably
556 already seen some of the names of these tasks - bitbake displays them as
557 they are processed:<screen>~%> bitbake lzo
558 NOTE: Psyco JIT Compiler (http://psyco.sf.net) not available. Install it to increase performance.
559 NOTE: Handling BitBake files: \ (4541/4541) [100 %]
560 NOTE: Parsing finished. 4325 cached, 0 parsed, 216 skipped, 0 masked.
561 NOTE: build 200705041709: started
563 OE Build Configuration:
565 OE_REVISION = "<unknown>"
570 DISTRO_VERSION = "0.1-20070504"
573 NOTE: Resolving missing task queue dependencies
574 NOTE: preferred version 2.5 of glibc not available (for item virtual/sh4-linux-libc-for-gcc)
575 NOTE: Preparing Runqueue
576 NOTE: Executing runqueue
577 NOTE: Running task 208 of 226 (ID: 11, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, <emphasis
578 role="bold">do_fetch</emphasis>)
579 NOTE: package lzo-1.08: started
580 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_fetch</emphasis>: started
581 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_fetch</emphasis>: completed
582 NOTE: package lzo-1.08: completed
583 NOTE: Running task 209 of 226 (ID: 2, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, <emphasis
584 role="bold">do_unpack</emphasis>)
585 NOTE: package lzo-1.08: started
586 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_unpack</emphasis>: started
587 NOTE: Unpacking /home/lenehan/devel/oe/sources/lzo-1.08.tar.gz to /home/lenehan/devel/oe/build/titan-glibc-25/tmp/work/lzo-1.08-r14/
588 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_unpack</emphasis>: completed
589 NOTE: package lzo-1.08: completed
590 NOTE: Running task 216 of 226 (ID: 3, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, <emphasis
591 role="bold">do_patch</emphasis>)
592 NOTE: package lzo-1.08: started
593 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_patch</emphasis>: started
594 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_patch</emphasis>: completed
595 NOTE: package lzo-1.08: completed
596 NOTE: Running task 217 of 226 (ID: 4, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, <emphasis
597 role="bold">do_configure</emphasis>)
598 NOTE: package lzo-1.08: started
599 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_configure</emphasis>: started
600 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_configure</emphasis>: completed
601 NOTE: package lzo-1.08: completed
602 NOTE: Running task 218 of 226 (ID: 12, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, <emphasis
603 role="bold">do_qa_configure</emphasis>)
604 NOTE: package lzo-1.08: started
605 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_qa_configure</emphasis>: started
606 NOTE: Checking sanity of the config.log file
607 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_qa_configure</emphasis>: completed
608 NOTE: package lzo-1.08: completed
609 NOTE: Running task 219 of 226 (ID: 0, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, <emphasis
610 role="bold">do_compile</emphasis>)
611 NOTE: package lzo-1.08: started
612 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_compile</emphasis>: started
613 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_compile</emphasis>: completed
614 NOTE: package lzo-1.08: completed
615 NOTE: Running task 220 of 226 (ID: 1, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, <emphasis
616 role="bold">do_install</emphasis>)
617 NOTE: package lzo-1.08: started
618 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_install</emphasis>: started
619 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_install</emphasis>: completed
620 NOTE: package lzo-1.08: completed
621 NOTE: Running task 221 of 226 (ID: 5, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, <emphasis
622 role="bold">do_package</emphasis>)
623 NOTE: package lzo-1.08: started
624 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_package</emphasis>: started
626 NOTE: Checking Package: lzo-dbg
627 NOTE: Checking Package: lzo
628 NOTE: Checking Package: lzo-doc
629 NOTE: Checking Package: lzo-dev
630 NOTE: Checking Package: lzo-locale
631 NOTE: DONE with PACKAGE QA
632 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_package</emphasis>: completed
633 NOTE: package lzo-1.08: completed
634 NOTE: Running task 222 of 226 (ID: 8, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, <emphasis
635 role="bold">do_package_write</emphasis>)
636 NOTE: package lzo-1.08: started
637 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_package_write</emphasis>: started
638 Packaged contents of lzo-dbg into /home/lenehan/devel/oe/build/titan-glibc-25/tmp/deploy/ipk/sh4/liblzo-dbg_1.08-r14_sh4.ipk
639 Packaged contents of lzo into /home/lenehan/devel/oe/build/titan-glibc-25/tmp/deploy/ipk/sh4/liblzo1_1.08-r14_sh4.ipk
640 NOTE: Not creating empty archive for lzo-doc-1.08-r14
641 Packaged contents of lzo-dev into /home/lenehan/devel/oe/build/titan-glibc-25/tmp/deploy/ipk/sh4/liblzo-dev_1.08-r14_sh4.ipk
642 NOTE: Not creating empty archive for lzo-locale-1.08-r14
643 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_package_write</emphasis>: completed
644 NOTE: package lzo-1.08: completed
645 NOTE: Running task 223 of 226 (ID: 6, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, do_populate_staging)
646 NOTE: package lzo-1.08: started
647 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_populate_staging</emphasis>: started
648 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_populate_staging</emphasis>: completed
649 NOTE: package lzo-1.08: completed
650 NOTE: Running task 224 of 226 (ID: 9, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, do_qa_staging)
651 NOTE: package lzo-1.08: started
652 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_qa_staging</emphasis>: started
653 NOTE: QA checking staging
654 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_qa_staging</emphasis>: completed
655 NOTE: package lzo-1.08: completed
656 NOTE: Running task 225 of 226 (ID: 7, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, do_distribute_sources)
657 NOTE: package lzo-1.08: started
658 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_distribute_sources</emphasis>: started
659 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_distribute_sources</emphasis>: completed
660 NOTE: package lzo-1.08: completed
661 NOTE: Running task 226 of 226 (ID: 10, /home/lenehan/devel/oe/build/titan-glibc-25/packages/lzo/lzo_1.08.bb, do_build)
662 NOTE: package lzo-1.08: started
663 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_build</emphasis>: started
664 NOTE: package lzo-1.08-r14: task <emphasis role="bold">do_build</emphasis>: completed
665 NOTE: package lzo-1.08: completed
666 NOTE: Tasks Summary: Attempted 226 tasks of which 213 didn't need to be rerun and 0 failed.
667 NOTE: build 200705041709: completed</screen><note>
668 <para>The output may look different depending on the version of
669 bitbake being used, and some tasks are only run when specific options
670 are enabled in your distribution. The important point to note is that
671 the various tasks are being run and bitbake shows you each time it
672 starts and completes a task.</para>
675 <para>So there's a set of tasks here which are being run to generate the
676 final packages. And if you'll notice that every recipe runs through the
677 same set of tasks (ok I'll admit that it is possible that some additional
678 tasks could be run for some recipes, but we'll talk about that later). The
679 tasks that you'll need to be most familiar with are:</para>
686 <para>The <emphasis>fetch</emphasis> task is responsible for
687 fetching any source code that is required. This means things such as
688 downloading files and checking out from source control repositories
689 such as git or svn.</para>
697 <para>The <emphasis>unpack</emphasis> task is responsible for
698 extracting files from archives, such as <emphasis
699 role="bold">.tar.gz</emphasis>, into the working area and copying
700 any additional files, such as init scripts, into the working
709 <para>The <emphasis>patch</emphasis> task is responsible for
710 applying any patches to the unpacked source code</para>
715 <term>configure</term>
718 <para>The <emphasis>configure</emphasis> task takes care of the
719 configuration of the package. Running a configure script
720 (<emphasis>"./configure <options>"</emphasis>) is probably the
721 form of configuration that is most recognised but it's not the only
722 configuration system that exists.</para>
730 <para>The <emphasis>compile</emphasis> task actually compiles the
731 software. This could be as simple as running <emphasis
732 role="bold">make</emphasis>.</para>
737 <term>populate_staging (stage)</term>
740 <para>The <emphasis>populate_staging</emphasis> task (stage is an
741 alternate, easier to type name, that can be used to refer to this
742 task) is responsible for making available libraries and headers (if
743 any) that may be required by other packages to build. For example if
744 you compile zlib then it's headers and the library need to be made
745 available for other applications to include and link against.</para>
748 <para>This is different to the <emphasis>install</emphasis> task
749 in that this is responsible for making available libraries and
750 headers for use during build on the development host. Therefore
751 it's libraries which normal have to stage things while
752 applications normally don't need to. The
753 <emphasis>install</emphasis> task on the other hand is making
754 files available for packaging and ultimately installation on the
764 <para>The <emphasis>install</emphasis> task is responsible for
765 actually installing everything. Now this needs to install the
766 software into the destination directory, <emphasis
767 role="bold">D</emphasis>. This directory won't actually be a part of
768 the final package though. In other words if you install something
769 into <emphasis role="bold">${D}/bin</emphasis> then it will end up
770 in the <emphasis role="bold">/bin</emphasis> directory in the
771 package and therefore on the target.</para>
779 <para>The <emphasis>package</emphasis> task takes the installed
780 files and splits them into separate directories under the <emphasis
781 role="bold">${WORKDIR}/install</emphasis> directory, one per
782 package. It moves the files for the destination directory, <emphasis
783 role="bold">${D}</emphasis>, that they were installed in into the
784 appropriate packages subdirectory. Usually there will be a main
785 package a separate documentation (-doc), development (-dev) and
786 debugging packages (-dbg) for example.</para>
791 <term>package_write</term>
794 <para>The <emphasis>package_write</emphasis> task is responsible for
795 taking each packages subdirectory and creating any actual
796 installation package, such as .ipk, .deb or .rpm. Currently .ipk is
797 the only fully supported packing format although .deb packages are
798 being actively worked on. It should be reasonably easy for an
799 experienced OpenEmbedded developer to add support for any other
800 packaging formats they might required.</para>
806 <para>You'll notice that the bitbake output had tasks prefixed with
807 <emphasis>do_</emphasis>, as in <emphasis>do_install</emphasis> vs
808 <emphasis>install</emphasis>. This is slightly confusing but any task
809 <emphasis>x</emphasis> is implemented via a function called
810 <emphasis>do_x</emphasis> in the class or recipe where it is defined.
811 See places refer to the tasks via their name only and some with the
812 <emphasis>do</emphasis> prefix.</para>
815 <para>You will almost certainly notice tasks beyond these ones - there are
816 various methods available to insert additional tasks into the tasks
817 sequence. As an example the <emphasis
818 role="bold">insane.bbclass</emphasis>, which performs various QA checks,
819 does these checks by inserting a new task called
820 <emphasis>qa_configure</emphasis> between the
821 <emphasis>configure</emphasis> and <emphasis>compile</emphasis> tasks and
822 another new task called <emphasis>qa_staging</emphasis> between
823 <emphasis>populate_staging</emphasis> and <emphasis>build</emphasis>
824 tasks. The former validates the result of the
825 <emphasis>configure</emphasis> task and the late the results of the
826 <emphasis>populate_staging</emphasis> task.</para>
828 <para>To determine the full list of tasks available for a specific recipe
829 you can run bitbake on the recipe and asking it for the full list of
830 available tasks:<screen>~%> bitbake -b packages/perl/perl_5.8.8.bb -c listtasks
831 NOTE: package perl-5.8.8: started
832 NOTE: package perl-5.8.8-r11: task do_listtasks: started
847 do_distribute_sources
852 NOTE: package perl-5.8.8-r11: task do_listtasks: completed
853 NOTE: package perl-5.8.8: completed
854 ~%> </screen></para>
856 <para>If your being observant you'll note that
857 <emphasis>listtasks</emphasis> is in fact a task itself, and that the
858 <emphasis role="bold">-c</emphasis> option to bitbake allows you to
859 explicitly run specific tasks. We'll make use of this in the next section
860 when we discuss working with a recipe.</para>
863 <section id="usage_workwithsinglerecipe"
864 xreflabel="working with a single recipe">
865 <title>Working with a single recipe</title>
867 <para>During development you're likely to often find yourself working on a
868 single bitbake recipe - maybe trying to fix something or add a new version
869 or perhaps working on a totally new recipe. Now that you know all about
870 tasks you can use that knowledge to help speed up the development and
871 debugging process.</para>
873 <para>Bitbake can be instructed to deal directly with a single recipe file
874 by passing it via the <emphasis role="bold">-b</emphasis> parameter. This
875 option takes the recipe as a parameter and instructs bitbake to process
876 the named recipe only. Note that this ignores any dependencies that are in
877 the recipe, so these must have already been built previously.</para>
879 <para>Here's a typically example that cleans up the package (using the
880 <emphasis>clean</emphasis> task) and the rebuilds it with debugging output
881 from bitbake enabled:<screen>~%> bitbake -b <bb-file> -c clean
882 ~%> bitbake -b <bb-file> -D</screen></para>
884 <para>The options to bitbake that are most useful here are:</para>
888 <term>-b <bb-file></term>
891 <para>The recipe to process;</para>
896 <term>-c <action></term>
899 <para>The action to perform, typically the name of one of the tasks
900 supported by the recipe;</para>
908 <para>Display debugging information, use two <emphasis
909 role="bold">-D</emphasis>'s for additional debugging;</para>
917 <para>Force an operation. This is useful in getting bitbake to
918 perform some operation it normally wouldn't do. For example, if you
919 try and call the <emphasis>compile</emphasis> task twice in a row
920 then bitbake will not do anything on the second attempt since it has
921 already performed the task. By adding <emphasis
922 role="bold">-f</emphasis> it will force it to perform the action
923 regardless of if it thinks it's been done previously.</para>
928 <para>The most common actions (used with -c) are:</para>
935 <para>Try to download all of the required source files, but don't do
936 anything else with them.</para>
944 <para>Unpack the source file but don't apply the patches yet.
945 Sometimes you may want to look at the extracted, but not patched
946 source code and that's what just unpacking will give you (some
947 time's handy to get diffs generated against the original
956 <para>Apply any patches.</para>
961 <term>configure</term>
964 <para>Performs and configuration that is required for the
973 <para>Perform the actual compilation steps of the software.</para>
981 <para>If any files, such as header and libraries, will be required
982 by other packages then they need to be installed into the staging
983 area and that's what this task takes care of.</para>
991 <para>Install the software in preparation for packaging.</para>
999 <para>Package the software. Remember that this moves the files from
1000 the installation directory, D, into the packing install area. So to
1001 re-package you also need to re-install first.</para>
1009 <para>Delete the entire directory for this version of the software.
1010 Usually done to allow a test build with no chance of old files or
1011 changes being left behind.</para>
1016 <para>Note that each of the actions that corresponds to task's will run
1017 any preceding tasks that have not yet been performed. So starting with
1018 compile will also perform the fetch, unpack, patch and configure
1021 <para>A typically development session might involve editing files in the
1022 working directory and then recompiling until it all works:<screen>[<emphasis>... test ...</emphasis>]
1023 ~%> bitbake -b packages/testapp/testapp_4.3.bb -c compile -D
1025 [<emphasis>... save a copy of main.c and make some changes ...</emphasis>]
1026 ~%> vi tmp/work/testapp-4.3-r0/main.c
1027 ~%> bitbake -b packages/testapp/testapp_4.3.bb -c compile -D -f
1029 [<emphasis>... create a patch and add it to the recipe ...</emphasis>]
1030 ~%> vi packages/testapp/testapp_4.3.bb
1032 [<emphasis>... test from clean ...</emphasis>]
1033 ~%> bitbake -b packages/testapp/testapp_4.3.bb -c clean
1034 ~%> bitbake -b packages/testapp/testapp_4.3.bb
1036 [<emphasis>... NOTE: How to create the patch is not covered at this point ...</emphasis>]</screen></para>
1038 <para>Here's another example showing how you might go about fixing up the
1039 packaging in your recipe:<screen>~%> bitbake -b packages/testapp/testapp_4.3.bb -c install -f
1040 ~%> bitbake -b packages/testapp/testapp_4.3.bb -c stage -f
1041 ~%> find tmp/work/testapp_4.3/install
1043 ~%> vi packages/testapp/testapp_4.3.bb</screen>At this stage you play with
1044 the <emphasis role="bold">PACKAGE_</emphasis> and <emphasis
1045 role="bold">FILES_</emphasis> variables and then repeat the above
1048 <para>Note how we install and then stage. This is one of those things
1049 where understanding the tasks helps a lot! Remember that stage moves the
1050 files from where they were installed into the various subdirectories
1051 (under <emphasis role="bold">${WORKDIR}/instal</emphasis>l) for each
1052 package. So if you try and run a stage task without a prior install there
1053 won't be any files there to stage! Note also that the stage tasks clears
1054 all the subdirectories in <emphasis
1055 role="bold">${WORKDIR}/install</emphasis> so you won't get any left over
1056 files. But beware, the install task doesn't clear <emphasis
1057 role="bold">${D}</emphasis> directory, so any left over files from a
1058 previous packing attempt will be left behind (which is ok if all you care
1059 about it staging).</para>
1062 <section id="usage_interactive_bitbake" xreflabel="interactive bitbake">
1063 <title>Interactive bitbake</title>
1065 <para>To interactively test things use:<screen>~%> bitbake -i</screen>this
1066 will open the bitbake shell. From here there are a lot of commands
1067 available (try help).</para>
1069 <para>First thing you will want to do is parse all of the recipes (recent
1070 bitbake version do this automatically when needed, so you don't need to
1071 manually do this anymore):<screen>BB>> parse</screen>You can now
1072 build a specific recipe:<screen>BB>> build net-snmp</screen>If it
1073 fails you may want to clean the build before trying again:<screen>BB>> clean net-snmp</screen>If
1074 you update the recipe by editing the .bb file (to fix some issues) then
1075 you will want to clean the package, reparse the modified recipe, and the
1076 build again:<screen>BB>> clean net-snmp
1077 BB>> reparse net-snmp
1078 BB>> build net-snmp</screen>Note that you can use wildcards in the
1079 bitbake shell as well:<screen>BB>> build t*</screen></para>
1084 <section id="usage_devshell" xreflabel="devshell">
1085 <title>Devshell</title>
1087 <para>One of the areas in which OpenEmbedded helps you out is by setting
1088 various environment variables, such as <emphasis role="bold">CC</emphasis>
1089 and <emphasis role="bold">PATH</emphasis> etc, to values suitable for
1090 cross-compiling. If you wish to manually run configure scripts and compile
1091 file during development it would be nice to have all those values set for
1092 you. This is what devshell does - it provides you with an interactive
1093 shell with all the appropriate variables set for cross-compiling.</para>
1096 <title>devshell via inherit</title>
1098 <para>This is the newer method of obtaining a devshell and is the
1099 recommended way for most users now. The newer method requires that the
1100 devshell class be added to you configuration by inheriting it. This is
1101 usually done in your <emphasis role="bold">local.conf</emphasis> or your
1102 distributions conf file:<screen><emphasis role="bold">INHERIT +=</emphasis> "src_distribute_local insane multimachine <emphasis
1103 role="bold">devshell</emphasis>"</screen></para>
1105 <para>With the inclusion of this class you'll find that devshell is
1106 added as a new task that you can use on recipes:<screen>~%> bitbake -b packages/lzo/lzo_1.08.bb -c listtasks
1107 NOTE: package lzo-1.08: started
1108 NOTE: package lzo-1.08-r14: task do_listtasks: started
1109 <emphasis role="bold">do_devshell</emphasis>
1124 do_distribute_sources
1129 NOTE: package lzo-1.08-r14: task do_listtasks: completed
1130 NOTE: package lzo-1.08: completed</screen></para>
1132 <para>To bring up the devshell you call bitbake on a recipe and ask it
1133 for the devshell task:<screen>~%> ./bb -b packages/lzo/lzo_1.08.bb -c devshell
1134 NOTE: package lzo-1.08: started
1135 NOTE: package lzo-1.08-r14: task do_devshell: started
1136 [<emphasis>... devshell will appear here ...</emphasis>]
1137 NOTE: package lzo-1.08-r14: task do_devshell: completed
1138 NOTE: package lzo-1.08: completed</screen></para>
1140 <para>How the devshell appears depends on the settings of the <emphasis
1141 role="bold">TERMCMD</emphasis> variable - you can see the default
1142 settings and other possible values in <emphasis
1143 role="bold">conf/bitbake.conf</emphasis>. Feel free to try settings this
1144 to something else in your local.conf. Usually you will see a new
1145 terminal window open which is the devshell window.</para>
1147 <para>The devshell task is inserted after the patch task, so if you have
1148 not already run bitbake on the recipe it will download the source and
1149 apply any patches prior to opening the shell.</para>
1152 <para>This method of obtaining a devshell works if you using <emphasis
1153 role="bold">bash</emphasis> as your shell, it does not work if you are
1154 using <emphasis role="bold">zsh</emphasis> as your shell. Other shells
1155 may or may not work.</para>
1160 <title>devshell addon</title>
1162 <para>The devshell addon was the original method that was used to create
1165 <para>It requires no changes to your configuration, instead you simply
1166 build the devshell recipe:<screen>bitabike devshell</screen></para>
1168 <para>and then manually startup the shell. Once in the shell you'll
1169 usually want to change into the working directory for the recipe you are
1170 working on:<screen>~%> ./tmp/deploy/addons/sh4-linux-erouter-titan-devshell
1171 bash: alias: `./configure': invalid alias name
1172 [OE::sh4-linux-erouter-titan]:~$ cd tmp/work/lzo-1.08-r14/lzo-1.08
1173 [OE::sh4-linux-erouter-titan]:~tmp/work/lzo-1.08-r14/lzo-1.08$</screen><note>
1174 <para>The name of the devshell addon depends on the target
1175 architecture, operating system and machine name. So you name will be
1176 different - just check for the appropriate name ending in
1182 <title>Working in the devshell</title>
1184 <para>[To be done]</para>
1188 <section id="usage_patches" xreflabel="patching">
1189 <title>Patching and patch management</title>
1191 <para>[To be done]</para>
git.openpandora.org / openembedded.git / blob