[Top] [Prev] [Next] [Bottom]

12 Script Libraries and Packages

Collections of Tcl commands are kept in libraries and organized into packages. Tcl automatically loads libraries as an application uses their commands. Tcl commands: package, pkg_mkIndex, auto_mkIndex, and unknown.

Libraries group useful sets of Tcl procedures so they can be used by multiple applications. For example, you could use any of the code examples that come with this book by creating a script library and then directing your application to check in that library for missing procedures. One way to structure a large application is to have a short main script and a library of support scripts. The advantage of this approach is that not all the Tcl code needs to be loaded to get the application started. Applications start up quickly, and as new features are accessed the code that implements them is loaded automatically.

A package facility was added in Tcl 7.5. It supports version numbers and has a provide/require model of use. Typically each file in a library provides one package with a particular version number. Packages also work with shared object libraries that implement Tcl commands in compiled code, which are described on page 523. A package can be provided by a combination of script files and object files. Applications specify which packages they require and the libraries are loaded automatically. The package facility is an alternative to the auto loading scheme used in earlier versions of Tcl. You can use either mechanism, and this chapter describes them both.

If you create a package you may which to use the namespace facility to avoid conflicts between procedures and global variables used in different packages. Namespaces are the topic of Chapter 14. Before Tcl 8.0 you had to use your own conventions to avoid conflicts. This chapter explains a simple coding convention for large Tcl programs. I use this convention in exmh, a mail user interface that has grown from about 2,000 to over 25,000 lines of Tcl code. Easily half of the code has been contributed by the exmh user community. Such growth might not have been possible without a module system.

Locating Packages: The auto_path Variable

The package facility assumes that Tcl libraries are kept in well-known directories. The list of well-known directories is kept in the auto_path Tcl variable. This is initialized by tclsh and wish to include the Tcl script library directory, the Tk script library directory (for wish), and the parent directory of the Tcl script library directory. For example, on my Macintosh auto_path is a list of these three directories:

Disk:System Folder:Extensions:Tool Command Language:tcl7.6

Disk:System Folder:Extensions:Tool Command Language

Disk:System Folder:Extensions:Tool Command Language:tk4.2

On my Windows 95 machine the auto_path lists these directories:

c:\Program Files\Tcl\lib\Tcl7.6

c:\Program Files\Tcl\lib

c:\Program Files\Tcl\lib\Tk4.2

On my UNIX workstation the auto_path lists these directories:

/usr/local/tcl/lib/tcl7.6

/usr/local/tcl/lib

/usr/local/tcl/lib/tk4.2

The package facility searches these directories and their subdirectories for packages. The easiest way to manage your own packages is to create a directory at the same level as the Tcl library. Packages in this location, for example, will be found automatically because the auto_path list includes /usr/local/tcl/lib:

/usr/local/tcl/lib/welchbook

You can also add directories to the auto_path explicitly:

lappend auto_path directory

Using Packages

Each script file in a library declares what package it implements with the package provide command:

package provide name version

The name identifies the package, and the version has a major.minor format. The convention is that the minor version number can change and the package implementation will still be compatible. If the package changes in an incompatible way, then the major version number should change. For example, Chapter 16 defines several procedures that use the HTTP network protocol. These include Http_Open, Http_Get, and Http_Validate. The file that contains the procedures starts with this command:

package provide Http 1.0

More than one file can contribute to the same package simply by specifying the same name and version. In addition, different versions of the same package can be kept in the same directory but in different files.

An application specifies the packages it needs with the package require command:

package require name ?version? ?-exact?

If the version is left off, then the highest available version is loaded. Otherwise the highest version with the same major number is loaded. For example, if the client requires version 1.1, version 1.2 could be loaded if it exists, but version 1.0 would not be loaded. You can restrict the package to a specific version with the -exact flag. If no matching version can be found, then the package require command raises an error.

Loading Packages Automatically

The package require command does not load procedures directly. Instead, the commands implemented by a package are loaded into your application when they are used the first time. The automatic loading depends on an index to record which files implement which packages. The index must be maintained by you, your project librarian, or your system administrator when packages change. The index is computed by a Tcl procedure, pkg_mkIndex, and the results are put into the pkgIndex.tcl file in each library directory. The pkg_mkIndex command takes the name of a directory and one or more glob patterns that specify files within that directory. File name patterns are described on page 106. For example:

pkg_mkIndex /usr/local/lib/welchbook *.tcl

The pkg_mkIndex command sources all the files matched by the pattern, detects what packages they provide, and computes the index. You should be aware of this behavior because it only works well for libraries. If the pkg_mkIndex command hangs or starts random applications, it is because it sourced an application file instead of a library file.

Packages Implemented in C Code

The files in a library can be either script files that define Tcl procedures, or binary files in shared library format that define Tcl commands in compiled code (i.e., a Dynamic Link Library (DLL)). Chapter 41 describes how to implement Tcl commands in C. There is a C API to the package facility that you use to declare the package name for your commands. This is shown in Example 41-1 on page 523. Chapter 37 also describes the Tcl load command that is used instead of source to link in shared libraries. The pkg_mkIndex command also handles shared libraries:

pkg_mkIndex directory *.tcl *.so *.shlib *.dll

In this example, .so, .shlib, and .dll are file suffixes for shared libraries on UNIX, Macintosh, and Windows systems, respectively. You can have packages that have some of their commands implemented in C, and some implemented as Tcl procedures. The script files and the shared library just have to declare that they implement the same package. The pkg_mkIndex procedure will detect this and set up the auto_index so some commands are defined by sourcing scripts, and some are defined by loading shared libraries.

If your file servers support more than one machine architecture, such as Solaris and Linux systems, you probably keep the shared library files in machine-specific directories. In this case the auto_path should also list the machine-specific directory so the shared libraries there can be loaded automatically. If your system administrator configured the Tcl installation properly, this should already be set up. If not, or you have your shared libraries in a non-standard place, you must append the location to the auto_path variable.

The package Command

The package command has several operations that are used primarily by the pkg_mkIndex procedure and the automatic loading facility. These operations are summarized in Table 12-1. The basic structure of package loading works like this:

An application does a package require command.
The package facility checks to see if it knows about the package. If it does, then it sets up the auto_index so the commands in the package will be automatically loaded.
If the package is unknown, the tclPkgUnknown procedure is called to find it. Actually, you can specify what procedure to call to do the lookup with the package unknown command, but the standard one is tclPkgUnknown.
The tclPkgUnknown procedure looks through the auto_path directories, and their subdirectories for pkgIndex.tcl files. It sources those to build an internal database of packages and version information. The pkgIndex.tcl files contain calls to package ifneeded that specify what to do to set up automatic loading of the package. The standard action is to call the tclPkgSetup procedure.

The tclPkgSetup procedure defines the auto_index array to contain the correct source or load commands to define each command in the package. Automatic loading and the auto_index array are described in more detail later

The package command.
package forget package	Delete registration information for package.
package ifneeded package ?command?	Query or set the command used to set up automatic loading of a package.
package names	Return the set of registered packages.
package provide package version	Declare that a script file defines commands for package with the given version.
package require package ?version? ?-exact?	Declare that a script uses commands from package. The -exact flag specifies that the exact version must be loaded. Otherwise the highest matching version is loaded.
package unknown ?command?	Query or set the command used to locate packages.
package vcompare v1 v2	Compare version v1 and v2. Returns 0 if they are equal, -1 if v1 is less than v2, or 1 if v1 is greater than v2.
package versions package	Returns which versions of the package are registered.
package vsatisfies v1 v2	Returns 1 if v1 is greater or equal to v2 and still has the same major version number. Otherwise returns 0.

Libraries Based on the tclIndex File

You can create libraries without using the package command. The basic idea is that a directory has a library of script files, and an index of the Tcl commands defined in the library is kept in a tclIndex file. The drawback is that versions are not supported and you may need to adjust the auto_path to list your library directory. The main advantage of this approach is that this mechanism has been part of Tcl since the earliest versions. If you currently maintain a library using tclIndex files, it will still work.

When you create a script library without packages, you must generate the index that records what procedures are defined in the library. The auto_mkindex procedure creates the index, which is stored in a file named tclIndex that is kept in the script library directory. Suppose all the examples from this book are in the directory /usr/local/tcl/welchbook. You can make the examples into a script library by creating the tclIndex file:

auto_mkindex /usr/local/tcl/welchbook *.tcl

You will need to update the tclIndex file if you add procedures or change any of their names. A conservative approach to this is shown in the next example. It is conservative because it re-creates the index if anything in the library has changed since the tclIndex file was last generated, whether or not the change added or removed a Tcl procedure.

Maintaining a tclIndex file.

proc Library_UpdateIndex { libdir } {

	set index [file join $libdir tclIndex]

	if ![file exists $index] {

		set doit 1

	} else {

		set age [file mtime $index]

		set doit 0

		# Changes to directory may mean files were deleted

		if {[file mtime $libdir] > $age} {

			set doit 1

		} else {

			# Check each file for modification

			foreach file [glob [file join $libdir *.tcl]] {

				if {[file mtime $file] > $age} {

					set doit 1

					break

	if { $doit } {

		auto_mkindex $libdir *.tcl

Tcl uses the auto_path variable to record a list of directories to search for unknown commands. To continue our example, you can make the procedures in the book examples available by putting this command at the beginning of your scripts:

lappend auto_path /usr/local/tcl/welchbook

This has no effect if you have not created the tclIndex file. If you want to be extra careful, you can call Library_UpdateIndex. This will update the index if you add new things to the library.

lappend auto_path /usr/local/tcl/welchbook

Library_UpdateIndex /usr/local/tcl/welchbook

This will not work if there is no tclIndex file at all because Tcl won't be able to find the implementation of Library_UpdateIndex. Once the tclIndex has been created for the first time, then this will ensure that any new procedures added to the library will be installed into tclIndex. In practice, if you want this sort of automatic update it is wise to include something like the Library_UpdateIndex procedure directly into your application as opposed to loading it from the library it is supposed to be maintaining.

The unknown Command

Automatic loading of Tcl commands is implemented by the unknown command. Whenever the Tcl interpreter encounters a command that it does not know about, it calls the unknown command with the name of the missing command. The unknown command is implemented in Tcl, so you are free to provide your own mechanism to handle unknown commands. This chapter describes the behavior of the default implementation of unknown, which can be found in the init.tcl file in the Tcl library. The location of the library is returned by the info library command. In order to bootstrap the library facility, the Tcl shells (tclsh and wish) invoke the following Tcl command:

source [file join [info library] init.tcl]

How Auto Loading Works

The unknown command uses an array named auto_index. One element of the array is defined for each procedure that can be automatically loaded. The auto_index array is initialized by the package mechanism or by tclIndex files. The value of an auto_index element is a command that defines the procedure. Typical commands are:

source [file join $dir bind_ui.tcl]

load [file join $dir mime.so] Mime

The $dir gets substituted with the name of the directory that contains the library file, so the result is a source or load command that defines the missing Tcl command. The substitution is done with eval, so you could initialize auto_index with any commands at all. Example 12-2 is a simplified version of the code that reads the tclIndex file.

Loading a tclIndex file.

# This is a simplified part of the auto_load command.

# Go through auto_path from back to front.

set i [expr [llength $auto_path]-1]

for {} {$i >= 0} {incr i -1} {

	set dir [lindex $auto_path $i]

	if [catch {open [file join $dir tclIndex]} f] {

		# No index

		continue

	# eval the file as a script. Because eval is

	# used instead of source, an extra round of

	# substitutions is performed and $dir gets expanded

	# The real code checks for errors here.

	eval [read $f]

	close $f

Disabling the Library Facility: auto_noload

If you do not want the unknown procedure to try and load procedures, you can set the auto_noload variable to disable the mechanism:

set auto_noload anything

Interactive Conveniences

The unknown command provides a few other conveniences. These are only used when you are typing commands directly. They are disabled once execution enters a procedure or if the Tcl shell is not being used interactively. The convenience features are automatic execution of programs, command history, and command abbreviation. These options are tried, in order, if a command implementation cannot be loaded from a script library.

Auto Execute

The unknown procedure implements a second feature: automatic execution of external programs. This makes a Tcl shell behave more like other UNIX shells that are used to execute programs. The search for external programs is done using the standard PATH environment variable that is used by other shells to find programs. If you want to disable the feature all together, set the auto_noexec variable:

set auto_noexec anything

History

The history facility described in Chapter 13 is implemented by the unknown procedure.

Abbreviations

If you type a unique prefix of a command, unknown recognizes it and executes the matching command for you. This is done after automatic program execution is attempted and history substitutions are performed.

Tcl Shell Library Environment

It may help to understand how the Tcl shells initialize their library environment. The first toehold on the environment is made when the shells are compiled. At that point the default pathname of the library directory is defined. For Tcl, this pathname is put into the tcl_library variable. This value is also returned by the info library command for backward compatibility with early versions of Tcl:

info library

As of Tcl 7.6, another variable is also defined when Tcl is compiled. The tcl_packagePath variable is defined and is used as the initial value of the auto_path variable. It contains a list of the Tcl script library directory, its parent directory, and the directory containing compiled shared libraries. Changing tcl_packagePath has no effect after auto_path is initialized.

A Tcl shell initializes itself by sourcing init.tcl:

source [file join $tcl_library init.tcl]

The primary thing defined by init.tcl is the implementation of the unknown procedure. It also initializes auto_path from the value of tcl_packagePath.

The Tk library pathname is defined by the tk_library variable. For Tk, wish also does this:

source [file join $tk_library tk.tcl]

This initializes the scripts that support the Tk widgets. There are still more scripts, and they are organized as a library. So, the tk.tcl script sets up the auto_path variable so the Tk script library is accessible. It does this:

lappend auto_path $tk_library

To summarize, the bootstrap works as follows:

The Tcl C library defines the pathname stored in the $tcl_library variable. This default can be overridden with the TCL_LIBRARY environment variable.
The Tcl C library defines the tcl_packagePath variable, and this default can be overridden with the TCL_PACKAGE_PATH environment variable.
The Tcl interpreter sources init.tcl to define the unknown command and initialize auto_path from tcl_packagePath.
The Tk C library defines a pathname and stores it into tk_library, a Tcl variable. The default can be overridden with the TK_LIBRARY environment variable.
The Tk interpreter sources init.tcl as above, and tk.tcl.
The Tk initialization script appends $tk_library to auto_path. Normally these details are taken care of by the proper installation of the Tcl and Tk software, but I find it helps to understand things when you see all the steps in the initialization process. If you have only a binary distribution of Tcl and Tk (e.g., a Linux shared library) you may need to adjust TCL_LIBRARY and TK_LIBRARY to reflect the location of the libraries on your system.

Coding Style

If you supply a package, you need to follow some simple coding conventions to make your library easier to use by other programmers. You can use the namespace facility introduced in Tcl 8.0. You can also just use some conventions to avoid name conflicts with other library packages and the main application. This section describes the conventions I developed before namespaces were added to Tcl.

A Module Prefix for Procedure Names

The first convention is to choose an identifying prefix for the procedures in your package. For example, the preferences package in Chapter 39 uses Pref as its prefix. All the procedures provided by the library begin with Pref. This convention is extended to distinguish between private and exported procedures. An exported procedure has an underscore after its prefix, and it is acceptable to call this procedure from the main application or other library packages. Examples include Pref_Add, Pref_Init, and Pref_Dialog. A private procedure is meant for use only by the other procedures in the same package. Its name does not have the underscore. Examples include PrefDialogItem and PrefXres.

This naming convention precludes casual names like doit, setup, layout, and so on. There is no way to hide procedure names, so you must maintain the naming convention for all procedures in a package.

A Global Array for State Variables

You should use the same prefix on the global variables used by your package. You can alter the capitalization, just keep the same prefix. I capitalize procedure names and use lowercase for variables. By sticking with the same prefix you identify what variables belong to the package and you avoid conflict with other packages.

Collect state in a global array.

In general I try to use a single global array for a package. The array provides a convenient place to collect a set of related variables, much as a struct is used in C. For example, the preferences package uses the pref array to hold all its state information. It is also a good idea to keep the use of the array private. It is better coding practice to provide exported procedures than to let other modules access your data structures directly. This makes it easier to change the implementation of your package without affecting its clients.

If you do need to export a few key variables from your module, use the underscore convention to distinguish exported variables. If you need more than one global variable, just stick with the prefix convention to avoid conflicts.

[Top] [Prev] [Next] [Bottom]

welch@acm.org