Developing for SAGE

This talk consists of three parts, it is about certain aspects you may encounter while extending SAGE.
The first part deals with the documentation, its organisation and how to modify it.
The second part deals with the conventions and technical aspects of SAGE package creation.
The last part covers different smaller topics like how to use tools intended for Python with SAGE.

The first and second part are a summary of the SAGE Programming Guide, Chapter 2, 3 ,4.

$SAGE_ROOT is an environment variable pointing to the root directory of the SAGE installation. At home it points to /usr/local/sage, on linserv it points to /usr/local/share/sage-2.8.15.

1.) Part: SAGE Documentation

The documentation is stored in $SAGE_ROOT/devel/doc. There are sub-directories for the Tutorial (tut), the Programming Guide (prog), the Constructions (const), the Installation Guide (inst), and the Reference Manual (ref). The first four are handwritten LaTeX files, but the Reference Manual is created automatically from a manual selection.

1.1.) The Documentation is Version Controlled

SAGE comes with a full version control system called Mercurial and the $SAGE_ROOT/devel/doc directory is version controlled. So is any modification to the documentation: you can access repository functions from the hg_doc object within SAGE.

At first we need to configure Mercurial with our user-name. Create the file ~/.hgrc with (hg: the chemical symbol for mercury)

[ui]
username = UserName <EMail-Adress>

The Programming Guide, Chapter 4.4 has an example session. 

sage: hg_doc.pull()
cd "/home/jaap/sage/devel/doc" && hg status
cd "/home/jaap/sage/devel/doc" && hg status
cd "/home/jaap/sage/devel/doc" && hg pull -u http://sagemath.org/sage//hg//doc-main
pulling from http://sagemath.org/sage//hg//doc-main
searching for changes
adding changesets
adding manifests
adding file changes
added 3 changesets with 3 changes to 1 files
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
If it says use 'hg merge' above, then you should
type hg_sage.merge(), where hg_sage is the name
of the repository you are using.  This might not
work with the notebook yet.
hg_doc.pull() updates your local repository. After modifying some file you can commit the changes: 
sage: hg_doc.commit()
cd "/home/jaap/downloads/sage-1.6/devel/doc" && hg diff  | less
cd "/home/jaap/downloads/sage-1.6/devel/doc" && hg commit
and create a patch-file to send to the SAGE-Developers: 
sage: hg_doc.send('prog20070118.hg')
Writing to /home/jaap/sage/prog20070118.hg
cd "/home/jaap/sage/devel/doc" && hg bundle  tmphg http://sagemath.org/sage//hg//doc-main
searching for changes
Successfully created hg patch bundle /home/jaap/sage/prog20070118.hg

1.2.) Common infrastructure

Each directory contains build_dvi and build_pdf scripts. The command make html in $SAGE_ROOT/devel/doc rebuilds the html-documentation. It is also possible to use make pdf from $SAGE_ROOT/devel/doc. You can choose between letter and a4 in the Makefile.

The file $SAGE_ROOT/devel/doc/commontex/macros.tex defines some macros. If you need more tell William Stein what to include (or send a hg patch).

The SAGE-Code examples inside the LaTeX files should be in verbatim environments and can be tested with 

sage -t filename.tex
Two verbatim environments can be tested together by giving a %link comment at the end of the first and the beginning of the second environment. So some text between the code is possible.
\begin{verbatim}
sage: def add7(n):return n+7
....:
\end{verbatim}%link
Some Explanation
%link
\begin{verbatim}
sage: add7(7)
14
\end{verbatim}
Without %link both environments would be tested independently and add7 would be unknown in the second environment.

1.3.) The Reference Manual

In $SAGE_ROOT/devel/doc/ref: 

ref.tex:
  groups.tex
  rings.tex
  rings-standard.tex
  rings-padic.tex
  rings-numerical.tex
  number-fields.tex
  polynomial-rings.tex
And much more files are read in with \input from ref.tex. ref.tex and the files it includes, are created by hand. This determines the selection and the order of items in the Reference. The files which are inputted by ref.tex consists again of many inputs: for example structures.tex contains: 
chapter{Basic Structure\label{ch:structures}}

\input{sage.structure.sage-object}

\input{sage.structure.parent-gens}

\input{sage.structure.formal-sum}

\input{sage.structure.factorization}

\input{sage.structure.element}

\input{sage.structure.mutability}

\input{sage.structure.sequence}

\input{sage.sets.set}

\input{sage.sets.primes}

The files beginning with sage are created by scripts (update and update_script.py in $SAGE_ROOT/devel/doc/ref) from the doc-strings of the corresponding modules.

For example open the file $SAGE_ROOT/devel/sage/sage/structure/element.pyx and compare it with $SAGE_ROOT/devel/doc/ref/sage.structure.element.tex and Chapter "10.5 Elements" of the SAGE Reference Manual

1.3.1.) Modifying the Reference Manual

Just change the LaTeX file, test your examples, then run build_dvi or update.

For doc-strings containing LaTeX commands, Python's raw strings are the best choice, so you can use r"\tau" instead of "\\tau".

Send a patch to William Stein.

2.) Part: Creating Packages

2.1.) Coding Conventions

Directory names can be plural, and file names should be singular. Package author are encouraged to include notes as plain text files in a sub-directory notes. In the context of an import statement, directory and file-name become identifiers (from the grammar):

identifier ::= 
             (letter|"_") (letter | digit | "_")*	
import_stmt ::= 
             "import" module ["as" name]
                ( "," module ["as" name] )*
                | "from" module "import" identifier
                    ["as" name]
                  ( "," identifier ["as" name] )*
                | "from" module "import" "(" identifier
                    ["as" name]
                  ( "," identifier ["as" name] )* [","] ")"
                | "from" module "import" "*"
module ::= 
             (identifier ".")* identifier
which means that only underscore and letters and digits (but not in the first place) are allowed for file and directory names which are likely to become package and module names in the near future.
I.E: I plead for renaming 
/finite-quadratic-modules/cn-group/quadratic-modules.sage
to 
/finite_quadratic_modules/cn_group/quadratic_modules.sage

2.2.) Documentation Strings

2.2.1.) ... for the file

Every code file should start with a documentation string of the following format:

r"""
Short Summary

Paragraph with a more detailed description

AUTHORS
  - First author (year-month-date): initial version
  - other author (year-month-date): short description
  ...

Some text in LaTeX

EXAMPLES:
  ...
"""
#*****************************************************************************
#       Copyright (C) 2006 William Stein <wstein@gmail.com>
#                     2006 YOUR NAME <your email>
#
#  Distributed under the terms of the GNU General Public License (GPL)
#                  http://www.gnu.org/licenses/
#*****************************************************************************
It is important to use the GPL or a lesser license (the Programming Guide also suggests the BSD license) and to put William Stein and yourself in the Copyright notice and yourself in the Authors log (to receive credit).

2.2.2.) ... for a function

Every function should be documented with a documentation string. The following fields are suggested:

Field Description optional
One sentence summary followed by a blank line no
INPUT Should describe the arguments, types are descriptive, i.e. integer not int, if the argument should behave like an integer. if there are no arguments
OUTPUT Description of the return value, also with descriptive type. if nothing is returned
EXAMPLES Examples no
NOTES References, notes, ... yes
AUTHORS List of authors and date given as (year-month-day) no

The next example shows how to use the different fields: 

def point(self, x=1, y=2):
    r"""           # note the r for "raw string"
    This function returns the point $(x^5,y)$.

    INPUT:
        self -- anything special about self
        x -- integer (default: 1) the ...
        y -- integer (default: 2) the ...

    OUTPUT:
        integer -- the ...

    EXAMPLES:
    This example illustrates ...
        sage: A = ModuliSpace()
        sage: A.point(2,3)
        xxx

    We now ...
        sage: B = A.point(5,6)
        sage: xxx

    It is an error to ...
        sage: C = A.point('x',7)
        Traceback (most recent call last):
        ...
        TypeError: unable to convert x (=r) to an integer

    NOTES:
        This function uses the algorithm of [BCDT] to determine whether
        an elliptic curve E over Q is modular.

        ...

        REFERENCES:
             [BCDT] Breuil, Conrad, Diamond, Taylor, "Modularity ...."

    AUTHORS: 
        - William Stein (2005-01-03)
        - First_name Last_name (yyyy-mm-dd)
    """

2.3.) Automatic testing

sage -t collects the examples from the doc-strings in your file, then executes the code and compares the actual output with the output recorded. Code-lines are identified by the sage: prompt, so the examples can and should be copied from an interactive SAGE session.

 def add7(x):
    r"""Add 7 to x.

    EXAMPLES:
      sage: add7(1)
      8
      sage: add7(2)
      9
      sage: add7(3)
      10
      sage: add7(4) 
      11
    """
    return x+7
  
#      sage: add7(1.3) # todo: not tested  
#      71
     
sudo sage -t DocTest.sage
sage -t  DocTest.sage
Example 0 (line 4)
         [3.7 s]

----------------------------------------------------------------------
All tests passed!
Total time for all tests: 3.7 seconds
In the background, the file .doctest_DocTest.py is created, it uses the Python doctest module.

I used sudo (at least for the first time), because some temporary directory in $SAGE_ROOT, needed to be created. SAGE expects that code examples run without error message. Every line with a sage: is considered not only inside the EXAMPLES section.

There are a few modifiers:

There is another possibility: If the file has defines a class Test with a random() method, this method is called on a regular basis. The method should try random inputs and pass it to the other classes and methods in the file.

2.4.) Creating a SPKG file

  1. create a directory for example 
    mkdir quadratic_module-0.1
  2. copy quadratic-modules.sage there, renaming it to singular quadratic_module.sage: 
      cp ~/Sage/finite-quadratic-modules/cn-group/quadratic-modules.sage
     quadratic_module-0.1/quadratic_module.sage
  3. create a file quadratic_module-0.1/spkg-install: 
    #!/bin/bash

# Three possible locations:
#   1) a local python lib, more separated from sage
INSTALLPATH="$SAGE_LOCAL/lib/python/site-packages/finite_modules"
#   2) under sage:
#INSTALLPATH="$SAGE_ROOT/devel/sage/sage/modules"
#   3) if we want to install under a new dir e.g.
#INSTALLPATH="$SAGE_ROOT/devel/sage/sage/modules/finite_modules"
#      we need to change 
#      $SAGE_ROOT/devel/sage/setup.py

if [[ ! -d "$INSTALLPATH" ]]; then
	mkdir "$INSTALLPATH"
        #  mark the dir as a package-dir
	touch "$INSTALLPATH/__init__.py" 
fi

# create the py file:
sage quadratic_module.sage
# and copy it
cp quadratic_module.py "$INSTALLPATH"

# if we install under sage and not under $SAGE_LOCAL/lib/python/
# we need the next two lines: 
#cd $SAGE_ROOT/devel/sage/
#python setup.py install
# and the effect is:
#   copying sage/modules/quadratic_module.py -> 
#      build/lib.linux-i686-2.5/sage/modules
#   copying build/lib.linux-i686-2.5/sage/modules/quadratic_module.py -> 
#     /usr/local/sage/local/lib/python2.5/site-packages/sage/modules
# perhaps we can use 
#   build/lib.linux-i686-2.5/sage/modules
# or
#   /usr/local/sage/local/lib/python2.5/site-packages/sage/modules
# from the beginning?
    and make it executable: 
    chmod +x quadratic_module-0.1/spkg-install
  4. Now create the SAGE-Package file from the directory: 
    sage -pkg quadratic_module-0.1
  5. Install the SAGE-Package: 
    [lf@velocity:~/SAGE/SageTalk] sudo sage -i quadratic_module-0.1.spkg
[sudo] password for lf:
Installing quadratic_module-0.1.spkg
Calling sage-spkg on quadratic_module-0.1.spkg
quadratic_module-0.1
Machine:
Linux velocity 2.6.22-14-generic #1 SMP Sun Oct 14 23:05:12 GMT 2007 i686 GNU/Linux
Deleting directories from past builds of previous/current versions of quadratic_module-0.1
Extracting package /home/lf/SAGE/SageTalk/quadratic_module-0.1.spkg ...
-rw-r--r-- 1 lf lf 13080 2008-01-09 20:52 /home/lf/SAGE/SageTalk/quadratic_module-0.1.spkg
quadratic_module-0.1/
quadratic_module-0.1/quadratic_module.sage
quadratic_module-0.1/spkg-install
Finished extraction
****************************************************
Host system
uname -a:
Linux velocity 2.6.22-14-generic #1 SMP Sun Oct 14 23:05:12 GMT 2007 i686 GNU/Linux
****************************************************
****************************************************
GCC Version
gcc -v
Es werden eingebaute Spezifikationen verwendet.
Ziel: i486-linux-gnu
Konfiguriert mit: ../src/configure -v --enable-languages=c,c++,fortran,objc,obj-c++,treelang
  --prefix=/usr --enable-shared --with-system-zlib --libexecdir=/usr/lib
  --without-included-gettext --enable-threads=posix --enable-nls
  --with-gxx-include-dir=/usr/include/c++/4.1.3 --program-suffix=-4.1 --enable-__cxa_atexit
  --enable-clocale=gnu --enable-libstdcxx-debug --enable-mpfr
  --enable-checking=release i486-linux-gnu
Thread-Modell: posix
gcc-Version 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)
****************************************************
1.68user 0.31system 0:02.10elapsed 94%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+22809minor)pagefaults 0swaps
Successfully installed quadratic_module-0.1
Now cleaning up tmp files.
Making SAGE/Python scripts relocatable...
Making script relocatable
    
  6. sage: import finite_modules.quadratic_module
sage: finite_modules.quadratic_module?
Type:           module
Base Class:     <type 'module'>
String Form:    <module 'finite_modules.quadratic_module' from
'/usr/local/sage/local/lib/python2.5/site-packages/finite_modules/quadratic_module.py'>
Namespace:      Interactive
File:           /usr/local/sage/local/lib/python2.5/site-packages/finite_modules/quadratic_module.py
  7. Send quadratic_module-0.1.spkg to William Stein
  8. To remove the package: (I didn't find an automatic way for it: change SAGE_ROOT) 
    sudo rm -f $SAGE_ROOT/spkg/installed/quadratic_module-0.1
sudo rm -f $SAGE_ROOT/devel/sage/sage/modules/quadratic_module.py
sudo rm -f $SAGE_ROOT/devel/sage/build/lib.linux-i686-2.5/sage/modules/quadratic_module.py
sudo rm -f $SAGE_ROOT/local/lib/python2.5/site-packages/sage/modules/quadratic_module.py

Open questions:

Installation under $SAGE_ROOT/devel/sage/sage/ or one of its sub-directories requires setup.py install. For a new directory even modifications on setup.py are necessary.

3.) Part: Mixed Items

3.1.) The Python in SAGE

SAGE consists of many parts held together with Python code.

I will show you how to deal with this.

3.1.1.) Advice: If you have a tool, you want to apply to a SAGE-file, the first thing to try is: use the Python-file.
Every time you run SAGE on a file, say sage test.sage, SAGE creates a corresponding Python-File test.py. The differences between both files are:

  1. test.py has an import statement: 
    from sage.all_cmdline import *
  2. literal Numbers like 42 are replaced with Integer(42)
  3. SAGE syntax violating Python Syntax is converted to Python:
    • 7^3 becomes 7**3 
    • K.<z>=CyclotomicField(7)
      becomes 
      K=CyclotomicField(Integer(7),names=('z',)); (z,) = K._first_ngens(Integer(1))

3.1.2.) Advice: instead of the python command use the command 

sage -python
which runs SAGE's version of Python, which knows how to find SAGE's modules.

3.2.) Emacs as a developer-tool

Now we apply the previous section on "debugging within Emacs". I will show how to use the GUD (Emacs's Grand unified debugger). And in the next part we see a short overview of different Code-Browser-Plugins for Emacs.

3.2.1.) Emacs: Debugger for SAGE/Python

(Continuation of the example above.)
To Debug a Python script you would use PDB, the Python debugger. We take an example SAGE-Script an have a look how we can use PDB with SAGE.

## First possibility: uncomment,
#import pdb
#pdb.set_trace()
# run sage to create the py file and open it in emacs
## then in emacs:
## C-c ! and then C-c C-c
## then n p etc in the buffer

K.=CyclotomicField(7)
for d in range(K.degree()):
    print d
    print K.polynomial()

print z
## second possibility, nicer:
# to debug, run once sage test2.sage to create the test2.py file,
# then in emacs: M-x pdb
# then at the next prompt enter  sage -python -m pdb test2.py instead of pdb
# use n,s, u and p for next, step into, up and print
# and remeber  z^6 in sage is z**6 in python, so use p z**6 and z**7
# use w and c for refresh and continue in the debug buffer, if pdb seems to have lost track
Store the file locally as e.g. test2.sage, run sage on it, to create test2.py. Open test2.py in emacs and then issue 
M-x pdb
sage -python -m pdb test2.py

With compiled languages like C, you can give the -g flag during compilation. It stores debugging symbols in the binary. Then you can use gdb (from emacs: M-x gdb, be sure to select Gud, GDB-UI, Display others Windows). Judging from my limited experience during the last week, gdb-mode has more features, but it is not suited for Python and SAGE (although you can run sage under the control of gdb: sage -gdb).

Why debugging?

3.2.2.) Emacs: Code-Browser-Plugins

There are several Emacs-Plugins which show the structure of a program-file in a tree structure. Depending on the programming language one or the other is better.

ECB offers file and directory browser, a class browser and a special analyse windows which displays some context information. You can change between different layouts from the menu: ECB, Change Layout, when prompted in the minibuffer, hit tab for the available layouts. I like either left15 or leftright-analyse.

With Html or LaTeX files, ecb provides an outline of the chapters and sections.

For more information about Emacs and its configuration, please visit the Emacs-section on my private homepage or have a look at the Emacs-Wiki.

3.3.) A second look at the lack of private attributes in Python

Consider the next example of some C++ code, which uses a private attribute secret and notice how easy someone can access it (at least with the knowledge of the header file or the actual source file). 

#include "iostream"

class UUPs {	
	private:
		int secret;
	public:
		int pub;

		UUPs(int s, int p) {secret = s; pub= p;};
	
		int getSecret();
		int getPublic();
};

int UUPs::getSecret() {  return secret;  }
int UUPs::getPublic() {  return pub;     }

int main(){
	UUPs One(1,1);
/* The next line gives the expected error:
	One.secret = -1;
   main.cpp: In function »int main()«:
   main.cpp:5: Fehler: »int UUPs::secret« ist privat  */

	std::cout << "Object One has Pub:     " << One.getPublic() << "\n"; 
	std::cout << "Object One has Secret:  " << One.getSecret() << "\n";
	
/* Now we modify secret and no private can protect it:
   Go back one unit from the memory location of pub, 
   that is the address of secret:  (&(One.pub ) - 1 ) 
   a Pointer to One.secret and nothing can  protect us from
   shooting in our own foot:    */
	*(&(One.pub ) - 1 ) = -1;

	std::cout << "Object One has Secret: " << One.getSecret() << "\n";

/* also:  get the address of One, at location offset 0 there is our secret,
   we only need some typecast to int: */
	*((int*)(&One)) = 17;

	std::cout << "Object One has Secret: " << One.getSecret() << "\n";
} 

/*
g++ uups.cpp -o uups
 
./uups
Object One has Pub:     1
Object One has Secret:  1
Object One has Secret: -1
Object One has Secret: 17
*/

nach oben

Letzte Änderung: 26.02.2009
© Lars Fischer

Valid XHTML 1.1!  Valid CSS!