# Developing for SAGE

This talk consists of three parts, it is about certain aspects you may encounter while extending SAGE.
The first part deals with the documentation, its organisation and how to modify it.
The second part deals with the conventions and technical aspects of SAGE package creation.
The last part covers different smaller topics like how to use tools intended for Python with SAGE.

The first and second part are a summary of the SAGE Programming Guide, Chapter 2, 3 ,4.

$SAGE_ROOT is an environment variable pointing to the root directory of the SAGE installation. At home it points to /usr/local/sage, on linserv it points to /usr/local/share/sage-2.8.15. # 1.) Part: SAGE Documentation The documentation is stored in$SAGE_ROOT/devel/doc. There are sub-directories for the Tutorial (tut), the Programming Guide (prog), the Constructions (const), the Installation Guide (inst), and the Reference Manual (ref). The first four are handwritten LaTeX files, but the Reference Manual is created automatically from a manual selection.

SAGE comes with a full version control system called Mercurial and the $SAGE_ROOT/devel/doc directory is version controlled. So is any modification to the documentation: you can access repository functions from the hg_doc object within SAGE. At first we need to configure Mercurial with our user-name. Create the file ~/.hgrc with (hg: the chemical symbol for mercury) [ui] username = UserName <EMail-Adress>  The Programming Guide, Chapter 4.4 has an example session. sage: hg_doc.pull() cd "/home/jaap/sage/devel/doc" && hg status cd "/home/jaap/sage/devel/doc" && hg status cd "/home/jaap/sage/devel/doc" && hg pull -u http://sagemath.org/sage//hg//doc-main pulling from http://sagemath.org/sage//hg//doc-main searching for changes adding changesets adding manifests adding file changes added 3 changesets with 3 changes to 1 files 1 files updated, 0 files merged, 0 files removed, 0 files unresolved If it says use 'hg merge' above, then you should type hg_sage.merge(), where hg_sage is the name of the repository you are using. This might not work with the notebook yet.  hg_doc.pull() updates your local repository. After modifying some file you can commit the changes: sage: hg_doc.commit() cd "/home/jaap/downloads/sage-1.6/devel/doc" && hg diff | less cd "/home/jaap/downloads/sage-1.6/devel/doc" && hg commit  and create a patch-file to send to the SAGE-Developers: sage: hg_doc.send('prog20070118.hg') Writing to /home/jaap/sage/prog20070118.hg cd "/home/jaap/sage/devel/doc" && hg bundle tmphg http://sagemath.org/sage//hg//doc-main searching for changes Successfully created hg patch bundle /home/jaap/sage/prog20070118.hg  ## 1.2.) Common infrastructure Each directory contains build_dvi and build_pdf scripts. The command make html in$SAGE_ROOT/devel/doc rebuilds the html-documentation. It is also possible to use make pdf from $SAGE_ROOT/devel/doc. You can choose between letter and a4 in the Makefile. The file$SAGE_ROOT/devel/doc/commontex/macros.tex defines some macros. If you need more tell William Stein what to include (or send a hg patch).

The SAGE-Code examples inside the LaTeX files should be in verbatim environments and can be tested with

sage -t filename.tex
Two verbatim environments can be tested together by giving a %link comment at the end of the first and the beginning of the second environment. So some text between the code is possible.
\begin{verbatim}
....:
Some Explanation
\begin{verbatim}
14
\end{verbatim}

Without %link both environments would be tested independently and add7 would be unknown in the second environment.

## 1.3.) The Reference Manual

In $SAGE_ROOT/devel/doc/ref: ref.tex: groups.tex rings.tex rings-standard.tex rings-padic.tex rings-numerical.tex number-fields.tex polynomial-rings.tex  And much more files are read in with \input from ref.tex. ref.tex and the files it includes, are created by hand. This determines the selection and the order of items in the Reference. The files which are inputted by ref.tex consists again of many inputs: for example structures.tex contains: chapter{Basic Structure\label{ch:structures}} \input{sage.structure.sage-object} \input{sage.structure.parent-gens} \input{sage.structure.formal-sum} \input{sage.structure.factorization} \input{sage.structure.element} \input{sage.structure.mutability} \input{sage.structure.sequence} \input{sage.sets.set} \input{sage.sets.primes}  The files beginning with sage are created by scripts (update and update_script.py in$SAGE_ROOT/devel/doc/ref) from the doc-strings of the corresponding modules.

For example open the file $SAGE_ROOT/devel/sage/sage/structure/element.pyx and compare it with$SAGE_ROOT/devel/doc/ref/sage.structure.element.tex and Chapter "10.5 Elements" of the SAGE Reference Manual

### 1.3.1.) Modifying the Reference Manual

Just change the LaTeX file, test your examples, then run build_dvi or update.

For doc-strings containing LaTeX commands, Python's raw strings are the best choice, so you can use r"\tau" instead of "\\tau".

Send a patch to William Stein.

# 2.) Part: Creating Packages

## 2.1.) Coding Conventions

• Indent with 4 spaces and no tabs
• Function and Class names are lowercase and words are separated with underscores. No CamelCase
• Whenever you use a try/except block: in the except statement, use a list of exceptions, do not use an empty list. In that case the try block would catch every exception. For example Ctrl-C should be caught by the SAGE-interpreter.

Directory names can be plural, and file names should be singular. Package author are encouraged to include notes as plain text files in a sub-directory notes. In the context of an import statement, directory and file-name become identifiers (from the grammar):

identifier ::=
(letter|"_") (letter | digit | "_")*
import_stmt ::=
"import" module ["as" name]
( "," module ["as" name] )*
| "from" module "import" identifier
["as" name]
( "," identifier ["as" name] )*
| "from" module "import" "(" identifier
["as" name]
( "," identifier ["as" name] )* [","] ")"
| "from" module "import" "*"
module ::=
(identifier ".")* identifier

which means that only underscore and letters and digits (but not in the first place) are allowed for file and directory names which are likely to become package and module names in the near future.
/finite-quadratic-modules/cn-group/quadratic-modules.sage
to
/finite_quadratic_modules/cn_group/quadratic_modules.sage

## 2.2.) Documentation Strings

### 2.2.1.) ... for the file

Every code file should start with a documentation string of the following format:

r"""
Short Summary

Paragraph with a more detailed description

AUTHORS
- First author (year-month-date): initial version
- other author (year-month-date): short description
...

Some text in LaTeX

EXAMPLES:
...
"""
#*****************************************************************************
#       Copyright (C) 2006 William Stein <wstein@gmail.com>
#
#*****************************************************************************

It is important to use the GPL or a lesser license (the Programming Guide also suggests the BSD license) and to put William Stein and yourself in the Copyright notice and yourself in the Authors log (to receive credit).

### 2.2.2.) ... for a function

Every function should be documented with a documentation string. The following fields are suggested:

Field Description optional
One sentence summary followed by a blank line no
INPUT Should describe the arguments, types are descriptive, i.e. integer not int, if the argument should behave like an integer. if there are no arguments
OUTPUT Description of the return value, also with descriptive type. if nothing is returned
EXAMPLES Examples no
NOTES References, notes, ... yes
AUTHORS List of authors and date given as (year-month-day) no

The next example shows how to use the different fields:

def point(self, x=1, y=2):
r"""           # note the r for "raw string"
This function returns the point $(x^5,y)$.

INPUT:
self -- anything special about self
x -- integer (default: 1) the ...
y -- integer (default: 2) the ...

OUTPUT:
integer -- the ...

EXAMPLES:
This example illustrates ...
sage: A = ModuliSpace()
sage: A.point(2,3)
xxx

We now ...
sage: B = A.point(5,6)
sage: xxx

It is an error to ...
sage: C = A.point('x',7)
Traceback (most recent call last):
...
TypeError: unable to convert x (=r) to an integer

NOTES:
This function uses the algorithm of [BCDT] to determine whether
an elliptic curve E over Q is modular.

...

REFERENCES:
[BCDT] Breuil, Conrad, Diamond, Taylor, "Modularity ...."

AUTHORS:
- William Stein (2005-01-03)
- First_name Last_name (yyyy-mm-dd)
"""

• The blank lines between explanations in the EXAMPLES section are required.
• LaTeX is not possible in the INPUT and OUTPUT sections
• The EXAMPLES section is tested automatically and regularly.
You can try an automatic-test with sage -t filename.sage. The test should give any errors.
It is possible to use special comments: long, optional, todo, for code that takes longer than a second, needs optional packages, or is not yet implemented (search_doc("todo") and search_src("todo")).

## 2.3.) Automatic testing

sage -t collects the examples from the doc-strings in your file, then executes the code and compares the actual output with the output recorded. Code-lines are identified by the sage: prompt, so the examples can and should be copied from an interactive SAGE session.

 def add7(x):

EXAMPLES:
8
9
10
11
"""
return x+7

#      sage: add7(1.3) # todo: not tested
#      71

sudo sage -t DocTest.sage
sage -t  DocTest.sage
Example 0 (line 4)
[3.7 s]

----------------------------------------------------------------------
All tests passed!
Total time for all tests: 3.7 seconds

In the background, the file .doctest_DocTest.py is created, it uses the Python doctest module.

I used sudo (at least for the first time), because some temporary directory in $SAGE_ROOT, needed to be created. SAGE expects that code examples run without error message. Every line with a sage: is considered not only inside the EXAMPLES section. There are a few modifiers: • # todo: not implemented the documentation says an example marked as todo is never tested. But it has no effect (with sage 2.8.14) • # long, an example marked with long is only tested if -long is given on the command line. (Again, this options has no effect with sage 2.8.14.) • # optional, an example marked with optional is only tested if -optional is given on the command line. (Again, this options has no effect with sage 2.8.14.) This is intended for examples which need optional packages. There is another possibility: If the file has defines a class Test with a random() method, this method is called on a regular basis. The method should try random inputs and pass it to the other classes and methods in the file. ## 2.4.) Creating a SPKG file 1. create a directory for example mkdir quadratic_module-0.1 2. copy quadratic-modules.sage there, renaming it to singular quadratic_module.sage:  cp ~/Sage/finite-quadratic-modules/cn-group/quadratic-modules.sage quadratic_module-0.1/quadratic_module.sage  3. create a file quadratic_module-0.1/spkg-install: #!/bin/bash # Three possible locations: # 1) a local python lib, more separated from sage INSTALLPATH="$SAGE_LOCAL/lib/python/site-packages/finite_modules"
#   2) under sage:
#INSTALLPATH="$SAGE_ROOT/devel/sage/sage/modules" # 3) if we want to install under a new dir e.g. #INSTALLPATH="$SAGE_ROOT/devel/sage/sage/modules/finite_modules"
#      we need to change
#      $SAGE_ROOT/devel/sage/setup.py if [[ ! -d "$INSTALLPATH" ]]; then
mkdir "$INSTALLPATH" # mark the dir as a package-dir touch "$INSTALLPATH/__init__.py"
fi

# create the py file:
# and copy it
cp quadratic_module.py "$INSTALLPATH" # if we install under sage and not under$SAGE_LOCAL/lib/python/
# we need the next two lines:
#cd $SAGE_ROOT/devel/sage/ #python setup.py install # and the effect is: # copying sage/modules/quadratic_module.py -> # build/lib.linux-i686-2.5/sage/modules # copying build/lib.linux-i686-2.5/sage/modules/quadratic_module.py -> # /usr/local/sage/local/lib/python2.5/site-packages/sage/modules # perhaps we can use # build/lib.linux-i686-2.5/sage/modules # or # /usr/local/sage/local/lib/python2.5/site-packages/sage/modules # from the beginning?  and make it executable: chmod +x quadratic_module-0.1/spkg-install 4. Now create the SAGE-Package file from the directory: sage -pkg quadratic_module-0.1 5. Install the SAGE-Package: [lf@velocity:~/SAGE/SageTalk] sudo sage -i quadratic_module-0.1.spkg [sudo] password for lf: Installing quadratic_module-0.1.spkg Calling sage-spkg on quadratic_module-0.1.spkg quadratic_module-0.1 Machine: Linux velocity 2.6.22-14-generic #1 SMP Sun Oct 14 23:05:12 GMT 2007 i686 GNU/Linux Deleting directories from past builds of previous/current versions of quadratic_module-0.1 Extracting package /home/lf/SAGE/SageTalk/quadratic_module-0.1.spkg ... -rw-r--r-- 1 lf lf 13080 2008-01-09 20:52 /home/lf/SAGE/SageTalk/quadratic_module-0.1.spkg quadratic_module-0.1/ quadratic_module-0.1/quadratic_module.sage quadratic_module-0.1/spkg-install Finished extraction **************************************************** Host system uname -a: Linux velocity 2.6.22-14-generic #1 SMP Sun Oct 14 23:05:12 GMT 2007 i686 GNU/Linux **************************************************** **************************************************** GCC Version gcc -v Es werden eingebaute Spezifikationen verwendet. Ziel: i486-linux-gnu Konfiguriert mit: ../src/configure -v --enable-languages=c,c++,fortran,objc,obj-c++,treelang --prefix=/usr --enable-shared --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --enable-nls --with-gxx-include-dir=/usr/include/c++/4.1.3 --program-suffix=-4.1 --enable-__cxa_atexit --enable-clocale=gnu --enable-libstdcxx-debug --enable-mpfr --enable-checking=release i486-linux-gnu Thread-Modell: posix gcc-Version 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2) **************************************************** 1.68user 0.31system 0:02.10elapsed 94%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+22809minor)pagefaults 0swaps Successfully installed quadratic_module-0.1 Now cleaning up tmp files. Making SAGE/Python scripts relocatable... Making script relocatable  6. sage: import finite_modules.quadratic_module sage: finite_modules.quadratic_module? Type: module Base Class: <type 'module'> String Form: <module 'finite_modules.quadratic_module' from '/usr/local/sage/local/lib/python2.5/site-packages/finite_modules/quadratic_module.py'> Namespace: Interactive File: /usr/local/sage/local/lib/python2.5/site-packages/finite_modules/quadratic_module.py  7. Send quadratic_module-0.1.spkg to William Stein 8. To remove the package: (I didn't find an automatic way for it: change SAGE_ROOT) sudo rm -f$SAGE_ROOT/spkg/installed/quadratic_module-0.1
sudo rm -f $SAGE_ROOT/devel/sage/sage/modules/quadratic_module.py sudo rm -f$SAGE_ROOT/devel/sage/build/lib.linux-i686-2.5/sage/modules/quadratic_module.py
sudo rm -f $SAGE_ROOT/local/lib/python2.5/site-packages/sage/modules/quadratic_module.py  Open questions: • Which location? • all.py ? Without it, no tab-completion for finite_modules and quadratic_module, are they maintained from a script? • Is setup.py install really necessary? (copy direct to$SAGE_ROOT/local/lib/python2.5/site-packages/sage/modules)
• There are too many directories:
1. $SAGE_ROOT/devel/sage/build/lib.linux-i686-2.5: I think only needed during spkg installation. 2.$SAGE_ROOT/devel/sage/sage: I think this is the development repository and it is not used at run-time.
3. $SAGE_ROOT/local/lib/python2.5/site-packages/sage: I think during run-time everything comes from here. (a link to$SAGE_ROOT/devel/sage/build/sage)

Installation under \$SAGE_ROOT/devel/sage/sage/ or one of its sub-directories requires setup.py install. For a new directory even modifications on setup.py are necessary.

# 3.) Part: Mixed Items

## 3.1.) The Python in SAGE

SAGE consists of many parts held together with Python code.

• This is good because there are many useful tools for Python (even complete tool chains).
• This is bad because the Python in SAGE is a bit special (SAGE != Python).
I will show you how to deal with this.

3.1.1.) Advice: If you have a tool, you want to apply to a SAGE-file, the first thing to try is: use the Python-file.
Every time you run SAGE on a file, say sage test.sage, SAGE creates a corresponding Python-File test.py. The differences between both files are:

1. test.py has an import statement:
from sage.all_cmdline import *
2. literal Numbers like 42 are replaced with Integer(42)
3. SAGE syntax violating Python Syntax is converted to Python:
• 7^3 becomes 7**3
• K.<z>=CyclotomicField(7)
becomes
K=CyclotomicField(Integer(7),names=('z',)); (z,) = K._first_ngens(Integer(1))

sage -python
which runs SAGE's version of Python, which knows how to find SAGE's modules.

## 3.2.) Emacs as a developer-tool

Now we apply the previous section on "debugging within Emacs". I will show how to use the GUD (Emacs's Grand unified debugger). And in the next part we see a short overview of different Code-Browser-Plugins for Emacs.

### 3.2.1.) Emacs: Debugger for SAGE/Python

(Continuation of the example above.)
To Debug a Python script you would use PDB, the Python debugger. We take an example SAGE-Script an have a look how we can use PDB with SAGE.

## First possibility: uncomment,
#import pdb
#pdb.set_trace()
# run sage to create the py file and open it in emacs
## then in emacs:
## C-c ! and then C-c C-c
## then n p etc in the buffer

K.=CyclotomicField(7)
for d in range(K.degree()):
print d
print K.polynomial()

print z
## second possibility, nicer:
# to debug, run once sage test2.sage to create the test2.py file,
# then in emacs: M-x pdb
# then at the next prompt enter  sage -python -m pdb test2.py instead of pdb
# use n,s, u and p for next, step into, up and print
# and remeber  z^6 in sage is z**6 in python, so use p z**6 and z**7
# use w and c for refresh and continue in the debug buffer, if pdb seems to have lost track


Store the file locally as e.g. test2.sage, run sage on it, to create test2.py. Open test2.py in emacs and then issue
M-x pdb
sage -python -m pdb test2.py


With compiled languages like C, you can give the -g flag during compilation. It stores debugging symbols in the binary. Then you can use gdb (from emacs: M-x gdb, be sure to select Gud, GDB-UI, Display others Windows). Judging from my limited experience during the last week, gdb-mode has more features, but it is not suited for Python and SAGE (although you can run sage under the control of gdb: sage -gdb).

Why debugging?

• To find bugs in our code
• To watch SAGE and find out how and where things happen

### 3.2.2.) Emacs: Code-Browser-Plugins

There are several Emacs-Plugins which show the structure of a program-file in a tree structure. Depending on the programming language one or the other is better.

• speedbar: it works with Python but not with SAGE.
• ecb: Emacs Code Browser: it works with SAGE, but has to be installed separately.

ECB offers file and directory browser, a class browser and a special analyse windows which displays some context information. You can change between different layouts from the menu: ECB, Change Layout, when prompted in the minibuffer, hit tab for the available layouts. I like either left15 or leftright-analyse.

With Html or LaTeX files, ecb provides an outline of the chapters and sections.

For more information about Emacs and its configuration, please visit the Emacs-section on my private homepage or have a look at the Emacs-Wiki.

## 3.3.) A second look at the lack of private attributes in Python

Consider the next example of some C++ code, which uses a private attribute secret and notice how easy someone can access it (at least with the knowledge of the header file or the actual source file).

#include "iostream"

class UUPs {
private:
int secret;
public:
int pub;

UUPs(int s, int p) {secret = s; pub= p;};

int getSecret();
int getPublic();
};

int UUPs::getSecret() {  return secret;  }
int UUPs::getPublic() {  return pub;     }

int main(){
UUPs One(1,1);
/* The next line gives the expected error:
One.secret = -1;
main.cpp: In function »int main()«:
main.cpp:5: Fehler: »int UUPs::secret« ist privat  */

std::cout << "Object One has Pub:     " << One.getPublic() << "\n";
std::cout << "Object One has Secret:  " << One.getSecret() << "\n";

/* Now we modify secret and no private can protect it:
Go back one unit from the memory location of pub,
that is the address of secret:  (&(One.pub ) - 1 )
a Pointer to One.secret and nothing can  protect us from
shooting in our own foot:    */
*(&(One.pub ) - 1 ) = -1;

std::cout << "Object One has Secret: " << One.getSecret() << "\n";

/* also:  get the address of One, at location offset 0 there is our secret,
we only need some typecast to int: */
*((int*)(&One)) = 17;

std::cout << "Object One has Secret: " << One.getSecret() << "\n";
}

/*
g++ uups.cpp -o uups

./uups
Object One has Pub:     1
Object One has Secret:  1
Object One has Secret: -1
Object One has Secret: 17
*/



Letzte Änderung: 26.02.2009