PyPedia:Documentation

Introduction

The Philosophy behind PyPedia

The question that PyPedia comes to answer is: Can there be any form of objectively accepted code from a certain community?. The wikipedia paradigm showed that although there is a vast variety of opinions, biases and cultural backgrounds there can be a consensus that guides the creation of encyclopedic and highly informative text about subjects spanning through almost all areas of interest, even for controversial and sensitive ones. Can this concept be applied in code authoring? My view is that the algorithms are available today through a dispersed, non-systematic way, just as the knowledge was before wikipedia. PyPedia is an effort to synchronize programmers, algorithm and math enthusiasts to create objectively accepted implementations of algorithms by using the Mediawiki CMS and the Python programming language.

A similar idea is expressed here: http://wiki.python.org/moin/SandboxedPython

How does it work?

PyPedia is a normal installation of MediaWiki, the Content Management System that Wikipedia is based on. Additionally some Hooks have been written in order to alter the normal behavior of MediaWiki when it creates a new article and when it saves an edit. The new behavior allows only for a specific structure for each article. In each article an author describes the documentation, the parameters, the source code and the unit tests of a python method. This method has the same name as the article. As soon as this article is saved, the code, can be called by a simple function call (no import needed) by another PyPedia article or loaded locally by using the PyPedia python library. As with wikipedia every edit a user does is instantly accessible to all users.

How safe is it?

Articles can belong in one of two virtual namespaces.

The articles ending with "_user_<username>" are created and edited by PyPedia users who define the content and the permissions. These articles can have (almost) any kind of python code. The format of the title of these articles is: <Function or Class name>_user_<Username>. For example: Foo_user_JohnDoe. These articles belong to the _user_ virtual namespace.
The normal articles are articles taken from the _user_ virtual namespace that fulfill certain qualitative criteria. These criteria include: qualitative, "pythonic", commented code that solves a known problem with complete documentation, parameters and unitests. These articles can be edited only by the admins and shouldn't call any _user_ articles.

You shouldn't confuse these virtual namespaces with the mediawiki namespaces. All the articles belong to the same main mediawiki namespace. The difference between _user_ and normal articles is the naming of the title. With this distinction we allow users to experiment and define their own sub-communities in PyPedia while there is a space for objectively qualitative and safe articles in the normal virtual namespace. You can express your support of a _user_ article to be moved in the normal namespace in the Talk pages. There are thoughts to include a voting mechanism in the future.

This practically means that articles in the _user_ virtual namespace shouldn't be considered safe. You shouldn't run these articles in your local computer if you don't trust the code, the user or if you aren't running in a sandbox. It is perfectly safe though to run these methods via the "Execute in browser" button in each article (or through the Main Page). This is because through the "Execute in browser" button, the code is run in the google appengine python 2.7 sandbox and not in your local computer. Only what is printed in the output (or the errors) is printed in your browser.

Moreover, it is safe to run methods that belong in the normal namespace (that is articles not starting with _user_, i.e. Hanoi towers). These articles have been moved there after thorough testing of their validity and safety. Of course a _user_ article is never called by an article in the normal namespace.

Who can contribute?

Anonymous Users can only edit the Talk Pages and the Development Code section of each article. The code in the Development Code section is not parsed, run or downloaded. It is there to allow anyone to make corrections or suggestions to the authors of an article.
Signed in users can only create articles with the name: <Function or class name>_user_<Username>. The creator of an article is the only one allowed to edit this article. Users can change the permissions of each section of their articles and allow other users to edit specific sections by adding other user's usernames in the permissions sections. For example if you want to allow JohnDoe to edit the Documentation section of your article add "JohnDoe" with a comma next to your username in the Documentation Permissions section.
Admins can edit the normal virtual namespace. They can also delete or alter a User article if it is deliberately harmful, or misleading.

Special pages

To request for a method / algorithm edit this page PyPedia:Wanted_algorithms

Writing PyPedia code

Each Article has the following structure:

==Documentation==

This is sample article from user JohnDoe for demonstation reasons.

===Parameters===
<!-- DO NOT EDIT HERE! AUTOMATICALLY GENERATED -->
{{#form:action=http://www.pypedia.com/extensions/pypedia_server/pypdownload.php|method=post|target=_blank|enctype=multipart/form-data}} 
<p>Enter value for argument 1:{{#input:type=text|name=data__arg_1|value=default_input}} <br>
Enter value for argument 2:{{#input:type=text|name=eval__arg_2|value=4}} <br>

{{#input:type=hidden|name=article_title|value=Foo_user_JohnDoe}}
{{#input:type=hidden|name=pyp_username|value={{CURRENTUSER}}}}
{{#input:type=ajax|value=Download code|id=dc}}
{{#input:type=ajax|value=Execute on remote computer|id=eorc}}
{{#input:type=ajax|value=Execute on browser|id=eob}}
{{#formend:}}

<!-- EDIT HERE! -->
<source lang="xml">

<inputs>
<param name="arg_1" type="data" value="default_input" label="Enter value for argument 1:"/>
<param name="arg_2" type="eval" value="4" label="Enter value for argument 2:"/>
</inputs>
</source>

===Return===

===See also===

==Code==

<source lang="py">
def Foo_user_JohnDoe(arg_1 = None, arg_2 = None):
	print "function foo"
	print "Arguments:", arg_1, arg_2
	return 42

</source>

==Unit Tests==

<source lang="py">

def uni1():
	if Foo_user_JohnDoe() != 42:
		return "The edit broke the method"

	return True

</source>

==Development Code==

<source lang="py">
def Foo_user_JohnDoe():
	pass

</source>

==Permissions==

===Documentation Permissions===

JohnDoe

===Code Permissions===

JohnDoe

===Unit Tests Permissions===

JohnDoe

===Permissions Permissions===

JohnDoe

When creating a new article do not alter the pre filled text. Save the article with the pre filled text unchanged. Then you can edit the article section by section.

The Documentation

Use wiki text formatting to write a complete and useful documentation of your method. Include the general usage, the parameters, the return value, running examples, references and anything you feel that it is important for the documentation is your method.

The Parameters

In this section you can define the parameters of your method. The description uses a simple XML schema that is similar to the one used by the Galaxy Tool. The whole Parameters sections have to be included in a <inputs></inputs> tag. In these tags you can define:

Text Boxes:

<param name="<name of the parameter>" type="<data or eval>" value="<the pre-inserted value in the textbox"> label="<The text displayed before the textbox>"/>

name is the name of the parameter. Use the same name as an argument in the function defined in this article in order to access the value of the parameter.
type could be "data" for alphanumeric values, or eval for any python expression. This distinction happens on order to wrap with double quotes any alphanumeric constant.
value the pre-inserted value in the textbox
label the text displayed before the textbox.

Combo Boxes:

<param name="name of the parameter" type="select" label="The text displayed before this combobox" help="Tool tip to be displayed on mouse over (not yet implemented)"> 
<option value="Name_of_first_value">first_option</option> 
<option value="Name_of_second_value">second_option</option> 
</param>

name of the name of the parameter. Use the same name as an argument in the function defined in this article in order to access the value of the parameter.
type it should be "select"
label the text displayed before the combo box
help Tool tip tp be displayed on mouse over (not yet implemented)

File selectors. It is the same as the Text Boxes if you set type="file".

Return

Describe here what are the returned values of your method. You can use mediawiki formatiing

The Code

In this section exists the code of the article. The code should be any Python 2.7 script. For example:

<source lang="py">
def Foo_user_JohnDoe(arg_1 = None, arg_2 = None):
	print "function foo"
	print "Arguments:", arg_1, arg_2
	return 42
</source>

Some notes:

The source tags are needed.
The name of the function (or class) should be the same as the title of the article.
The first name of the function (or class) should be a capital letter. (This is in contrast with the guidelines about python function naming but in MediaWiki every article has a capital first letter).
The names of the arguments in the function should have the same name as the name of the parameters.
Calling other methods: Simply call a method in an article by writing a function call: ArticleName(argument_Lists)
Synonymous methods can be defined by creating redirect articles. For example the article: Hoo_user_JohnDoe could be a redirect to the article Foo_user_JohnDoe. To create a redirect substitute all the text of the Hoo_user_JohnDoe article with the text: #REDIRECT [[Foo_user_JohnDoe]]

The Unitests

The Unitests sections contain functions that act as assertions for the correction of the code. These functions should return True/False values according to the validity of the test. After an edit of the code, all unitests functions are evaluated. If any unitest fail then the edit is not saved. Unit tests are functions that have any name, no arguments and return True or False values.

An example of a unit test to verify that a function returns the minimum value might be:

def unitTest():
   return Foo_user_JohnDoe() == 42

Alternatively, you can return a string that gives more details why the Unit test failed:

def unitTest():
   if Foo_user_JohnDoe() != 42:
      return "This edit broke the function Foo_User_JohnDoe. It didn't return the correct value (42)"

   return True

The Development Code

The "Development Code" section contains code that is not tested for syntax error or unit tests, nor parsed or downloaded. It is intended for edits that will lead to future refinements of the "Code" section of the article. This is the only section of the article that can be edited by anyone, even anonymous users.

The Permissions

Here you can define permissions for specific section edits. Each section can be edited only by the list of users explicitly defined here. The list should be coma separated. Initially only the creator of the page is allowed to edit all sections. Special user permissions are:

"ALL" : All users (even anonymous) are allowed to edit this section.
"SIGNED": All signed in users are allowed to edit this section.

Execution

There are four ways to execute the PyPedia code: Through the PyPedia python library in your local computer and through the "Download code", "Execute on Remote computer" and "Execute on browser" buttons in each article. In cases that you are running in your local computer, you only need to have installed is python. The recommended python version is 2.7.2 . Python 2.6.X could not work for all articles (for example the one that have dictionary comprehensions) and python 2.7.0 will not work at all as it contains a bug in the httplib.py file.

PyPedia python package

To execute the PyPedia code locally you need a special package that connects to www.pypedia.com and downloads and manages the code. To install the package:

git clone git://github.com/kantale/pypedia.git

After installing the pypedia package simply type (assuming a suitable python environment):

>>> import pypedia
>>> from pypedia import Hello_world
>>> Hello_world()
Hello World!
>>>

The first statement (import pypedia), maintains a connection to this wiki while the second (from pypedia import Hello_world) downloads and imports the function or class Hello_world. Of course pypedia needs to be imported once, whereas for any function that you want to import you need a "from pypedia import XXXX" statement. You only have to import the main function that you want to run: If the function foo() calls goo() and you don't want to call goo() explicitly then you only have to import foo(). The docstring of the function or class contains the documentation that exists in the Documentation section of the article. For example try: print Hello_world.__doc__

Additional options:

pypedia.enable_cache = True , to download articles only if there aren't already downloaded. To clear the cache delete everything in the directory pypedia/pypCode/
pypedia.debug = True , for debug info
pypedia.warnings = False , to suppress warnings
pypedia.before_timestamp. Import the most recent revision of the articles right BEFORE the timestamp that you set in this variable. The format should be "YYYYMMDDHHMMSS". For example the following code will import the last revision of the method Hello_world as it was before the 16/4/2012.

>>> import pypedia
>>> pypedia.before_timestamp = "20120416000000"
>>> from pypedia import Hello_world

This revision filter is applied recursively to all functions and classes that are also imported. Use this option to reproduce an analysis that was done by pypedia at a specific time ignoring all the subsequent changes done to the articles.

"Download Code" button

In each article you can fill in the parameters of an algorithm. Then you can click the "Download Code" button and a <Article_name>.py file is downloaded (for example Hello_world.py). This file contains all the necessary python code needed to run the method. To execute the code type:

> python <Article_name>.py

In this script both the code and the parameters are included.

Execute on browser button

When you press the "Execute on browser button" the same code that is downloaded with the "Download code" button is sent to the Google App Engine (GAE) python 2.7 environment. The code is executed and the results are shown back in the browser. Among the packages that you can include and run on GAE is numpy ver. 1.6.1

"Execute on remote computer" button

There is also the ability to let PyPedia login via SSH in a remote computer, execute the code and fetch the results. You simply fill in the parameters in the form of each article and press the execute button. You have to be logged in and you have to have declared the hostname, username and the path where you want the computation to take place. To declare these elements, edit your userpage and add the following section:

==ssh==
host=www.example.com
username=JohnDoe
path=/home/JohnDoe/runPyPedia

This text is not saved in the wiki! The User table of the mediawiki has been altered in order to contain four more columns: The host, the username, the port and the path of a remote computer. You also need to have installed the PyPedia python library and the ssh_pyp_client utility in the remote computer. To do that go to the execution path of the remote computer (in our case /home/JohnDoe/runPyPedia) and run:

git clone git://github.com/kantale/pypedia.git
wget https://raw.github.com/kantale/PyPedia_server/master/utils/ssh_pyp_client.py

Now when you press the "Execute on remote computer" button. A text appears that asks for the password of your remote computer. Fill int the password and press the "GO" button. The code then is executed in the remote code and three values are shown. (1) What the method printed in standard output, (2) What the method printed in standard error and (3) what the method returned. The output is shown in a new tab and it looks like this:

-------Error-------
None
-------Printed-----
function foo
Arguments: default_input 4

-------Returned----
42

Notes:

The username declared in the ==ssh== section doesn't have to be the same with your PyPedia username
You can run any PyPedia method with this way, regardless if you have editing rights or not.
The connection will close if the method does more than 30 seconds.
When running in a remote computer, PyPedia uses the python that is declared in the $PATH variable of the remote computer. Since this run through ssh connection the $PATH should be declared in .bashrc since it is sourced by non-interactive non-login shells.

Forking an article

Forking is the procedure of creating a personal copy of the current version of an article. By pressing the button "Fork this article" on the top of any article, a new article is created that contains almost the same content. The name of the article and the permissions are changed in order to comply with the new owner. You cannot fork an article if you are not signed in or if the article to be created already exists. You can fork an article regardless if it belongs in the User or in the Main category. If you fork a Main article then the created article will belong in the User category.

The REST interface

PyPedia offers a simple REST interface to access the constraint-free, standalone version of articles. The constraint-free version of an article contains the source code of the article plus the source code of the articles that depends on. For example if Foo() calls Goo() and you request the constraint-free version of Foo() then the source code of both Foo and Goo will be delivered. You can also use the standard REST api of MediaWiki for more extensive queries.

Install PyPedia_server

PyPedia_server is the MediaWiki extension that offers the presented features. To install it:

Download and configure a MediaWiki version 1.18.0 or higher
Go to extensions/ directory of the mediawiki and run:

git clone git://github.com/kantale/PyPedia_server.git

PyPedia:Documentation

Contents

Introduction

The Philosophy behind PyPedia

How does it work?

How safe is it?

Who can contribute?

Special pages

Writing PyPedia code

The Documentation

The Parameters

Return

See also

The Code

The Unitests

The Development Code

The Permissions

Execution

PyPedia python package

"Download Code" button

Execute on browser button

"Execute on remote computer" button

Forking an article

The REST interface

Install PyPedia_server

More

Contact

Personal tools

Namespaces

Variants

Views

Actions

Search

Navigation

Toolbox