Python-based Scientific and Engineering Worktools

There are so many Python versions, add-ons, packages, development tools and applications, that you can put together a tailored set of tools, just right for any field of interest. The problem is that there are too many choices and combinations, and probably quite a bit of searching to end up with the perfect setup to do what you want.

It is nice for beginners to have a defined (and to some extent, prepackaged) starting point to get going in. After that, they can explore additional elements freely. This page describes a very good starting setup. Here we dwell on the Scientific and Engineering fields, particularly as suited to Process Control Engineering (not that there is much specific to this domain on this page).

Overview

We need three main elements to get going:

• Python base-language (one of the most powerful, object-oriented, interactive languages available)

• Science and Engineering Toolboxes (the numerical algorithms and prepackaged tools we need)

• Integrated Development Tool (IDE) to help integrate script editing, debugging, and execution.

Python

One of the most popular languages available today is Python, having strong object-oriented and functional-programming features. Even this web-site uses Moin-Moin, a python-based Wiki server. In conjunction with a set of "toolboxes", Python can take on similar functionality to Matlab. Because Python is a real, broad-base, popular programming language, you can extend your interactive algorithm tinkering to use other Python extensions to expand its capability or build a professional fully-fledged application with extras such as a GUI, backend database, or web-access. In this respect, Python is a much better base than Matlab, and Python is unrivaled in its array of extension modules which do most of the work to minimise the programming required. However, if you just want an open-source or free replacement look-alike for Matlab, then look at Octave and Scilab.

Science and Engineering Toolboxes

Python has ~20,000 packages/toolboxes registered with PyPI (The Python Package Index) including game libraries, encryption, web servers, ERP and Science and Engineering. Three packages are the primary starting point to Science and Engineering:

• numpy - a universal array toolbox (much like the base Matlab capability)

• SciPy - scientific numerical algorithms package (much like a lot of the "Toolboxes" of Matlab rolled into one)

• matplotlib - a publication quality graphing package (2D and some 3D capability like Matlab)

As an example of some of the capability of these pacakge, here's the packages or Toolboxes built into SciPy (which uses Numpy under the hood):

• ndimage --- n-dimensional image package
• stats --- Statistical Functions
• signal --- Signal Processing Tools
• lib --- Python wrappers to external libraries
• linalg --- Linear algebra routines
• linsolve --- Linear Solvers
• misc --- Various utilities that don't have another home
• sparse --- Sparse matrix
• interpolate --- Interpolation Tools
• optimize --- Optimization Tools
• cluster --- Vector Quantization / Kmeans
• linsolve.umfpack --- Interface to the UMFPACK library.
• fftpack --- Discrete Fourier Transform algorithms
• io --- Data input and output
• maxentropy --- Routines for fitting maximum entropy models
• integrate --- Integration routines
• lib.lapack --- Wrappers to LAPACK library
• special --- Special Functions
• lib.blas --- Wrappers to BLAS library

Integrated Development Environment

Probably the most popular and capable Python IDEs also happen to be open-source and free. PyDev builds on the popular Eclipse software platform (which itself has addons for everything from database browsing to java/C/fortran IDEs and GUI construction systems and ERPs). PyDev includes all the elements for a good IDE (syntax colouring editors, debugger, project/file/class browser), plus some really useful extras like background code checking, call signature and documentation hints on objects and functions of any package/module you may be using in that source file (even though you may not have run the file yet). It also has an interactive console to help test out code that you want to use.

However, some people are aiming at small-scale software development or just want to interactively process data and write reusable scripts which make this easier. For these users, Spyder (google spyderlib), a Matlab-like GUI, may be the best environment (although sparsely documented and still under rapid development). It is rather simpler to use and more oriented towards interactive use than PyDev (less initial "project" setup), but the IDE assistance is not as comprehensive as PyDev. However, being tailored for scientific and engineering use, Spyder, like the Matlab IDE, has lots of little features that help a great deal.

Getting It

Fortunately, someone has put all of this together for us in a single download that suits most scientists and engineers, so don't go searching around the web for all the different components that you might think you need.

Python(x,y), a new packaging of SciPy, numpy, and Spyder (with the important packages mentioned above, and many more useful ones that you'll get to know later) can be found on this link. This automatic installation of all the required packages makes installation very easy. On Windows, it also packages a nice utility to get access to all the tools through a task-bar applet.

Tips and Tricks for Python(x,y)

• I've found that Spyder satisfies most IDE requirements, including software development, interactive data manipulation and scripting. So no need to separately install PyDev to get the top-end functionality for software development.

• Learn most of the short-cut keys for Spyder.

• Learn to take advantage of having several concurrent Python consoles for different coding experiments while editing your modules and scripts.

Other extensions or packages that might be useful

Note, some of these are already included in the default installation of Python(x,y)!

• PyDSTool - a powerful, Python-based, interactive, continuous-time modelling/simulation framework compatible with numpy and SciPy (no GUI, but still a good replacement for Simulink?).

• PyTables - Python and numpy extension for rapidly and conveniently handling very large amounts (up to many Giga-bytes) of tabular and hierarchical data (e.g. like historical SCADA or data logging and time-series data). It is not suitable for Relational DataBase requirements. Conversely, RDBs are not suitable for handling large amounts of time-series data.

• pyChem - a Chemometrics package which (if you can figure out the required input file formats) is a prepackaged python generalised data analysis system with a nice GUI. Also, the open-source python source modules can be imported into your program (especially pychem.mva.chemometrics etc.) to give you access to the Principal Component Analysis (PCA), Cluster Analysis, Discriminant Function Analysis, Partial Least Squares Regression (PLSr), among others.

• Modular toolkit for Data Processing - Although designed as a mathematical processing flowsheet tool, this package includes a range of algorithms compatible with numpy and scipy, including: Principal Component Analysis (PCA), Independent Component Analysis (ICA), Slow Feature Analysis (SFA), Independent Slow Feature Analysis (ISFA), Growing Neural Gas (GNG), Factor Analysis, Fisher Discriminant Analysis (FDA), and Gaussian Classifiers.

• Ptolemy II - graphical hybrid simulation tool (somewhat broader and shallower in scope than Matlab's Simulink, but with process control block contributions, this could become a good alternative). Unfortunately, it is java-based and doesn't integrate well with python (despite being able to use python scripting (jython) for block functions.

• Simpy - Discrete Event Simulation using Python.

• PythonCard would be useful for creating quick GUIs for regular data analysis or for simple applications.

• Orange - Orange is an interesting Python-based Data-mining program under development. Its focus is on both Python-scripted and graphically-constructed data-flows.

Documentation Starting Point

Look on the ? menu of Spyder for the main Python package documentation. In addition, here are some direct links to get started with:

• http://www.scipy.org/ - SciPy main web-site and wiki

• http://docs.scipy.org/doc/ - the main reference documentation for SciPy and NumPy (huge!)

• http://www.scipy.org/Documentation - some of the most interesting documentation/reference elements, including some of those below

• http://www.scipy.org/Numpy_Example_List_With_Doc - examples reference for numpy (part of SciPy)

• http://www.scipy.org/scipy_Example_List - examples reference for SciPy itself (just started)

• http://www.scipy.org/Getting_Started - a starting point to get a feel for the tools.

• http://www.scipy.org/NumPy_for_Matlab_Users - for those who are already familiar with Matlab.

• http://mathesaurus.sourceforge.net/matlab-numpy.html - another translation table between Matlab and Python (with Scipy/Numpy)

Python Quick References

• http://rgruet.free.fr/#QuickRef - python quick reference in a variety of formats

• http://www.digilife.be/quickreferences/QRC/Python%202.4%20Quick%20Reference%20Card.pdf - compact quick reference, but only Python 2.4

Process Control related/example files

• Look at the Process Control page for other projects based on python and scipy/numpy.

• ProcessControl/DynSim is a dynamic simulation environment for python, created by using libraries of Process Control and Process "block" classes ready for scripting together.

• Hard-to-find Windows builds of many scientific/engineering packages for Python - http://www.lfd.uci.edu/~gohlke/pythonlibs/

Migrating from Matlab

• http://www.scipy.org/NumPy_for_Matlab_Users is probably the best guide (both as an introduction as well as a handy reference while using Python)

• http://www.scribd.com/full/26685?access_key=ffe3v3jpfltbv provides a good equivalence table between matlab/octave/scilab and python/scipy and R

• Website with links for migration - http://bci2000.org/downloads/BCPy2000/Migration.html

Other Related Stuff

• Quadratic Programming - http://www.danbbs.dk/~kibria/software.html#qpnumpy has a downloadable quadratic programming python interface to a quadratic programming solver code in fortran (provided). I've attached the complete zip file qp_numpy_01.tar.gz since it currently appears abandoned. It appears to contain an (early?) version of the Sequential Quadratic Programming (SQP) code ultimately marketed by Tits and Zhou (University of Maryland) - see qld.f header.