IPython is at the heart of the Python scientific stack. With its widely acclaimed web-based notebook, IPython is today an ideal gateway to data analysis and numerical computing in Python.

IPython Interactive Computing and Visualization Cookbook contains many ready-to-use focused recipes for high-performance scientific computing and data analysis. The first part covers programming techniques, including code quality and reproducibility; code optimization; high-performance computing through dynamic compilation, parallel computing, and graphics card programming. The second part tackles data science, statistics, machine learning, signal and image processing, dynamical systems, and pure and applied mathematics.

Read an Extract from the book

Creating a custom JavaScript widget in the notebook – a spreadsheet editor for pandas

In this recipe, we will look at how to go beyond the existing widgets provided by IPython 2.0. Specifically, we will create a custom JavaScript-based widget that communicates with the Python kernel.

We will create a basic interactive Excel-like data grid editor in the IPython notebook, compatible with pandas' DataFrame. Starting from a DataFrame object, we will be able to edit it within a GUI in the notebook. The editor is based on the Handsontable JavaScript library (http://handsontable.com). Other JavaScript data grid editors could be used as well.

Getting ready

You will need both IPython 2.0+ and the Handsontable JavaScript library for this recipe. The following are the instructions to load this Javascript library in the IPython notebook:

1. First, go to https://github.com/handsontable/jquery-handsontable/tree/master/dist.
2. Then, download jquery.handsontable.full.css and jquery.handsontable.full.js, and put these two files in ~\.ipython\profile_default\static\custom\.
3. In this folder, add the following line in custom.js: require(['/static/custom/jquery.handsontable.full.js']);
4. In this folder, add the following line in custom.css: @import "/static/custom/jquery.handsontable.full.css"
5. Now, refresh the notebook!

How to do it...

1. Let's import a few functions and classes as follows: In [1]: from IPython.html import widgets from IPython.display import display from IPython.utils.traitlets import Unicode
2. We create a new widget. The value trait will contain the JSON representation of the entire table. This trait will be synchronized between Python and JavaScript, thanks to the IPython 2.0's widget machinery. In [2]: class HandsonTableWidget(widgets.DOMWidget): _view_name = Unicode('HandsonTableView', sync=True) value = Unicode(sync=True)
3. Now, we write the JavaScript code for the widget. The three important functions that are responsible for the synchronization are as follows:
   • render is for the widget initialization
   • update is for Python to JavaScript update
   • handle_table_change is for JavaScript to Python update

   In [3]: %%javascript var table_id = 0; require(["widgets/js/widget"], function(WidgetManager){ // Define the HandsonTableView var HandsonTableView = IPython.DOMWidgetView.extend({ render: function(){ // Initialization: creation of the HTML elements // for our widget. // Add a <div> in the widget area. this.$table = $('<div />') .attr('id', 'table_' + (table_id++)) .appendTo(this.$el); // Create the Handsontable table. this.$table.handsontable({ }); }, update: function() { // Python --> Javascript update. // Get the model's JSON string, and parse it. var data = $.parseJSON(this.model.get('value')); // Give it to the Handsontable widget. this.$table.handsontable({data: data}); return HandsonTableView.__super__. update.apply(this); }, // Tell Backbone to listen to the change event // of input controls. events: {"change": "handle_table_change"}, handle_table_change: function(event) { // Javascript --> Python update. // Get the table instance. var ht = this.$table.handsontable('getInstance'); // Get the data, and serialize it in JSON. var json = JSON.stringify(ht.getData()); // Update the model with the JSON string. this.model.set('value', json); this.touch(); }, }); // Register the HandsonTableView with the widget manager. WidgetManager.register_widget_view( 'HandsonTableView', HandsonTableView); });

4. Now, we have a synchronized table widget that we can already use. However, we would like to integrate it with pandas. To do this, we create a light wrapper around a DataFrame instance. We create two callback functions for synchronizing the pandas object with the IPython widget. Changes in the GUI will automatically trigger a change in DataFrame, but the converse is not true. We'll need to re-display the widget if we change the DataFrame instance in Python: In [4]: from io import StringIO import numpy as np import pandas as pd In [5]: class HandsonDataFrame(object): def __init__(self, df): self._df = df self._widget = HandsonTableWidget() self._widget.on_trait_change( self._on_data_changed, 'value') self._widget.on_displayed(self._on_displayed) def _on_displayed(self, e): # DataFrame ==> Widget (upon initialization) json = self._df.to_json(orient='values') self._widget.value = json def _on_data_changed(self, e, val): # Widget ==> DataFrame (called every time the # user changes a value in the widget) buf = StringIO(val) self._df = pd.read_json(buf, orient='values') def to_dataframe(self): return self._df def show(self): display(self._widget)
5. Now, let's test all that! We first create a random DataFrame instance: In [6]: data = np.random.randint(size=(3, 5), low=100, high=900) df = pd.DataFrame(data) df Out[6]: 352 201 859 322 352 326 519 848 802 642 171 480 213 619 192
6. We wrap it in HandsonDataFrame and show it as follows: In [7]: ht = HandsonDataFrame(df) ht.show()
7. We can now change the values interactively, and they will be changed in Python accordingly: In [8]: ht.to_dataframe() Out[8]: 352 201 859 322 352 326 519 848 1024 642 171 480 213 619 192

How it works...

Let's explain briefly the architecture underlying the interactive Python-JavaScript communication in IPython 2.0+.

The implementation follows the Model-View-Controller (MVC) design pattern, which is popular in GUI applications. There is a model in the backend (Python kernel) that holds some data. In the frontend (browser), there are one or several views of that model. Those views are dynamically synchronized with the model. When an attribute of the model changes on Python's side, it also changes on JavaScript's side, and vice versa. We can implement Python and JavaScript functions to respond to model changes. These changes are generally triggered by a user action.

In Python, dynamic attributes are implemented as traits. These special class attributes automatically trigger callback functions when they are updated. In JavaScript, the Backbone.js MVC library is used. The communication between Python and the browser is done via Comms, a special communication protocol in IPython.

To create a new widget, we need to create a class deriving from DOMWidget. Then, we define trait attributes that can be synchronized between Python and JavaScript if sync=True is passed to the trait constructors. We can register callback functions that react to trait changes (from either Python or JavaScript), using widget.on_trait_change(callback, trait_name). The callback() function can have one of the following signatures:

callback() callback(trait_name) callback(trait_name, new_value) callback(trait_name, old_value, new_value)

In JavaScript, the render() function creates the HTML elements in the cell's widget area upon initialization. The update() method allows us to react to changes in the model in the backend side (Python). In addition, we can use Backbone.js to react to changes in the frontend (browser). By extending the widget with the {"change": "callback"} events, we tell Backbone.js to call the callback() JavaScript function as soon as the HTML input controls change. This is how we react to user-triggered actions here.

There's more...

The following are the ways this proof-of-concept could be improved:

• Synchronizing only changes instead of synchronizing the whole array every time
  (the method used here would be slow on large tables)
• Avoiding recreating a new DataFrame instance upon every change, but updating the same DataFrame instance in-place
• Supporting named columns
• Hiding the wrapper, that is, make it so that the default rich representation of DataFrame in the notebook is HandsonDataFrame
• Implementing everything in an easy-to-use extension

Here are a few references about the widget architecture in the IPython notebook 2.0+:

• Official example about custom widgets, available at http://nbviewer.ipython.org/github/ipython/ipython/tree/master/examples/Interactive%20Widgets
• MVC pattern in Wikipedia, at https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller
• Backbone.js, available at http://backbonejs.org/
• Course on Backbone.js, available at www.codeschool.com/courses/anatomy-of-backbonejs
• IPEP 21: Widget Messages (comms), available at https://github.com/ipython/ipython/wiki/IPEP-21%3A-Widget-Messages
• IPEP 23: IPython widgets, available at https://github.com/ipython/ipython/wiki/IPEP-23%3A-Backbone.js-Widgets