A pen-based interface that lets users sketch and handwrite within computer code was created by researchers from Cornell University.

Unlike conventional coding, which relies on typing, the interface, dubbed Notate, lets users of digital notebooks open drawing canvases and handwrite diagrams within lines of digitized computer code.

Source: Cornell UniversitySource: Cornell University

According to its developers, Notate is powered by a deep learning model, bridging the handwritten and textual programming contexts. In other words, notation in the handwritten diagram can correspond to textual code. Further, Notate can identify handwritten programming symbols, such as "n," and then links them up to their typewritten equivalents.

"A system like this would be great for data science, specifically with sketching plots and charts that then inter-operate with textual code," explained the resarchers.

"Our work shows that the current infrastructure of programming is actually holding us back. People are ready for this type of feature, but developers of interfaces for typing code need to take note of this and support images and graphical interfaces inside code."

The researchers suggest that this development demonstrates a way forward by introducing artificial intelligence-powered, pen-based coding as at a time drawing tablets are more widely used.

"Tools like Notate are important because they open us up to new ways to think about what programming is, and how different tools and representational practices can change that perspective," said Tapan Parikh, associate professor of information science and paper co-author.

The article, Notational Programming for Notebook Environments: A Case Study with Quantum Circuits, appears in the 35th Annual ACM Symposium on User Interface Software and Technology.

For more information on Notate, watch the accompanying video that appears courtesy of the University of Cornell.

To contact the author of this article, email engineering360editors@globalspec.com