This question is a corollary to: Editing programs “while they are running”? Why?
I'm only recently being exposed to the world of Clojure and am fascinated by a few examples I've seen of "live coding". The question linked above discusses the "why."
My question is: How is this live coding technique possible? Is it a characteristic of the clojure language which makes it possible? Or is it just a pattern that they applied which could be applied to any language? I've got a background in python and java. Would it be possible to "live code" in either of these languages like it is possible in clojure?
Source: Tips4all, CCNA FINAL EXAM
Some language implementations have that for a long time, especially many Lisp variants and Smalltalk.
ReplyDeleteLisp has identifiers as a data structure, called symbols. These symbols can be reassigned and they are looked up at runtime. This principle is called late binding. Symbols name functions and variables.
Additionally Lisp implementations either have at runtime an interpreter or even a compiler. The interface are the functions EVALand COMPILE. Plus there is a function LOAD, which allows loading of source code and compiled code.
Next a language like Common Lisp has an object system which allows changes to the class hierarchy, classes themselves, can add/update/remove methods and propagates these changes to already existing objects. So the object-oriented software and code can be updated itself. With the Meta-object Protocol one can even re-program the object system at runtime.
It's also important that Lisp implementations then can garbage collect removed code. That way the running Lisp will not grow in runtime size just because code is replaced.
Lisp often also has an error system which can recover from errors and allows replacing defective code from within the debugger.
JRebel is one solution for Java. Here's a brief passage from their FAQ:
ReplyDeleteJRebel integrates with the JVM and application servers mainly on the class loader level. It does not create any new class loaders, instead, it extends the existing ones with the ability to manage reloaded classes.
There are a lot of good answers here, and I'm not sure I can improve on any of them, but I wanted to add some comments around Clojure and Java.
ReplyDeleteFirst off, Clojure is written in Java, so you can definitely build a live-coding environment in Java. Just think of Clojure as a specific flavor of live-coding environment.
Basically, live coding in Clojure works via the read function in main.clj and the eval function in core.clj (src/clj/clojure/main.clj and src/clj/clojure/core.clj in the github repository). You read in the forms and pass them to eval, which calls the clojure.lang.Compiler (src/jvm/clojure/lang/Compiler.java in the repo).
Compiler.java converts Clojure forms into JVM bytecode using the ASM library (ASM website here, documentation here). I'm not sure what version of the ASM library is used by Clojure. This bytecode (an array of bytes => byte[] bytecode is the member of the Compiler class that will ultimately hold the bytes generated by the clojure.asm.ClassWriter class via ClassWriter#toByteArray) must then be converted to a class and linked into the running process.
Once you have a representation of a class as a byte array, it's a matter of getting ahold of a java.lang.ClassLoader, calling defineClass to turn those bytes into a Class, and then passing the resulting Class to the resolve method of the ClassLoader to link it to the Java runtime. This is basically what happens when you define a new function, and you can see the internals of the compiler in Compiler$FnExpr which is the inner class that generates the bytecode for function expressions.
There's more going on than that with respect to Clojure, such as the way in which it handles namespace and symbol interning. I'm not completely sure how it gets around the fact that the standard ClassLoader will not replace a linked Class with a new version of that Class, but I suspect it has to do with how classes are named and how symbols are interned. Clojure also defines its own ClassLoader, a certain clojure.lang.DynamicClassLoader, which inherits from java.net.URLClassLoader, so that might have something to do with it; I'm not sure.
In the end, all the pieces are there to do live-coding in Java between ClassLoaders and bytecode generators. You just have to provide a way to input forms into a running instance, eval the forms, and link them up.
Hope this sheds a little more light on the subject.
The concepts originated in the Lisp world, but just about any language can do it (certainly, if you have a repl, you can do this sort of stuff). It's simply better known in the Lisp world. I know there are slime-esque packages for haskell and ruby, and I would be very surprised if such a thing didn't exist for Python as well.
ReplyDeleteIt is a pattern that can be applied to any language, provided that the language was written with an environment that permits reassigning names associated with blocks of code.
ReplyDeleteIn the computer, code and data exists in memory. In programming languages, we use names to refer to those "chunks" of memory.
int a = 0;
would "name" some number of bytes of memory "a". It would also "assign" that memory the byte value corresponding to 0. Depending on the type system,
int add(int first, int second) {
return first + second;
}
would "name" some number of bytes of memory "add". It would also "assign" that memory to contain the machine instructions to look into the call stack for two "int" numbers, add them together, and put the result in the appropriate place on the call stack.
In a type system that separates (and maintains) names to blocks of code, the end result is that you can easily pass blocks of code around by reference much in the same way you can variable memory around by reference. The key is to make sure the type system "matches" only compatible types, otherwise passing around the blocks of code might induce errors (like returning a long when originally defined to return an int).
In Java, all types resolve to a "signature" which is a string representation of the method name and "type". Looking at the add example provided, the signature is
// This has a signature of "add(I,I)I"
int add(int first, int second) {
return first + second;
}
If Java supported (as Clojure does) method name assignment, it would have to expand on its declared type system rules, and allow method name assignment. A fake example of method assignment would logically look like
subtract = add;
but this would require the need to declare subtract, with a strongly typed (to match Java) "type".
public subtract(I,I)I;
And without some care, such declarations can easily tread upon already-defined parts of the language.
But to get back to your answer, in languages that support such, the names basically are pointers to blocks of code, and can be reassigned provided you don't break the expectations of input and return parameters.
It's possible in many languages, but only if you have the following features:
ReplyDeleteSome form of REPL or similar so you can interact with the running environment
Some form of namespace that can be modified at runtime
Dynamic binding against the namespace, so that if you change items in the namespace then referring code automatically picks up the change
Lisp/Clojure has all of these built in by default, which is one of the reasons why it is particularly prominent in the Lisp world.
Example demonstrating these features (all at the Clojure REPL):
; define something in the current namespace
(def y 1)
; define a function which refers to y in the current namespace
(def foo [x] (+ x y))
(foo 10)
=> 11
; redefine y
(def y 5)
; prove that the change was picked up dynamically
(foo 10)
=> 15
All that is required is:
ReplyDeletethe language must have the ability to load new code (eval)
an abstraction to redirect function/method calls (vars or mutable-namespaces)
Yes, it's also possible in other languages. I've done it in Python for an online server.
ReplyDeleteThe key feature needed is the ability to define or redefine new functions and methods at runtime and this is easy with Python where you have "eval", "exec" and where classes and modules are first-class objects that can be patched at runtime.
I implemented it practically by allowing a separate socket connection (for security reasons only from the local machine) accepting strings and exec-ing them in the context of the running server. Using this approach I was able to update the server while it was running without having the connected users to suffer a disconnection. The server was composed of two processes and was an online playfield with a client written in Haxe/Flash, using a permanent socket connection for realtime interaction between players.
In my case I used this possibility only for some quick fixes (the biggest was removing ghost connections that were remaining up in case of a network disconnect in a specific protocol state and I also fixed the bug that allowed the creation of these ghost connections).
I also used this management backdoor to get some resource use information while the server was running. As a funny note the very first bug I fixed on a running server was a bug in the backdoor machinery itself (but it wasn't online with real users in that case, just artificial users for load testing, so it was more like a check if it could be done than a real use as there would have been no problems at all shutting down the server for that).
IMO the bad part of doing this kind of live hacking is that once you fix the running instance and you can be sure that the fix works, you still have to do it in the regular source code and if the fix isn't trivial you cannot be 100% sure that the fix will work once you boot an updated version of the server.
Even if your environment allows to save the patched image without bringing it down, still you cannot be sure that the fixed image will start or will work correctly. The "fix" on the running program could for example break the startup process making it impossible to get to a correct running state.
Java's debugging api's allow you to do this - that's how edit/continue aka hotswapping is implemented.
ReplyDeletehttp://www.jug.mk/blogs/ipenov/entry/hotswapping_using_jvm