learning breadth vs depth

This is the list of languages that I learned and used at university, when studying Computer Science

1. C
2. C++
3. Java
4. Python
5. Perl
6. Common Lisp
7. Prolog
8. Shell scripting
9. motorola 68k assembler
10. MIPS assembler

This is the list of languages that I learned and/or used at work, when developing corporate software

1. Java
2. Perl
3. C++
4. C
5. Ruby
6. BPEL ? (this doesn't really count)

The above lists are ordered by frequency of use.
Now it may look like I learned a hell of a lot more languages at uni than I have at work; But really we only had time to learn the surface of those languages and just enough to get the subject projects completed.
At work we may only use a small number of languages, but I've had to study them in greater depth. In addition, I've had to learn a lot of different frameworks and tools (especially in Java) to learn.

It's a matter of :

• breadth vs depth
• core-language vs tools and frameworks.

Is it a framework or a library?

Is it a framework or a library?
Paul Chiusano expresses a very similar sentiment in push-libraries-vs-pull-libraries

correct or configurable?

I would rather work on software that changes easily and does the correct thing, than the kind of overly configurable software described in http://www.thoughtclusters.com/2009/09/software-analysis-paralysis/ also http://www.thoughtclusters.com/2007/08/hard-coding-and-soft-coding/

I've worked on projects where the software was incredibly configurable. One was able to support over 11 different versions of over 30 different interfaces to external systems. You could reconfigure the core behaviour to a ridiculous degree.
The system was so configurable that finding a working configuration became a problem in development, and getting your hands on the actual correct configuration was practically impossible.
The users had the one gold configuration that they modified for each new version; development did the same.
Creating a complete configuration from scratch would have been impossible.
The way it was managed, and updated, the config was actually treated as part of the code base. It's just in a different language (external DSL). So the only benefit was being able to change parts in a live system, without compiling. Using an interpreted language would have the same effect.
So maybe this type of system should really be built from two languages, a compiled language for the performance critical sections, and an interpreted language for the more dynamic behaviours of the application.
The dynamic behaviour would not be something the average user would modify.
A third part of the application is the user defined configuration.

Games based on the Unreal engine (one of the most used 3D games engines), almost exactly follow the above layering scheme:

• The core engine is written in C++.
• The content of the game (and mods) is written in UnrealScript. (Which is compiled in this case rather than my proposed interpreted language).
• The user configuration is stored in key/value text property files.

If the script layer was interpreted, then users would be able to change the game rules themselves; so it's understandable that they've gone with compiled.

Having this kind of layering would really help with the average Java programmers obsession with being able to make changes without compiling. This obsession is part of the reason that there are so many frameworks that require tonnes of xml config. To me it all stems from the fact that Java is compiled, and for most purposes the dynamic features require too much overhead to bother with.

On the JVM, Java could still be used for the compiled stuff, with groovy, JRuby or Jython being used for the middle layer, finally xml and property files for the config layer. The key is that the engine and first and middle layers should be able to call both ways, and should both have access to the config layer.
If reloading this middle layer at runtime is required, I'm sure it's possible in most interpreted languages. I've done something recently in ruby which did exactly that.

Another option would be to use a language such as Scala or a lisp, which can operate at all 3 layers.

Compiled statically typed code, interpreted script possibly in an internal DSL, and simple DSL for storing user config.

With this kind of architecture, it would be possible to have different base configurations available for different individual users, while having different middle layers for different classes of users.

hacker howto

http://catb.org/~esr/faqs/hacker-howto.html

Teach Yourself Programming in Ten Years

Thinking Functionally

I've been thinking about software solutions more and more as pure functions.
This quote describes perfectly the way my thinking has changed:

In fact, you can think of any impure function as having three "steps": 1) An "input" side effect, Unit => B, a pure function (A,B) => C, and an "output" side effect C => Unit. It makes perfect software engineering sense to decouple these components - and that is exactly what is done in purely functional programming.

from actors-are-not-good-concurrency-model.

One side benefit is that the core logic of the functionality is now perfectly testable; as pure functions are the easiest thing in the world to write tests for.

a project just for fun

I've started a project of my own.

I've decided to use a bunch of technologies I'm actually interested in, rather than the usual crap I have to use:

• scala - general programming
• gradle - build
• mongodb - persistence
• swing from scala - UI
• vlc / winamp / wmp - media playback
• lift - web interface, if I get around to it

I've used scala before, but just for experiments and tools.

I've been reading up on gradle, as I've been using ant fairly extensively. I am not a fan of maven, but I do like the ideas behind the dependency management.

I've used swing a fair bit before, and the application will have a fairly simple interface. I thought about trying out SWT, but I wasn't sure how it would integrate with scala.

I wanted it to work with any of the media players on my PC.

There'll probably be some sort of admin interface, if I get around to it done in lift. Lift seems to be the obvious choice in scala. I've been told that it's very different from struts, which I have used and am not a fan of.

The biggest technology leap has been moving from an SQL db to mongodb. I've heard a fair bit about nosql, and it sounded like a much better persistence mechanism for a small project like this one.

The difference between mongodb and sql is bigger than people make out.

It's similar to the difference between a strong/statically typed language and a strong/dynamically typed language.

With both, you need to be sure of what type you are expecting.

With the strong/statically typed language (sql), you have to be very careful with the exact data structures that have been defined, and be sure that the exact nullable/mandatory/empty list rules match between the data type and what the code assumes.

With the strong/dynamic  typed language (mongodb), each time you do a method call (query or update) you need to be aware that a method missing (eg, "mandatory" data missing), could actually happen with any action. The syntax doesn't allow you to just assume that something will work, you need to think about how the error cases will be handled.

With mongodb, there's no guarantees put on the data structure or content at all.
Just this simple difference,  puts a fair bit more of the data consistency checking onto the application programmer.

This trade off is perfect for a exploratory project like mine, which has a single client application. But in any data store that had multiple client applications, this trade off would be horrible to manage. There's no way that three different application could be consistent in their consistency checking, especially if any of them attempted to rectify data problems.

I've previously worked on an LDAP db, that had a provisioning interface used to ensure data consistency. All updates would go through that app, and reads would go directly to the LDAP interface.

I can imagine a similar configuration used for nosql data stores. A shared provisioning interface (library/REST/SOAP) that ensures the specific data consistency required for that application domain, instead of the SQL's one and only structure. That way the queries could make certain assumptions about the data structure.

I think that the flexibility of nosql, offers us a new way to abstract over data; which requires us to think differently about the best way of managing that data.

what is it you are really doing?

I was at a family event, and my very young cousin asked my uncle what my job was.

Instead of talking about computers or games or the internet, or any of those things that people normal fall back on when trying to explain what software engineering/development/programming is; he described the actual purpose. He said described it as :

Automating all the boring jobs, so that everyone can have a better life.

Which is a much bigger job than just making websites, games or anything like that.

It reminded me of the SICP lectures, where they broke down the title "Computer Science".
They talked about Geometry, as derived from the Greek for measuring the earth. Modern Geometry has nothing to do with actual measuring of the earth, it's just how the abstract modern geometry started out.

In the same way, Computer Science has nothing to do with computers; That's just how it's starting out.

What Computer Science (Which isn't really a science either, but I'll leave that for a later discussion) is really about is understanding process; thinking about how we do stuff.

This more abstract definition, feeds directly into my uncle's description of the true purpose of software engineering/development and programming.
It has nothing to do with computers, it's about working out how people do things, and automating (simplifying) them.
The upshot is:

• If you don't know how to do it manually, you can't automate it.
• Just because you're using a computer to do it, doesn't mean it's better than doing it manually.

Everyone knows what happens when the requirements are wrong, but people don't think enough about if the solution is actually an improvement.

There's been many times when people have shown me software solutions, that don't actually simplify (automate) the process at all. It a few cases, the software was actually more complicated than the manual process.
Those ideas always struck me as stupid, but I couldn't think of why it was fundamentally wrong.

You need to ask yourself, is this actually going to help?