Python Tokenize

Tags

, ,

Recently I thought to myself, “Hey self,… Although I’ve never looked into them, I know Python has tools for parsing Python code. I wonder if they might make generating syntax-highlighted HTML pages of Python scripts pretty easy?”

I found the help page for the tokenize module, and the introduction says it’s “useful for implementing ‘pretty-printers,’ including colorizers for on-screen displays.” That sounds like a strong yes.

Continue reading

The Synchronization Problem

Tags

, , , ,

One of the great battles programmers fight involves synchronization between two different parts of the code. The most common example of this is what the comments say versus what the code really does. Another common example is the structure of data stored somewhere (like a database) and the code that manipulates it.

A lot of the programmer’s effort and technique is devoted to managing, if not preventing, synchronization issues. The rules encouraging encapsulation, or against global objects, have a lot to do with this goal.

Continue reading

Python Generators, part 3

Tags

, ,

In the last two posts I’ve explored Python generator functions. The first post went over the basics and showed how they return an iterable object usable in for-loops and list constructors. The second post explored how generator functions work in more detail.

This time I’ll wrap it up with some examples that are a bit more involved. If Python generators were a mystery before these three posts, I hope you feel more comfortable with them after!

Continue reading

Python Generators, part 2

Tags

, , ,

Last time I began exploring Python generator functions. I mostly covered their basic function as iterables, and why they’re good in for-loops. They’re also good for infinite lists. Much of their value lies in how they defer processing an item until it is actually asked for.

This time I’ll dig into the send method, which allows sending messages to a running generator. I’ll also introduce the yield from statement, which allows us to wrap a generator with another function that also acts like a generator.

Continue reading

Python Generators, part 1

Tags

, ,

Python has a feature called a generator — it’s a way of writing a function that allows that function to trade control back and forth with a calling function. It’s not something one needs often, but it can very useful in certain situations.

Create a Python generator, firstly, in using yield instead of return and, secondly, by designing the function a little differently from one that generates and then returns a list of items.

Continue reading

Abusing #define in C

Tags

, , , ,

When I was a callow young programmer learning the ropes and enjoying the sheer power of the C language, I got a bit carried away with the macro pre-processor’s ability to let you redefine the language.

As much fun as that is, and as much as it can make your source code look cool, it’s a really bad idea. At some point the folks in comp.lang.c read me the riot act about it, and they were right.

Continue reading

The Blessing of Unicode

Tags

, , , ,

Computers process numbers using arithmetic and logic (which amount to the same thing). Processing text, however, requires at least two levels of abstraction. Firstly, a definition of a textual atom — informally a character. Secondly, a definition of a textual unit — typically called a string. A string is an ordered list of characters.

Therein lies a whole field of computer science. From a practical point of view, implementing text has become much easier with Unicode. Different character sets was one of the more awful aspects of dealing with text. (Remember CP-1252?)

Continue reading

A Simple DL Parser

Tags

, , , ,

Last time I introduced a general Definition Language (DL) I created for defining structured information. The end goal was an extension of DL, called Data Definition Language (DDL), intended for defining memory and file formats. It was intended for tools that examine that data, allowing them more knowledgeable output than a raw hex dump.

I mentioned that DL has been on my mind lately, and as it turns out I spent the day yesterday writing a DL parser in Python.

Continue reading

Data Definition: DL and DDL

Tags

, ,

A long, long time ago I came up with a simple something I called Definition Language (DL) and an extension of that I called Data Definition Language (DDL). This was before XML (let alone JSON) became popular, and DL and DDL turned out to be somewhat akin to those.

My intention was a configuration language that would allow a data-dumping tool that knew the structure of the data it was dumping. Debuggers can sometimes do that in context. I wanted a tool that could do that with any file format given some DDL config file. (These days I’d probably just use XML.)

Continue reading

Always Implement toString

Tags

, , , , , , , ,

Although I’m categorizing this one as really good advice, rather than as a rule, I think it should be viewed as basically a rule. I think it should be a rule in any object-oriented language that supports it natively (Java and Python, for example).

The advice (rule of thumb, say) is to always create a useful implementation of toString when you create a class. It makes your development and maintenance life ever so much better.

Continue reading