Last time we looked at dealing with files in Python, looked into filename handling, and left off after creating a couple of base classes to support general file operations.

This time we’ll extend those classes into some useful file utility classes for handling data from different types of files (for instance, binary, plain text, line-oriented text, and any type of structured file).

We’ll extend the filecontent base class from last time to create a class for opening a generic file. Since we don’t know the file type, we’ll open the file in binary and treat it as an array of bytes:

The dunder init method (lines #6-#9) delegates to the parent class and then just adds a data property (line #9). Lines #11-#22 implement a load method while lines #24-#35 implement a corresponding save method. Both assume the data attribute for the file contents. Line #37 implements dunder contains so clients can use the in operator. Lastly, lines #39-59 exercise the class with a binary data file.

Note that both the load and save methods use the ‘b’ (binary) mode to open the file, so loaded data attribute will be bytes even if the file isn’t binary, and data to be saved must also consist of bytes. The upside is that datafile objects can open any type of file.

When run, this prints:

read: C:\demo\hcc\python\colors.png

fo.filename='colors.png'
len(fo)=1992
fo[0:10]=b'\x89PNG\r\n\x1a\n\x00\x00'
fo[9]=0
0x0a in fo=True
0xff in fo=True
b=89
b=50
b=4e
b=47
b=0d
b=0a

This is fine for binary files, such as images, but working with text files will typically involve the extra steps of converting the bytes to string when reading data and vice versa when writing it.

We can make life easy by defining an almost identical class, but one that reads text files:

This is very much like the datafile class, but here the data attribute is a (single) string. One notable change is the addition of an encoding parameter to the load and save methods. This defaults to ‘utf-8’ but can be changed for other file encodings. The method passes the value to the open function.

Associated with the file encoding awareness is the code on lines #18-#21 and lines #34-#36. The former checks for the Unicode BOM (byte order mark) and silently removes it from the string if found. The latter adds the BOM to the output if the use_bom parameter is True.

When run this prints:

read: C:\demo\hcc\python\the-dream.txt

fo.filename='the-dream.txt'
len(fo)=3543
fo[0:60]="I had what HAS to be the longest, strangest dream that I've "
fo[100]='e'
"house" in fo=True
"foobar" in fo=False
nbr_of_spaces=663

These two classes can handle just about any file type, but text files (and binary files, for that matter) are often structured in a regular way we can use to load the file data into a more structured data object.

One simple example is loading line-oriented text files as a list of lines rather than a single string (which avoids the need to use str.splitlines on the textfile.data string). A more involved example is loading a JSON file as the dictionary object it defines.

Let’s start simple with a class that loads line-oriented text files as a list of strings:

Now the data attribute is a list, initially empty (line #9). We’re dealing with text files, so we still want encoding parameters for the load and save methods. We also keep the use_bom parameter. (Text files always use these, so it won’t be remarked again.). The mechanics of the load and save functions are slightly different to deal with a list of strings.

More noticeably, we’ve added two methods. The dunder call method is a convenience method that returns a single string with all the file lines joined by newline characters. The add method provides a convenient way to create a line-oriented file from scratch (see below for some examples of creating new file objects and using the save method).

When run, this prints:

read: C:\demo\hcc\python\the-dream.txt

fo.filename='the-dream.txt'
len(fo)=61
fo[3:5]=['probably as close to a nightmare as I ever come!!', '']

I had what HAS to be the longest, strangest dream that I've had in a long,
long time.  I usually have fun dreams, or weird dreams, sometimes
disturbing dreams - never nightmares.  This dream was very disturbing;
probably as close to a nightmare as I ever come!!

I dreamed I was visiting a film company on location in Cincinnati.  I don't

In all of these file classes, we can create a new instance, set the data attribute, and call the save method to create a new file (or overwrite an existing one). The only requirement is setting the data attribute with the right kind of data. A byte array for datafile, a string for textfile, and a list for linefile:

Note that the linefile class doesn’t expose all possible list methods, just the key ones, length, indexing, and iterating. For example, the delete and pop methods aren’t exposed, but clients can access them through the data attribute.

Because Python has JSON-handling library routines, a class for JSON files is simpler than the previous example:

Python does all the work! Note that the data attribute is a dict object now (line #10). The save method doesn’t use the use_bom parameter because the json.load and json.dump functions don’t allow easy access to it. If controlling the BOM is required, the best option is to take over the file read/write operation and use json.loads and json.dumps.

When run, this prints:

read: C:\demo\hcc\python\example.json
C:\demo\hcc\python\example.json (1040 bytes)

len(fo)=4 (# of top-level dicts)
Top-level dicts: ['version', 'head', 'body', 'tail']

fo["version"]=1
fo["head"]["title"]='JSON Test File'

Note that in all of these file classes, in lots of use cases the client code doesn’t need to know about the data attribute. The exception is creating a new file with code-supplied data. We can use methods that hide the data attribute — for instance, linefile.add — but otherwise we just write our output directly to the data attribute:

Note that the examples module used in the import statements here is assumed to contain whatever previously mentioned code is necessary for the import to succeed. Which means the contents must be a little fluid. For example, the jsonfile class requires filecontent (which requires fileobj), but the example above assumes the jsonfile class itself is defined in the examples module (which clearly it couldn’t be if its definition imports the module).

Alternately, all the code here could go in one source code file and all imports of “examples” deleted. Just keep it in mind if you use any of these “as is”.

This last class loads various kinds of script files into a list that has been stripped of blank lines and lines that start with a specified comment string:

Note that the save function has been disabled. This class treat script files as read-only.

Lines #39-#45 illustrate a simple use case, reading a vim file. In such files, the comment character is the double-quote (hence line #40).

This is just one example of format-specific file classes. Another easy one is the traditional INI file. A harder one is an XML file. Regardless, it’s helpful to encapsulate access to common file types. All these — minus the demonstration code — can be put into a module file so you can import whatever file classes you need in your work.

That’s all for this time.

Link: Zip file containing all code fragments used in this post.

∅