The last two posts looked at Python list comprehensions. [See Simple Tricks #2 and Simple Tricks #3] This time we look at file handling, one of the most common tasks programmers deal with, especially with script languages such as Python.

Python’s native file object, created with the built in open function, is simple and easy, but here are some tricks that make file access even simpler and easier.

First, though, let’s consider how we use filenames. On most file systems, a file has specific name, the filename, and a directory (or folder) path, the pathname, that specifies where it is on disk. On some systems, the pathname can begin with a drive or resource identifier. These two comprise the full specification, the filespec, identifying the desired file.

Python, to a great extent in os.path, provides a rich set of functions for dealing with these aspects of a file specification. Most relevant here is the os.path.join function, which is the main tool for constructing the filespec. Canonically:

The above fragment assumes that pathname, subdir, and filename are defined strings with relevant text. Note how the path.join function combines multiple parts into a single filespec string. It takes care of the separator character, usually a forward or backward slash, between parts of the file specification.

A general rule of thumb is to always make use of the path.join function to keep file path information separate from file name information. This gives you much more flexibility when (not if, when) file locations change.

It’s not uncommon to write a function that, given a filename, opens a file and either does something with it or returns data from that file. Typically, then, the function takes the file’s name as a parameter. I’ve found it useful to have the function take both the filename and the pathname. For one thing, this allows me to specify a default pathname. Here’s an example:

Line #3 defines a pathname, which the function definition on line #5 uses as the default value for fpath. (However, it requires a value for fname.) We can imagine that the function takes other arguments, as represented by the *args parameter.

What’s important here is line #6, which creates the filespec, uses the fpath variable unless it has been explicitly set to None by the calling code. In that case, it uses the fname variable as the entire filespec. This allows for multiple ways to call this function depending on the desired effect.

When run, this prints:

Filename: C:\users\me\proj\foo.txt
Filename: foo.txt
Filename: c:\users\me\proj\blog\foo.txt

Filename: C:\users\me\proj\temp\foo.txt
Filename: temp\foo.txt

Filename: C:\users\me\proj\blog\foo.txt
Filename: C:\users\me\proj\blog\foo.txt

The first three show, respectively, using the default filepath, setting it to None, and supplying a different filepath. Setting fpath to None completely eliminates path information from the filespec.

The middle two show how path information can be part of the filename but are otherwise like the first two examples. The last two show what happens when full path information is passed in fname, even if the default in fpath is used by the path.join function. The complete filespec in fname overrides the fpath value. Even so, callers wanting to override the fpath default should set it to None.

Here’s a slight variation on how we treat fname and fpath:

The only difference is that, rather than set fpath to a default, we set it to None. This makes the default behavior treating fname as containing the entire filespec. This is handy if your code usually has a full filespec rather than filename and pathname pairs.

When run, this prints:

Filename: foo.txt
Filename: foo.txt
Filename: c:\users\me\proj\blog\foo.txt

Filename: temp\foo.txt
Filename: c:\users\me\temp\foo.txt

Filename: C:\users\me\proj\blog\foo.txt
Filename: C:\users\me\proj\blog\foo.txt

Compare this to the first one to see how this second version of the function treats the inputs differently.

The first example is better when your code usually has just a filename and usually uses some default location for the files. That said, hardcoding any specific file or path information into your code has (usually eventually bad) consequences. At the very least, use a global variable that only needs to be changed in one location.

Despite that I almost always deal in separate filenames and pathnames, I generally use the second version to avoid having to specify a default pathname.

Before we look at tricks for dealing with files, let’s review the native capability:

This uses the path.join function to create the filespec (line #6) and then the open function (line #7) to create fp, an open file object.

Line #8 uses the readlines method to read all the lines of the file into the lines variable. This assumes the file is a typical text file comprised of lines of text. Once the data has been read, line #9 calls the close method to release the file (something you should always do, even when just reading a file).

The rest, line #11 through line #16, just prints the file lines. Note that we use the end parameter to prevent the print function from applying a newline because the line already has a newline at the end.

But this bare-bones example lacks a vital feature: error handling. In particular, if the readlines method fails, we never reach the close method to properly close the file. At the very least, we prefer to write it like this:

The try-catch configuration ensures that errors reading the file are caught and handled. More importantly, the finally block guarantees we always call the close method. We can also leverage the else block for the status report, which gathers all the file-handling into the try structure. The remainder (lines #19-#22), as in the previous example, just print the file lines.

In this example, we print the error and re-raise the exception, but what action we actually take depends on the overall context of the code.

Python offers a much shorter (and generally preferred) way to do largely the same thing:

But this version has no opportunity to react to the exception locally (the calling function needs to catch it and deal with it). And the status report, as in the first example, is no longer coupled so nicely with the file operation.

As a general rule, though, this last example is the “Python way” of doing it.

I’ve long been a fan of object-oriented programming, so naturally my approach to file handling convenience involves defining some new classes. The first one is a base class that implements some fundamental file handling:

There’s a fair amount packed into this! New instances, line #8, take a filename parameter assumed to be a filespec such as would be passed to the open function. Lines #10-#18 break the filespec into its parts and assign those to various instance attributes. Lines #21-#23 establish defaults for the size and file time attributes. Lines #26-#28 establish default for the directory and file flags and set the exists flag using the path.exists function. If the file does exist, lines #29-#43, set the file and directory flags, the file size attribute, and the two file times (created and updated).

Lines #45-#50 provide obvious aliases for the three flags by overriding the dunder getattribute method. Lines #52-#54 implement the dunder bool method and couple it to the file exists flag. Thus, a file instance object can be treated as a boolean that returns True if the file exists. Lines #56-#61 allow file instance objects to be sorted based on the filename. Lastly, as we always should, line #63 implements the “to string” method dunder str.

Some of the above might be optional if your fileobj class is intended for a specific use. For instance, you might never need to sort instance objects by filespec, so you don’t need to implement the comparison operators. You could also go the other direction and make them more powerful by, if the names match, comparing sizes, and if those match, comparing file dates. You might also prefer to use the @property decorator to create read-only properties for the exists, isdir, and isfile attributes [for details see Python Descriptors, part 1 and part 2]. Or you might decide you don’t need the properties and will use the flag attributes if necessary. The code above should be taken only as a guide.

Lastly, lines #65-#79 test the class by creating an instance object and printing out its many attributes. When run, this prints:

fo=C:\demo\hcc\python\the-dream.txt (3604 bytes)
fo.fullname='C:\\demo\\hcc\\python\\the-dream.txt'
fo.pathname='C:\\demo\\hcc\\python'
fo.filename='the-dream.txt'
fo.name='the-dream'
fo.ext='txt'
fo.fsize=3604
fo.created=datetime.datetime(2024, 6, 7, 15, 52, 8)
fo.updated=datetime.datetime(2024, 6, 7, 15, 52, 28)
fo.exists=True
fo.isfile=True
fo.isdir=False

Note that we can use this class for some kinds of file operations. For example, we could use it, and the Python listdir function, to create a recursive directory listing function that included file sizes and dates. (Leave a comment to let me know if you’d be interested in seeing an example in a future Simple Trick post.)

To represent a physical file with contents, we can extend our base class with an abstraction with a key attribute and some useful methods:

New instances (lines #6-#9) still take a filespec, but we add the data attribute intended to hold the file’s data (line #9). We also implement the dunder len, getitem, setitem, and iter methods.

Because this is an abstract class, and the data attribute is set to None, none of these methods work (they’ll all raise an Exception). They depend on the subclasses we’ll make for dealing with different kinds of files.

Which is where we’ll pick up next time.

Link: Zip file containing all code fragments used in this post.

∅