Classical Chinese DH: Regular expressions¶

By Donald Sturgeon

[View this notebook online] [Download this notebook] [List of tutorials]

Regular expressions¶

A regular expression (a.k.a. regex or RE) is a pattern to be searched for in some body of text. These are not specific to Python, but by combining simple regular expressions with basic Python statements, we can quickly achieve powerful results.

Commonly used regex syntax

To use regexes in Python, we use another module called “re” (this is a very common module and should already be installed).

[Aside: in Python, regexes are often written in strings with a “r” in front of them, e.g. r”德” rather than just “德”. All this does is tells Python not to try to interpret the contents of the string (e.g. backslashes) as meaning something else. The result of r”德” is still an ordinary string variable with 德 in it.]

Exercise 1 (very easy): Change the above code to verify the results of some of the simple example regexes from the slides. Try these ones:

• 而無以為
• 是以.德
• 失.而後.
• 上[仁義]為之
• 後(.)，失\1

For the last of these (“後(.)，失\1”), see what happens to the output when you change group(0) to group(1). (Change it back to group(0) afterwards though, as we will reuse this code using group(0).)

Exercise 2: Write regular expressions to match the following things (you can keep on modifying the example above to check that they work, but you may want to write down your answers somewhere – remember, you can edit this cell by double-clicking on it).

• Match any three characters where the middle character is “之” – i.e. “為之而”, “莫之應”, etc. Modify your regex so that it does not match things with punctuation in them, like “扔之。”.
• Match each “phrase” (i.e. punctuated section) of the text. In other words, the first match should be “上德不德”, the second should be “是以有德”, and so on. You only need to handle the three punctuation marks “。”, “，”, and “；”.
• Match each phrase which contains the term “之” in it. (Double check that you get 5 matches.)

We can do the same kind of thing on an entire text in one go if we have the whole text in a single string, as in the next example. (If we wanted to know which paragraph or chapter each match appeared in, we would want to run the same regex on each paragraph or chapter in turn so that we know which paragraph or chapter each match occurs in.)

Exercise 3

• Often we don’t want to include matches that have punctuation in them. Modify the regex from the last example so that it excludes all the matches where the character after “足” is “，”, “。”, or “；”. (You should do this by modifying the regex; the rest of the code does not need to change.)

• Find all the occurrences of X可X – i.e. “道可道” and “名可名” (there is one more item that should be matched too).

• Modify your regex so you match all occurrences of XYX – i.e. not just “道可道” but also things like “學不學”. You may need to make some changes to avoid matching punctuation – we don’t want to match “三，三” or “、寡、”.

Exercise 4: (Optional) Using what was covered in the previous tutorial, write a program in the cell below to perform one of these searches again, but this time running it once on each paragraph in turn so that you can also print out the number of the passage in which each match occurs.

Dictionary variables¶

One of the advantages of using regexes from within a programming language like Python is that as well as simply finding results, we can easily do things to collate our data, such as count up how many times a regex gave various different results. Another type of variable that is useful here is the “dictionary” variable.

A dictionary variable works in a very similar way to a list, except that whereas in a list the items are numbered 0,1,2,… and accessed using these numbers, a dictionary uses other things – in the case we will look at, strings – to identify the items. This lets us “look up” values for different strings, just like looking up the translation of a word in a dictionary. The things we use instead of numbers to “look up” values in a dictionary are called “keys“.

Dictionaries can be defined in Python using the following notation:

The above example defines one dictionary variable called “my_titles”, and sets values for three keys: “論語”, “孟子”, and “荀子”. Each of these keys is set to have the corresponding value (“Analects”, “Mengzi”, and “Xunzi” respectively). In this simple example, our dictionary gives us a way of translating Chinese-language titles into English-language titles.

We can access the items in a dictionary in a very similar way to accessing items from a list:

Unlike in a list, our items don’t have numbers, and don’t come in any particular order. So one thing we will sometimes need to do is to get a list of all the keys – i.e., a list telling us what things there are in our dictionary.

Often we will store numbers in our dictionary; the keys will be strings, but the value for each key will be a number. This lets us do things like count how many times we’ve seen some particular string – for all of the strings we happen to come across at the same time, using just one dictionary variable. In cases like this, we will often want to sort the dictionary by the values of the keys. One way of doing this is using the “sorted” function:

Don’t worry about the rather complex looking syntax for sorted() – you can just follow this model whenever you need to sort a dictionary (and change “reverse=False” to “reverse=True” if you want to reverse the list):

list_of_keys = sorted(my_dictionary, key=my_dictionary.get, reverse=False)

Using a dictionary, we can keep track of every regex result we found, and at the same time collate the data. Instead of having a long list with repeated items in it, we build a dictionary in which the keys are the unique regex matches, and the values are the number of times we have seen that particular string.

We can use this idea and almost exactly the same code to start answering quite complex questions about patterns appearing in texts. This code can tell us which actual phrases matching a certain pattern occurred most frequently.

For example, in poetry we often find various kinds of repetition. We can use part of the 詩經 as an example, and using a regex quickly find out which repeated XYXY patterns are most common:

Exercise 5: Write a regex to match paired lines of four-character poetry that both begin with the same two characters (examples: “亦既見止、亦既覯止”, “且以喜樂、且以永日”, etc.). Re-run the program above to verify your answer.

Exercise 6: Create a regex to match book titles that appear in punctuated Chinese texts, e.g. “《呂氏春秋》”. Your regex should extract the title without the punctuation marks into a group – i.e. you must use “(” and “)” in your regex. You can test it using the short program below – your output should look like this:

爾雅
廣雅
尚賢
呂氏春秋·順民
呂氏春秋·不侵
左·襄十一年傳
韓詩外傳
廣雅

Now modify your regex so that instead of getting book titles together with chapter titles, your regex only captures the title of the work – i.e., capture “呂氏春秋” instead of “呂氏春秋·順民”, and “左” instead of “左·襄十一年傳”.

Optional: Bonus points if you can also capture the chapter title on its own in a separate regex group at the same time. This is a bit fiddly though, and we don’t need to do it for this exercise.

• Now modify the example code below (it’s almost identical to one of examples above) so that it lists how often every title was mentioned in the 墨子閒詁 (a commentary on the classic text “墨子” – in this example we only use the first chapter, though the code can also be run on the whole text by changing the URN).
• Then modify your code so that it only lists the top 10 most frequently mentioned texts. Hint: “unique_items” is a list, and getting part of a list is very similar to getting part of a string.

Dictionaries also allow us to produce graphs summarizing our data.

Now modify your regex so that you only match texts that are cited as pairs of book title and chapter, i.e. you should only match cases like “《呂氏春秋·順民》” (and not 《呂氏春秋》), and capture into a group the full title (“呂氏春秋·順民” in this example). This may be harder than it looks! You will need to be careful that your regex does not sometimes match too much text.

Re-run the above programs to find out (and graph) which chapters of which texts are most frequently cited in this way by this commentary.

Replacing and Splitting with Regexes¶

As well as finding things, regexes are ideal for other very useful tasks including replacing and splitting textual data.

For example, we saw in the last notebook cases where it would be easier to process a text without any punctuation in it. We can easily match all punctuation using a regex, and once we know how to search and replace, we can just replace each matched piece of punctuation with a blank string to get an unpunctuated text.

We can do a simple search-and-replace using a regex like this:

For very simple regexes that don’t use any special regex characters, this gives exactly the same result as replace(). But because we can specify patterns, we can do much more powerful replacements.

Of course, as usual the power of this is that we can quickly do it for however much data we like:

Another useful aspect is that we can use data from regex groups that we matched within our replacement. This makes it easy to write replacements that do things like add some particular string before or after something we want to match. This example finds any punctuation character, puts it in regex group 1, and then replaces it with regex group 1 followed by a return character – in other words, it adds a line break after every punctuation character.

Regexes and text files¶

Regular expressions can be very useful when we want to transform text from one format to another, or when we want to read text from a file and it isn’t in the format we want.

In this section, instead of using the ctext.org API, we will experiment with a text from Project Gutenberg. Before starting, download the plain text UTF-8 file from the website and save it on your computer as a file called “mulan.txt”. You should save this in the same folder as this Jupyter notebook (.ipynb) file.

Note: you don’t have to save files in the same folder as your Jupyter notebook, but if you save them somewhere else, when opening the file you will need to tell Python the full path to your file instead of just the filename – e.g. “C:\Users\user\Documents\mulan.txt” instead of just “mulan.txt”.

One practical issue when dealing with a lot of data in a string is that printing it to the screen so we can see what’s happening in our program may take up a lot of space. One thing we can do is to just print a substring – i.e. only print the first few hundred or so characters:

One thing that will be handy is if we can delete the English blurb at the top of this file automatically. There are several ways we could do this. One way is to use a negative character class – matching everything except some set of characters – to match all characters that are non-Chinese, and delete them.

The re.sub() function takes three parameters:

1. The regular expression to match
2. What we want to replace each match with
3. The string we want to do the matching in
   It returns a new string containing the result after making the substitution.

[The example below also makes use of another kind of special syntax in a character class: we can match a range of characters by their Unicode codepoint. Here we match everything from U+25A1 through U+FFFF, all of which are Chinese characters. Don’t worry too much about the contents of this regex – we won’t need to write regexes like this most of the time.]

We’ve got rid of the English text, but we’ve now got too many empty lines. Depending on what data is in the text, we might want to remove all the line breaks… but in this case there are some things like chapter titles that are best kept on separate lines so we can tell where the chapters begin and end.

Remember: “\n” means “one line break”, and “{3,}” will match 3 or more of something one after the other (and as many times as possible).

Exercise 7: (Harder) Make another substitution using a regex which removes only the line breaks within a paragraph (and does not remove linebreaks before and after a chapter title). The output should look like this:

序

嘗思人道之大，莫大於倫常；學問之精，莫精於性命。自有書籍以來，所載傳人不少，求其交盡乎倫常者鮮矣，求其交盡乎性命者益鮮矣。蓋倫常之地，或盡孝而不必兼忠，或盡忠而不必兼孝，或盡忠孝而安常處順，不必兼勇烈。遭際未極其變，即倫常未盡其難也。性命之理，有不悟性根者，有不知命蒂者，有修性命而旁歧雜出者，有修性命而後先倒置者。涵養未得其中，即性命未盡其奧也。乃木蘭一女子耳，擔荷倫常，研求性命，而獨無所不盡也哉！

  予幼讀《木蘭詩》，觀其代父從軍，可謂孝矣；立功絕塞，可謂忠矣。後閱《唐書》，言木蘭唐女，西陵人，嫻弓馬，諳韜略，轉戰沙漠，累大功十二，何其勇也。封武昭將軍，凱旋還里。當時筮者謂致亂必由武姓，讒臣嫁禍武昭，詔徵至京。木蘭具表陳情，掣劍剜心，示使者，目視而死。死後，位證雷部大神，何其烈也。去冬閱《木蘭奇女傳》，復知其幼而領悟者性命也，長而行持者性命也。且通部議論極精微，極顯豁，又無非性命之妙諦也。盡人所當盡，亦盡人所難盡。惟其無所不盡，則亦無所不奇。而人奇，行奇，事奇，文奇，讀者莫不驚奇叫絕也。此書相傳為奎斗馬祖所演，卷首有武聖帝序。今序已失，同人集貲付梓。書成，爰敘其緣起如此。

      書於滬江梅花書館南窗之下

Hint: Think about what you need to match to make the change. You may need to include some of the things that your regex matches in the replacement using references (i.e. \1, \2, etc.).

Exercise 8: The text contains comments in it which we might want to delete before doing further processing or calculating any statistics. Create a regex substitution which removes each of these comments.

Example comment: …居於湖廣黃州府西陵縣（今之黃陂縣）雙龍鎮。 => should become …居於湖廣黃州府西陵縣雙龍鎮。

Make sure to check that your regex does not remove too much text!

Exercise 9: Experiment with writing regexes to list things that look like chapter titles in the text. There are several ways this can be done. (There are 32 numbered chapters in this text.)

• Next, use your chapter-detecting regex to add immediately before each chapter the text “CHAPTER_STARTS_HERE”.

Lastly, we can use a regex to split a string variable into a Python list using the re.split() function. At any point in the string where the specified regex is matched, the data is split into pieces. For instance:

Use re.split() to split your full text into a Python list, in which each chapter is one list item. (For simplicity you can ignore things like the preface etc.)

Now we have this data in a Python list, we can do things to each chapter individually. We can also put each of the chapters into its own text file – this is something we will sometimes need to do when we want to use other tools that are not in Python.

Further reading:

• The browser-based Text Tools plugin for ctext.org supports regular expressions – an online tutorial for the plugin is available, which describes how to use it to investigate patterns with regexes.

Classical Chinese DH: Python programming and ctext.org API¶

By Donald Sturgeon

[View this notebook online] [Download this notebook] [List of tutorials]

Variables¶

Variables are named entities that contain some kind of data that can be changed at a later date. You can choose (almost) any name for a variable as long as it is not the same as a reserved word (i.e. has some special meaning in the Python language), though typically these names will be composed of letters a-z. The names given to variables have no special meaning to the computer, but giving variables names that describe their function in a particular program is usually very helpful to the programmer – and to anyone else who may look at your code. Spaces cannot be part of a variable name, so sometimes other allowed characters (e.g. “_”) are used instead for clarity. Although it is possible to use non-English characters for variable names, this is generally inadvisable as it may cause compatibility problems when running the same program on another computer.

A value is assigned to a variable using the syntax “variable_name = new_value“.

A variable only ever has one value at a time. When we assign a second value to a variable, anything that was in it before is lost.

In Python, variable names are case sensitive, so as far as Python is concerned, a variable called “thisone” is completely different from a variable called “ThisOne” or “THISONE”.

We can perform basic arithmetic on variables and numbers using symbols representing arithmetic operators:

Strings¶

One of the most important units of text in most programming languages is the string: an ordered sequence of zero or more characters. Strings can be “literal” strings – string data typed in to a program – or the contents of a variable.

Literal strings have to be enclosed in special characters so that Python knows exactly which part of what appears in the program belongs to the string being defined. You can use either a pair of double quotation marks (“…”) or single quotation marks (‘…’) for this. (Note: most programming languages including Python will not allow the use of ‘full-width’ Chinese / CJK punctuation characters for this purpose.)

Two strings can be joined together (concatenated) using the “+” operator to give a new string:

In Python, each variable has a particular “type”. The most common types are “string”, “integer” (…,-2,-1,0,1,2,…), and “float” (any real number, e.g. 3.1415, -26, …). When joining a string and a number using “+”, we need to specify that the number should be changed into a string:

Sometimes we may need to include “special” characters in our strings, like the return character (\n), or tab character (\t). Also, if we need to include quotation marks as part of our string, we can do this by putting a backslash before each quotation mark (\”).

Doing things with strings¶

Once we have data in a string, we can manipulate it in various ways. Often a program will be designed to work on arbitrary string input, i.e. the program will not know in advance what strings it will be asked to work with. So we need ways of finding out basic things about the string. First of all, how long (in characters) is the string:

N.B. “Characters” here means characters in the technical string sense. It includes, for example, all punctuation symbols, and other “special characters” that may be in the string (such as characters representing line breaks).

We can take a single character from a string and create a new string containing just that character using the notation “string_variable[m]“, where m is a number describing the position of the character we want to copy from the string.

N.B. In Python (and many other languages), the characters in a string are numbered starting from 0. So if a string has a length of 5, its characters are numbered 0, 1, 2, 3, and 4.

If we use a negative value for m, we can do the same thing but counting backwards from the end of the string:

Another useful basic function is making a new string from some part of an existing string – this is called a “substring”.
In Python, we get a substring of a string starting at position m and ending just before position n using the notation “string_variable[m:n]”:

If we want to count characters from the end of a string, instead of from the beginning, we can use a negative number for m (meaning “start from –m characters before the end of the string”) and either omit n entirely (meaning “up to the end of the string”) or use a negative number for n (meaning “up to –n characters before the end of the string”):

There are many other functions for doing things with strings – we will see more of these in week 3. In the meantime, two useful functions are count(), which returns the number of times one string occurs within another string.

Another is replace(), which creates a new string in which all matching occurrence of a substring have been replaced by something else. The general form looks something like this:

string_to_search_in.replace(thing_to_search_for, thing_to_replace_with)

N.B. This function does not change the data in the original variable. It just returns new data with the substitution made.

Lists¶

Lists are another kind of variable that work a lot like strings, except that whereas each location within a string is always exactly one character, each location in a list can be any kind of value, such as a number or a string.

We can make a list variable by separating each list element with commas and enclosing the whole lot in square brackets.

In Python, the items stored in a list are numbered starting from 0. This means if we have 7 items in our list, they are numbered 0, 1, 2, 3, 4, 5, and 6.

If we try to use an item that isn’t in our list, we will get an error.

We can set the value of items in a list just like regular variables.

Often when we process lists in a program, we will need to find out how long the list is. This is because our programs will usually be designed to work with any input of a certain type (for example, to work with any text, not just one we’ve chosen in advance), and so we will only find out many items there are in a particular list when our program is actually run. The len() function tells us how many items are in our list.

Remember that when we use len() on a string, it will give us the length of the string in numbers of characters. So for our days_of_week example:

Make sure you understand why we get this answer here.

True and false¶

Boolean logic – i.e. logic in which things are either true or false – is central to most commonly used programming languages. Typically programs make decisions as to what to do next based on whether some particular expression (e.g. comparison of variables) is true or false. Some basic comparison operators are:

N.B. Assignment, e.g. a=1, uses a single “=”, whereas comparison, e.g. a==2, uses a double “==”.

Often we want to know whether something occurs anywhere within a string. One way to do this is using the in operator. (We will look at more sophisticated types of searching for more complex patterns next week.)

Making decisions¶

Now that we can compare things, we can start to change what we do depending on the outcome of these comparisons.

The simplest type of decision is an “if … then …” decision: if something is true, then do something (otherwise don’t do it).

In Python, indentation (one or more spaces from the left-hand margin) is used to mark blocks of code (sequences of instructions) which are to be followed together. For example:

The “else” keyword can be used after an “if” to do one thing if a condition is true, and some other thing if it is not true:

Also useful are logical operators “and“, “or“, and “not“. These allow us to make more complex decisions based on several factors.

N.B. Matching pairs of brackets are used in complex expressions to remove ambiguity: the innermost brackets are always evaluated first. For instance:

(a and b) or c   # This will only be true when either: 1) a and b are both true; or 2) c is true
a and (b or c)   # This will only be true when *both*: 1) a is true; and 2) either b is true or c is true
a and b or c     # ???? Don't write this - it's not obvious which of the previous two lines it corresponds to

Suggestion: don’t write things like “a and b or c” without brackets, as these are confusing. (There are rules that mean they are not ambiguous, but instead of worrying about these now, always use brackets when mixing and, or, and not in an expression.)

Experiment with changing the values of the three variables below, making sure you understand why you get different results.

Repeating instructions¶

A lot of the power of digital methods comes from the fact that once we have a program that performs some task on an arbitrary object of some kind, we can easily have the computer perform that same task on large numbers of objects. One of the simplest ways of repeating instructions is the “for” loop. Just like with if, in Python we use indentation to indicate exactly which instructions we want repeated. For loops are often used with the range() function:

The range() function takes two parameters that determine what range of numbers we want: this first parameter determines the number we want to begin with, and the second determines the end of the range. Note: the “end” parameter to range() is not inclusive – so range(0,3) will give us a range with the numbers 0,1,2 but not 3.

As well as range(), we can also loop over each character in a string, or over each item in a list (i.e. do some processing for each character or item).

Working with characters in a string is just as easy – we can use the format “for character_variable in string:”

However, sometimes we also need to know where (i.e. at what index) we are in a string.

Sometimes we will need to have one loop inside another loop. In this case, we use progressively larger indentations to indicate which instructions should be repeated in which loop.

Getting Chinese texts¶

If you already have a copy of a text you’d like to process, you can easily read it from a text file into a string variable. However, as the formatting of each text may be different, the exact steps needed to process the text may differ slightly in each case. We will look at how to deal with this in detail next time when we look at regular expressions, since these provide powerful tools to quickly reorganize textual data.

An alternative way of getting textual data for historical texts is to use the ctext.org API, which lets us get the text for many historical texts in a consistent format. Texts are identified using a URN (Uniform Resource Name) – you can see this written at the bottom of the page when you view the corresponding text on ctext.org.

To make it easier to access these texts, we can install a specialized Python module to access the API. Before we can use it, we have to install it. Python makes this very easy to do, and it only needs to be done once; if you followed the instructions in the first tutorial, you should already have this module installed.

Once installed, we can use functions from this module to read textual data into Python variables. If we don’t care about the structure of a text, but only its contents, we can read the text into a single list of strings, each containing a single paragraph, like this:

Reading and writing files¶

To read or write to a file, we must first open it. When we open a file, we must specify both the name of the file, and whether we want to read data from it (“r”), or write data to it (“w”). The file.write() function works very much like print(), but writes its output directly to the file instead of to the screen. When writing to a file, however, we need to explicitly include return characters at the end of each line using “\n”. The file.read() function reads all the data from the file, which you can assign to a python variable.

N.B. Be careful when opening files! When you use “w” to open a file for writing, any file with that name in the same folder as your Python Notebook will immediately be replaced with a new, empty file.

Now take a look in Windows Explorer or Mac OS Finder and make sure you can see where this file is on your computer. It will be helpful for you to know in which folder Python expects files to be by default.

Exercises¶

1.i) Write a program using a for loop to output all of the substrings of length 2 contained in the variable input_string. Your program should produce output like this:

天命
命之
之謂
謂性
性，
，率
率性
性之
...
...
道也
也。

1.ii) Now modify your program so that you first define a variable called “substring_length” containing a number determining the length of substring to be listed. Your new program should still give the same output when run with “substring_length=2”, but should also work with substring_length set to 3, 4, etc. Remember, every line that your program outputs should have exactly substring_length characters in it (including punctuation characters).

1.iii) Modify your program again so that on each line it prints the total number of times that the substring occurs in input_string. For instance, each line beginning “之謂” should now read “之謂 3”, since “之謂” occurs three times in this string.

1.iv) Run your program again, but now use this slightly longer text instead:

input_string = "天命之謂性，率性之謂道，修道之謂教。道也者，不可須臾離也，可離非道也。是故君子戒慎乎其所不睹，恐懼乎其所不聞。莫見乎隱，莫顯乎微。故君子慎其獨也。喜怒哀樂之未發，謂之中；發而皆中節，謂之和；中也者，天下之大本也；和也者，天下之達道也。致中和，天地位焉，萬物育焉。"

Look at what the most frequent 2-grams are with this text. As before, some of them will include punctuation. Do any of these frequent 2-grams which include punctuation relate to facts about the language?

2.i) In the cell below, write a program to find and print out all passages in the Analects that include the term “仁”.

Note: if you’ve run the example program under Getting Chinese texts, the data is already stored in the passages variable.

2.ii) Modify your program so it instead lists only passages that mention both terms “仁” and “義”.

2.iii) Modify your program so it instead lists only passages that mention either the term “愛人” or “知人” but not both.

3) Write another program (if you like, you can copy and paste your answer to the previous question and modify it) to determine which passage in the Analects mentions the term “禮” the greatest number of times.

Hint: Use one variable to track the greatest number of times “禮” has appeared, and another to track which passage it appeared in.

4.i) Write a program in the cell below to store the full text of the Analects into a file on your computer called “analects.txt”. Put each paragraph on its own line, and in front of each paragraph put firstly the number of the paragraph, starting at 1, and secondly the length of the paragraph in characters. Separate each of these three pieces of data with a tab character. The beginning of your file should look like this:

1   41   子曰：「學而時習之，不亦說乎？有朋自遠方來，不亦樂乎？人不知而不慍，不亦君子乎？」
2   61   有子曰：「其為人也孝弟，而好犯上者，鮮矣；不好犯上，而好作亂者，未之有也。君子務本，本立而道生。孝弟也者，其為仁之本與！」

• Open the file in a text editor (e.g. Notepad for Windows, TextEdit for Mac OS X – usually double-clicking on the file you’ve created will do this) and check that the output looks correct.

4.ii) Modify your program so that the character counts only include Chinese characters, i.e. do not count punctuation characters.

4.iii) (Optional) If you have Excel or another spreadsheet program on your computer, try importing the file into it so that you get separate columns for the paragraph number, length, and content. (This may or may not be easy depending on your operating system and spreadsheet program. If you encounter encoding issues, try copying all of the data from the text editor straight into a blank spreadsheet instead.)

5) [Harder] Which passage in the Analects contains a character repeated more frequently than any other character in any other passage – and what is the character?
It will help if you firstly think carefully about what you need to keep track of between passages in order to answer this question.

Further reading

• http://phrasesinenglish.org/explorengrams.html
• Google books n-grams viewer – This provides an easy-to-use example of n-grams in action. Although this is useful as a practical example and for seeing how changes in n-gram patterns are potentially useful, there are many issues with the underlying data which mean it may not be suitable for research purposes, and the data for pre-modern Chinese in particular is not suitable for serious research. The related paper in Nature is certainly worth reading, but some of the arguments made using the data should be approached with a healthy degree of skepticism.

Bonus question

This section is optional as it includes several things we haven’t covered yet and will look at later on in when we look more closely at structured data.

The program below uses a dictionary variable to count all of the 1-grams in the Analects. [A dictionary variable is very similar to a list, except that its items are not numbered 0,1,2,… but instead indexed using arbitrary strings – for instance, my_dictionary[“論語”], which might contain a string value such as “analects”, or a number like 32. The term “dictionary” here is metaphorical, i.e. a dictionary variable often does not contain translations of words from one language to another – though this is one possible use case.]

It then uses the pandas library to select the top ten most frequent 1-grams, and the matplotlib library to draw a bar chart of this data.

Read through the code, and see if you can modify it to find the most frequent 2-grams, 3-grams, etc.

Classical Chinese DH: Getting started¶

By Donald Sturgeon

[View this notebook online] [Download this notebook] [List of tutorials]

Welcome to our first Jupyter Notebook!¶

A notebook is a hypertext document containing a mixture of textual content (like the part you’re reading now) and computer programs – lists of instructions written in a programming language (in our case, the Python language) – as well as the output of these programs.

Using the Jupyter environment¶

Before getting started with Python itself, it’s important to get some basic familiarity with the user interface of the Jupyter environment. Jupyter is fairly intuitive to use, partly because it runs in a web browser and so works a lot like any web page. Basic principles:

• Each “notebook” displays as a single page. Notebooks are opened and saved using the menus and icons shown within the Jupyter window (i.e. the menus and icons under the Jupyter logo and icon, not the menus / icons belonging to your web browser).

• Notebooks are made up of “cells”. Each cell is displayed on the page in a long list, one below another. You can see which parts of the notebook belong to which cell by clicking once on the text – when you do this, this will select the cell containing the text, and show its outline with a grey line.

• Usually a cell contains either text (like this one – in Jupyter this is called a “Markdown” cell), or Python code (like the one below this one).

• You can click on a program cell to edit it, and double-click on a text cell to edit it. Try double-clicking on this cell.

• When you start editing a text cell, the way it is displayed changes so that you can see (and edit) any formatting codes in it. To return the cell back to the “normal” prettified display, you need to “Run” it. You can run a cell by either:

  • choosing “Run” from the “Cell” menu above,
  • pressing shift-return when the cell is selected, or
  • clicking the “Run cell” icon.
• “Run” this cell so that it returns to the original mode of display.

The program in a cell doesn’t do anything until you ask Jupyter to run (a.k.a. “execute”) it – in other words, ask the system to start following the instructions in the program. You can execute a cell by clicking somewhere in it so it’s selected, then choosing “Run” from the “Cell” menu (or by pressing shift-return).

When you run a cell containing a Python program, any output that the program generates is displayed directly below that cell. If you modify the program, you’ll need to run it again before you will see the modified result.

A lot of the power of Python and Jupyter comes from the ability to easily make use of modules written by other people. Modules are included using lines like “from … import *”.
A module needs to be installed on your computer before you can use it; many of the most commonly used ones are installed as part of Anaconda.

“Comments” provide a way of explaining to human readers what parts of a program are supposed to do (but are completely ignored by Python itself). Typing the symbol # begins a comment, which continues until the end of the line.

N.B. You must install the “ctext” module before running the code below. If you get the error “ImportError: No module named ‘ctext'” when you try to run the code, refer to the instructions for how to install the ctext module.

‘Variables’ are named entities that contain some kind of data that can be changed at a later date. We will look at these in much more detail over the next few weeks. For now, you can think of them as named boxes which can contain any kind of data.

Once we have data stored in a variable (like the ‘paragraphs’ variable above), we can start processing it in whatever way we want. Often we use other variables to track our progress, like the ‘longest_paragraph’ and ‘longest_length’ variables in the program below.

Modules allow us to do powerful things like Principle Component Analysis (PCA) and machine learning without having to write any code to perform any of the complex mathematics which lies behind these techniques. They also let us easily plot numerical results within the Jupyter notebook environment.

For example, the following code (which we will go through in much more detail in a future tutorial – don’t worry about the contents of it yet) plots the frequencies of the two characters “矣” and “也” in chapters of the Analects versus chapters of the Fengshen Yanyi. (Note: this may take a few seconds to download the data.)

You can save changes to your notebook using “File” -> “Save and checkpoint”. Note that Jupyter often saves your changes for you automatically, so if you don’t want to save your changes, you might want to make a copy of your notebook first using “File” -> “Make a Copy”.

You should try to avoid having the same notebook open in two different browser windows or browser tabs at the same time. (If you do this, both pages may try to save changes to the same file, overwriting each other’s work.)

Exercises¶

Before we start writing programs, we need to get familiar with the Jupyter Notebook programming environment. Check that you can complete the following tasks:

• Run each of the program cells in this notebook that are above this cell on your computer, checking that each of the short programs produces the expected output.
• Clear all of the output using “Cell” -> “All output” -> “Clear”, then run one or two of them again.
• In Jupyter, each cell in a notebook can be run independently. Sometimes the order in which cells are run is important. Try running the following three cells in order, then see what happens when you run them in a different order. Make sure you understand why in some cases you get different results.

• Some of the programs in this notebook are very simple. Modify and re-run them to perform the following tasks:

  • Print out the squares of the numbers 3 through 20 (instead of 1 through 12)
  • Print out the cubes of the numbers 3 through 20 (i.e. 3 x 3 x 3 = 27, 4 x 4 x 4 = 64, etc.)
  • Instead of printing passages from the first chapter of the Analects, print passages from the Daodejing, and determine the longest passage in it. The URN for the Daodejing is: ctp:dao-de-jing

• Often when programming you’ll encounter error messages. The following line contains a bug; try running it, and look at the output. Work out which part of the error message is most relevant, and see if you can find an explanation on the web (e.g. on StackOverflow) and fix the mistake.

• Sometimes a program will take a long time to run – or even run forever – and you’ll need to stop it. Watch what happens to the circle beside the text “Python 3” at the top-right of the screen when you run the cell below.
• While the cell below is running, try running the cell above. You won’t see any output until the cell below has finished running.
• Run the cell below again. While it’s running, interrupt its execution by clicking “Kernel” -> “Interrupt”.

• The cell below has been set as a “Markdown” cell, making it a text cell instead of a program (“code”) cell. Work out how to make the cell run as a program.

for number in range(1,11):
print(“1/” + str(number) + ” = ” + str(1/number)) # In many programming languages, the symbol “/” means “divided by”

• Experiment with creating new cells below this one. Make some text cells, type something in them, and run them. Copy and paste some code from above into code cells, and run them too. Try playing around with simple modifications to the code.
• (Optional) You can make your text cells look nicer by including formatting instructions in them. The way of doing this is called “Markdown” – there are many good introductions available online.
• Lastly, save your modified notebook and close your web browser. Shut down the Python server process, then start it again, and reload your modified notebook. Make sure you can also find the saved notebook file in your computer’s file manager (e.g. “Windows Explorer”/”File Explorer” on Windows, or “Finder” on Mac OS X).

Further reading:

• Jupyter Notebook Users Manual, Bryn Mawr College Computer Science – This provides a thorough introduction to Jupyter features. This guide introduces many more features than we will need to use, but is a great reference.