The chapter provides instructions and examples of using computing skills for health data and technology research.

Programming languages are written using text editor applications. These applications allow users to create and edit free text, which can then be run as programs. Text editors differ in complexity, some including extra functionality for easier, more efficient programming. Text editors with auto-complete suggest common functions or existing variables as the programmer begins to type, which the programmer can then select without needing to finish typing. Some text editors offer options to run individual lines of code or entire programs while editing files.

Popular Text Editors

Microsoft Visual Studio Code (recommended)

• Available for Mac, Windows, and Linux operating systems

• Includes support for debugging, syntax highlighting, auto-complete, and additional user-friendly functionality

• Download Microsoft Visual Studio Code

Jupyter Notebooks

• Web application text editor, no download necessary

• Includes options for interactive output (HTML, images, videos, LaTeX, and custom MIME types), support for big data tools, such as Apache Spark, and options for sharing notebooks with others

• Run individual lines of code or entire programs at once

• Get started with Jupyter Notebooks

VI/VIM

• Highly configurable

• Included in most UNIX operating systems (e.g., Linux, or MacOS), no download necessary

• Write files from the Terminal

• Interactive VIM Tutorial

Emacs

• Highly configurable

• Included in most UNIX operating systems (e.g., Linux, or MacOS), no download necessary, also available for Windows

• Wide range of built-in features for text editing, such as syntax highlighting, automatic indentation, and search and replace

• Learn more about Emacs

Pico

• Included in most UNIX operating systems (e.g., Linux, or MacOS), no download necessary

• Most of the editing commands are displayed at the bottom of the editing screen for easy reference

• Learn more about Pico

Unix is a family of operating systems officially trademarked as UNIX®. These operating systems are computing environments that are optimized for multi-tasking across multiple users. The original system was developed by AT&T in 1969 as a text only system. There are many Unix variants or Unix-like systems (e.g. GNU/Linux, Sun Solaris, IBM AIX, and Mac OS X). On Windows, Cygwin is a program that provides a Unix-like environment.

The main components of a Unix operating system include:

• Kernel – bridge between hardware (i.e. silicon) and application (i.e. software)

• Shell – command line interface to enable user interaction with the system

• File System – the organization structure for how files are stored

Unix File System

The Unix file system organizes files and directories into a hierarchical structure like the root system of a tree.

• The "root" directory (e.g. "/") is the top of the hierarchy.

  • Standard directories within the root directory:

    • /bin and /usr contain commands needed by system administrators and users

    • /etc contains system-wide configuration files and system databases

    • /home contains the home directory (~) for each user (In some systems, the home directories may be in a different location such as /users or /Users)

• When traversing directories

  • working directory (.) is the directory that a user currently is in

  • parent directory (..) is the directory above the working directory

  • path or pathname specifies where a user is in the file system

  • full path or absolute path points to the same location regardless of the working directory (i.e., it is written in reference to the root directory)

  • relative path is the path relative to the working directory

If the working directory is the home directory for bcbi, the full path for the course directory is /home/bcbi/course while the relative path is just course. A schematic of this is below:

If code then becomes the working directory, the full path for the data directory from there is /home/bcbi/course/data while the relative path is ../data. A schematic of this is below:

Understanding Directory/File Ownership and Permissions

Unix Shell

The Unix shell provides a command line interface for interacting with the operating system and is where commands are entered. An example below is a Mac OS X Terminal Shell logged into a RedHat Linux Server as user_name.

• The prompt may look different depending on your shell (e.g., Bourne shell [sh], C shell [csh], or Bourne-Again shell [bash])

• Default prompts include $ and %

• The prompt # typically appears when logged in as the superuser or root user who can do anything on the system, so should be restricted to trusted users, used only when necessary and with caution. While you may be able to do this on a system you control, you are unlikely to ever have root priviledges on a shared computing resource (e.g. Oscar or Stronghold at Brown University)

• The prompt can be configured to include additional information such as hostname, username, and pathname (e.g., computer:/home/bcbi/course bcbi $).

Unix Commands

There are many Unix commands. Some commands will display output and then return to the shell prompt while others will just return to the shell prompt to indicate that it has executed the last command.

Unix command syntax:

• Case-sensitive (pwd ≠ PWD)

• May involve one or more arguments

• Argument may be an option (or flag or switch) for that command

• Argument may be a file or directory

To get to a Unix shell on your computer:

• For Mac, launch the Terminal application (under Applications → Utilities → Terminal)

• For Linux, launch the Terminal application

• For Windows, launch the PowerShell application

#Run applications like cal:
$ cal
    January 2020
Su Mo Tu We Th Fr Sa
          1  2  3  4
 5  6  7  8  9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30 31

Get help from manual (man) pages on commands: (Use spacebar or up and down arrows to scroll through pages and then press q to quit)

$ man cal

Determine what directory you are currently in with pwd (present working directory):

$ pwd
/home/bcbi

Get a listing of current directory contents using ls:

$ ls

Create course directory using mkdir: (Replace course with class name - e.g., methods2020 or biol6535)

$ mkdir course

Get a listing of current directory contents with details using ls:

$ ls
course
$ ls -l
total 4
drwxrwxr-x. 2 bcbi bcbi 4096 Jan 30 12:53 course

Change into course directory using cd: (Replace course with class name - e.g., methods2020 or biol6535)

$ cd course
$ pwd
/home/bcbi/course

Working with Data

File and Directory Navigation

System Information

Analyzing and Searching Files

Common Shortcuts

File Content Browsing

File Permissions

Exercises

• Analyze the MIMIC-IV Demo Files Using Unix Commands - Forthcoming!

• Analyze the SyntheticRI Demo Files Using Unix - Forthcoming!

Resources

• Brown CCV: Quick Reference / Common Linux Commands

• Brown CCV: Linux Basics for Oscar (requires Brown login)

• Unix in a Nutshell

• Data Science at the Command Line

All major operating systems organize files into hierarchical directories. Understanding these file directory structures is vital when interacting with data files using Unix commands or a programming language.

This page describes file directory structures generally as well as some of the differences between file directory structures within different operating systems.

Hierarchical Structure

Directories allow users to group files into an organized structure. They are typically visualized like root systems of trees, the highest level of which is called the "root directory". Subdirectories branch down from the root directory, containing files as well as additional subdirectories.

Directories and files are typically described using the path used to reach them through the directory structure, starting with the root directory. In Linux and Mac operating systems, the root directory is indicated as "/" (In Windows OS, the root directory is indicated as "\"). An additional "/" (or "\" for Windows OS) is placed between each object in the path.

For example, looking at Figure 1, File_B1a2 could be described with:

/Directory_B/Directory_B1/Directory_B1a/File_B1a2

GUI

All major operating systems also provide users with a graphical user interface, or GUI (often pronounced "gooey"), which allows interaction with software and files through visual icons. If you are not already familiar with accessing files and directories through the command line, you are likely familiar with using a GUI file system. While not the recommended method for interacting with files while programming, the GUI file system can be a useful tool for visualizing a directory structure.

Figure 2 displays the GUI file system for a computer running MacOS. Though the GUI directory structure is visualized horizontally, the "root system" is still clearly visible. Using its complete path, the file "medication_data" should be described as:

/Users/<username>/Documents/project_a/data_files/medication_data

Resources

• IBM's Directories Documentation

• Computing Skills / Unix

Instructions for installing Julia on macOS and Windows operating systems can be found here.

Package managers such as Homebrew (macOS and Linux) and Chocolatey (Windows) can be used to facilitate installation.

For most users, it is recommended to download the current stable release from https://julialang.org/downloads/.

Some developers might wish to use a different version, or to switch between versions. For this, the Juliaup version manager can be useful.

Julia is also available for use in Brown's Computing Environments:

• Oscar (for high-performance computing)

• Stronghold (for secure computing)

Resources

• Julia Documentation: Installation

Julia is an open source dynamic programming language for high-level, high-performance numerical computing [1]. Julia provides ease and expressiveness (similar to R, MATLAB, and Python), but also supports general programming [2].

Development of Julia began in 2009, and the first version was released in February 2012. The current version of Julia is 1.11 (as of November 2024).

References

1. Julia Website

2. Julia Documentation

Resources

• Julia Performance Metrics

• ThinkJulia: How to Think Like a Computer Scientist

• Julia Data Science

• Learn X in Y Minutes: X=Julia

• Julia Cheat Sheet

• Introducing Julia Wikibook

• JuliaHealth

GitHub is a code hosting platform that allows developers to create, store, manage, and share their code. It uses Git software, providing the distributed version control of Git plus access control, bug tracking, software feature requests, task management, continuous integration, and wikis for every project. Refer to GitHub Docs for additional GitHub documentation and tutorials.

Saving and Retrieving Code Changes

Like other cloud platforms (e.g., Google Docs), GitHub allows users to work on projects together. Please note, code changes must be manually saved. GitHub does not automatically save your work. To save changes, open the Terminal application, navigate to the cloned repository, and run the following commands, replacing "INSERT PROGRESS NOTE" with brief description of changes.

git add -A
git commit -"INSERT PROGRESS NOTE"
git push

• git add -A : adds all your code changes to the GitHub repository

• git commit -m"INSERT PROGRESS NOTE" : adds a note to the commit which you and your team can reference later. This note should be brief and informative, describing the purpose of your code changes.

• git push : saves your code changes to the GitHub repository.

If multiple users are pushing code changes to your GitHub repository, make sure to retrieve or "pull" these edits before you begin making code changes. To do so, open the Terminal application, navigate to the cloned repository, and run the following command. If you have made any code changes, you will need to save them first for the pull to work.

git pull

When your are making code changes, you should git pull before making any edits. This will keep your team from encountering "merge conflicts", which can become difficult to troubleshoot. To mitigate merge conflicts, make sure to communicate with your team. Inform your team whenever you push new code changes so that everyone is always working one the most updated version of the code.

Resolving Merge Conflicts

Merge conflicts happen when you attempt to merge code branches that have competing commits. They are often caused by users making code changes without pulling first. To resolve a merge conflict, work through the following steps:

1. Identify the location of the merge conflict.

2. Manually edit the conflicted file from a single machine, selecting the changes you want to keep in the final merge.

3. Push the selected changes to GitHub.

All team members should pull the corrected changes from GitHub before continuing to make code changes.

Resources

• GitHub Documentation

• Conventional Commits

This page provides syntax for using numbers and mathematic operations in Julia. Each section includes an example to demonstrate the described syntax and operations.

Types of Numbers

• Integer (positive and negative counting number) - e.g., -3, -2, -1, 0, 1, 2, and 3

  • Signed: Int8, Int16, Int32, Int64, and Int128

  • Unsigned: UInt8, UInt16, UInt32, UInt64, and UInt128

  • Boolean: Bool (0 = False and 1 = True)

• Float (real or floating point numbers) - e.g., -2.14, 0.0, and 3.777

  • Float16, Float32, Float64

Use typeof() function to determine type

Input:

# Define two variables x and y
x = 100
y = 3.14

# Print out the variable types for each
println(typeof(x))
println(typeof(y))

Output:

Int64
Float64

Arithmetic Operators

Input:

# Demonstrates different math operations
using Printf

n1 = 7    # First number
n2 = 3    # Second number
 
# Output results of different math operations
println("$n1 + $n2 = $(n1 + n2)")             # Addition 
println("$n1 - $n2 = $(n1 - n2)")             # Subtraction 
println("$n1 * $n2 = $(n1 * n2)")             # Multiplication 
println("$n1 / $n2 = $(n1 / n2)")             # Division 
@printf("%d / %d = %.2f\n", n1, n2, n1 / n2)  # Print to 2 decimal places
println("$n1 ^ $n2 = $(n1 ^ n2)")             # Power/Exponent
println("$n1 % $n2 = $(n1 % n2)")             # Modulo/Remainder

Output:

7 + 3 = 10
7 - 3 = 4
7 * 3 = 21
7 / 3 = 2.3333333333333335
7 / 3 = 2.33
7 ^ 3 = 343
7 % 3 = 1

Comparison Operators and Functions

Input:

# compare.jl                                                                                                 
# Demonstrate comparison operators                                                                               

# Assign values to variables using parallel assignment                                                           
c1, c2, c3, c4 = 25, 50, 75, 50
println("c1 = $(c1), c2 = $(c2), c3 = $(c3), c4 = $(c4)")

# Output results of different comparison operations                                                             
 
# Testing equality                                                                                               
println("  c1 = c3 is $(c1 == c3)")
println("  c2 = c4 is $(isequal(c2, c4))")

# Changing values using abbreviated assignment operators                                                        
c1 *= 3    	# Shorthand for c1 = c1 * 3                                                                       
c4 += 1    	# Shorthand for c4 = c4 + 1                                                                       

println("c1 = $(c1), c2 = $(c2), c3 = $(c3), c4 = $(c4)")
 
# Testing less than and greater than
println("  c1 < c2 is $(c1 < c2)")
println("  c4 <= c2 is $(c4 <= c2)")
println("  c1 > c2 is $(c1 > c2)")
println("  c3 >= c2 is $(c3 >= c2)")

Output:

c1 = 25, c2 = 50, c3 = 75, c4 = 50
  c1 = c3 is false
  c2 = c4 is true
c1 = 75, c2 = 50, c3 = 75, c4 = 51
  c1 < c2 is false
  c4 <= c2 is false
  c1 > c2 is true
  c3 >= c2 is true

Exercises

• Create a Health Calculator Using Julia - Forthcoming!

Resources

• Julia Documentation: Integers and Floating Point Numbers

• Julia Documentation: Mathematical Operations and Elementary Functions

• Julia Documentation: Numbers

• Julia Documentation: Mathematics

• Think Julia: Chapter 1 - The Way of the Program

Regular expressions (regex) are powerful tools for pattern matching and text processing. They are represented as a pattern that consists of a special set of characters to search for in a string str.

This page provides syntax for regular expressions in Julia . Each section includes an example to demonstrate the described methods.

Functions

Character Class

Character class specifies a list of characters to match ([...] where ... represents the list) or not match ([^...])

Anchors

Anchors are special characters that can be used to match a pattern at a specified position

Repetition and Quantifier Characters

Repetition or quantifier characters specify the number of times to match a particular character or set of characters

Input:

# regex.jl
number1 = "(555)123-4567"
number2 = "123-45-6789"

# check if matches
if occursin(r"\([0-9]{3}\)[0-9]{3}-[0-9]{4}", number1)
   println("match!")
end

if occursin(r"\([0-9]{3}\)[0-9]{3}-[0-9]{4}", number2)
  println("match!")
else
  println("no match!")
end

# capture matches
# use parentheses to "capture" different parts of a regular 
# expression for later use the first set of parentheses corresponds 
# to index 1, second to index 2, etc.

number_details = match(r"\(([0-9]{3})\)([0-9]{3}-[0-9]{4})", number1)

if number_details != nothing
   area_code = number_details[1]
   phone_number = number_details[2]

   println("area code: $area_code")
   println("phone number: $phone_number")
end

Output:

match!
no match!
area code: 555
phone number: 123-4567

Resources

• Julia Documentation: Manual - Strings (see Regular Expressions)

• Think Julia: Chapter 8 - Strings

• Regular Expressions 101

• Regular Expressions Library

• Regular Expressions Cheat Sheet

"Hello, World!" Program

This is the typical first program for those new to a general purpose programming language like Julia. It can be used to test that the Installation of Julia is working and also introduce Julia's basic syntax using the REPL environment or running code written using a Text Editor at the Unix command line.

Input:

# hello.jl
# This is a single line comment
#= 
This is a block comment to show
comments across multiple lines.
=#

print("Hello, World!")

Output:

Hello, World!

Here are variations of the "Hello, World!" programming using variables and different print statements.

Input:

# hello2.jl

greeting = "Hello, World!"

print(greeting) # print greeting
print("Greeting 1: $greeting") # print greeting as part of a string phrase
print("Greeting 2: $greeting\n") # print with newline (\n) character
println("Greeting 3: $greeting") # println automatically adds the newline character

Output:

Hello, World!
Greeting 1: Hello, World!
Greeting 2: Hello, World!

Greeting 3: Hello, World!

Variable Assignment

In order to assign variables in Julia, you write the desired name for your variable, an = sign, and what the value of the variable should be.

Input:

x = 7
x

Output:

7

Comments

• We can write comments on our code, which do not run, to describe what certain lines of code or section of code do

  • These comments are just for the programmer, they will not appear anywhere in the output and just are there to explain what the code is doing or to provide helpful notes

  • To make a comment in Julia, you can use the “#” symbol and then type your comment

• Sometimes you might want to write longer comments that span multiple lines – to do this you can surround these comments with #= above the start as well as =# below the end

Input:

# Assigns variable x to have value 7
x = 7

#=
Now we want to print out what x is. We can do this by simply typing x and 
hitting run. This comment spans multiple lines. These types of comments are 
useful when describing complex functions or algorithms.
=#

x

Output:

7

Print Statements

Without using a print statement, Julia will only print out the most recent item that has an output. In order to print multiple things, we can use the print() or println() functions.

Input:

Output:

7
10

Exercises

• Use Julia in Brown Oscar Computing Environment - Forthcoming!

• Use Julia in Brown Stronghold Computing Environment - Forthcoming!

Resources

• Julia Documentation: Variables

• Julia Documentation: Scope of Variables

• Think Julia: Chapter 1 - The Way of the Program

• Think Julia: Chapter 2 - Variables, Expressions and Statements

Julia comes with a full-featured interactive command-line REPL (read-eval-print loop) built into the julia executable. In addition to allowing quick and easy evaluation of Julia statements, it has a searchable history, tab-completion, many helpful keybindings, and dedicated help ? and shell modes ;. [1]

This page provides examples of using REPL on the command line.

Julia REPL Example (local)

• Type julia in terminal to launch REPL

Julia REPL Help Pages (local)

• Type "?" to enter help pages within REPL

• Type a function from Julia to read help pages (ex: println)

References

1. Julia Contributors. (n.d.). REPL - Standard Library - Julia Language. Retrieved May 1, 2024, from https://docs.julialang.org/en/v1/stdlib/REPL/

Resources

• Julia Documentation: The Julia REPL

• Julia Cheat Sheet (see REPL)

• Practice with this web-based Julia REPL

In computer programming, a package is a collection of modules or programs that are often published as tools for a range of common use cases, such as text processing and doing math. Programmers can install these packages and take advantage of their functionality within their own code.

This page provides instructions for installing, using, and troubleshooting packages in Julia.

Installing Packages

• Start Julia REPL by typing the following in Terminal or PowerShell (Note: do not need to type $ - this is to indicate the shell prompt)

• Go into REPL mode for Pkg, Julia’s built in package manager, by pressing ]

• Update package repository in Pkg REPL

• Add packages in Pkg REPL

• Check installation

• Get back to the Julia REPL and exit by pressing backspace or ^C.

• To see REPL history

Using Packages

Troubleshooting

• If you get an error like: ERROR: SystemError: opening file "C:\\Users\\User\\.julia\\registries\\General\\Registry.toml": No such file or directory

  • Delete C:\\Users\\User\\.julia\\registries where User is your computer’s username and try again

  • https://discourse.julialang.org/t/registry-toml-missing/24152

Resources

This page provides syntax for strings and characters in Julia as well as some of their associated functions. Each section includes an example to demonstrate the described syntax or function.

Characters and Strings

• Char is a single character

• String is a sequence of one or more characters (index values start at 1)

Some functions that can be performed on strings

Use typeof() function to determine type

Input:

Output:

Resources

Many Julia programs involve the input and output of files. When analyzing a dataset, that dataset file will need to be pulled into your program (input). If you want to see the results of your analysis, your program will need an output.

This section provides the syntax for inputing files (reading) and outputting results (writing) use base Julia (i.e., no packages such as CSV.jl).

UC Irvine Machine Learning Repository: Adult Data Set

• Tabulate and report counts for sex in from the .

Dataset (example lines from adult.data)

Input (process_file.jl)

Output

Terminal

Exercises

• Analyze the MIMIC-IV Demo Files Using Julia - Forthcoming!

• Analyze the SyntheticRI Demo Files Using Julia - Forthcoming!

Resources

In computer science, control flow (or flow of control) is the order in which individual statements, instructions or function calls of an imperative program are executed or evaluated.

This page provides syntax for some of the common control flow methods in Julia . Each section includes an example to demonstrate the described methods.

Use Cases and Syntax

• Test if a specified expression is true or false

• Short-circuit evaluation

  • Test if all of the conditions are true x && y

  • Test if any of the conditions are true x || y

  • Test if a condition is not true !z

• Conditional evaluation

  • if statement

  • if-else

  • if-elseif-else

  • ?: (ternary operator)

Conditional Statements

Input:

Output:

Loops

• Repeat a block of code a specified number of times or until some condition is met.

• while loop

• for loop

• Use break to terminate loop

Input:

Output:

Comparison Operators and Functions

Input:

Output:

References

Resources

In computer programming, a collection is a grouping of some variable number of data items (possibly zero) that have some shared significance to the problem being solved and need to be operated upon together in some controlled fashion.

This page provides syntax for different types of collections and data structures in Julia (arrays, sets, dictionaries, etc.). Each section includes an example to demonstrate the described methods.

Arrays

Arrays are ordered collection of elements. In Julia they are automatically indexed (consecutively numbered) by an integer starting with 1.

Creating arrays

Creating array from string

Accessing elements

Adding and removing elements

Sort and unique

Compare arrays

Convert array to string

Input:

Output:

Sets

Sets are an unordered collection of unique elements.

Creating sets

Interacting with sets

Comparing sets

Input:

Output:

Dictionaries

Dictionaries are unordered collection of key-value pairs where the key serves as the index (“associative collection”). Similar to elements of a set, keys are always unique.

Creating dictionaries

Accessing dictionaries

Converting dictionaries

Sorting dictionaries

Input:

Output:

References

Resources

DataFrames.jl is a Julia package that provides a set of tools for working with tabular data in Julia. Its design and functionality are similar to those of (in Python) and data.frame, and (in R), making it a great general purpose data science tool.

This page provides examples of using DataFrames.jl, demonstrating the syntax and common functions within the package.

Example

Install and Load DataFrames.jl Package

Create Dataframe

Display Dataframe

Input:

Output:

First two lines of dataframe:

Input:

Output:

Last two lines of dataframe:

Input:

Output:

Describe Dataframe

Dataframe size:

Input:

Output:

Dataframe column names:

Input:

Output:

Dataframe description:

Input:

Output:

Accessing DataFrames

Get "age" column (different ways to call the column)

Input:

Output:

Get row

Input:

Output:

Get element

Input:

Output:

Get subset (specific rows and all columns)

Input:

Output:

Get subset (all rows and specific columns)

Input:

Output:

Get subset (all rows meeting specified criteria - numbers)

Input:

Output:

Get subset (all rows meeting specified criteria - strings)

Input:

Output:

Get subset (all rows meeting specified criteria)

Input:

Output:

Add Column

New columns with specified values

Input:

Output:

New column with calculated value

Input:

Output:

Get counts/frequency

Input:

Output:

Transform DataFrame

sort

Input:

Output:

stack (reshape from wide to long format)

Input:

Output:

unstack (reshape from long to wide format)

Input:

Output:

Traversing DataFrame (for loops)

sort

Input:

Output:

Exercises

• Analyzing Health Datasets with DataFrames in Julia - Forthcoming!

References

Resources