Version control systems

VCS are systems that manage changes in scripts, documents, binary files…

What is git?

git is a VCS, one of many (svn, mercurial…)

UNIX (Linux & Mac)

Chances are that you already have git installed in your system, just type git -v in your terminal to check.

If is not installed:

• For linux, use your distribution package manager for installing, i.e. for Ubuntu, just type

$ sudo apt install git-all

• For Mac, just follow the official instructions at https://git-scm.com/download/mac.

Windows

In Windows, follow the instrunctions at https://git-scm.com/download/win

This will install 3 utilities, Git GUI, Git Bash and Git CMD, as well as it will allow using git inside RStudio or VSCode.

Configuring our git user

$ git config --global user.name "John Doe"
$ git config --global user.email johndoe@example.com

Projects everywhere!

Is always a good idea to have data, scripts and documents from a same task organized. Folders, RStudio projects, git projects and a mix of them help us with this, and make some healthy decisions mandatory.

(Mandatory meme)

So… we need a folder

A git projects is nothing else than a folder that is monitored for changes.
So the first thing to do is create a folder to work through this course.

In my case I’m gonna create a folder called project_iris.

Initialize the git project

Once we are in our folder, we start the git project with the git init command:

$ git init

> Inicializado repositorio Git vacío en /data/project_iris/

Initialize the git project

Check the status of the git project

We have our fresh git project just initialized, let’s check the state of the repository with git status:

$ git status

1En la rama main

2No hay commits todavía

no hay nada para confirmar (crea/copia archivos y usa
3"git add" para hacerles seguimiento)

1

Branch, more on branches later

2

No commits done yet, more on commits later

3

Helpful info

Make changes!

Basically, that’s it. We are now using git to track changes, so let’s make some changes:

1. Create a empty file called main_script.R

2. Open the file and modify with the following:

/data/iris_project/main_script.R

iris_data <- iris

Check the status again

We have made changes, let’s see if git is tracking them correctly

$ git status

1En la rama main

2No hay commits todavía

Archivos sin seguimiento:
3  (usa "git add <archivo>..." para incluirlo a lo que será confirmado)
    main_script.R

no hay nada agregado al commit pero hay archivos
4sin seguimiento presentes (usa "git add" para hacerles seguimiento)

1

Branch

2

Still no commits

3

Changes, in this case, a new file has appeared but is not yet tracked

4

Helpful info

Tracking and staging changes

Let’s see how files are tracked:

Files status lifecycle, obtained from the git book

Add files

We can start tracking files with git add (and check again with git status)

$ git add main_script.R

$ git status

1En la rama main

2No hay commits todavía

Cambios a ser confirmados:
3  (usa "git rm --cached <archivo>..." para sacar del área de stage)
    nuevos archivos: main_script.R

1

Branch

2

Still no commits

3

Helpful info

Commit changes

For now we know that a new file has been created and staged (is tracked), but no history of changes (commits) has started. For this, we need to commit the changes with git commit:

$ git commit -m 'created main_script.R file, loading data'

[main (commit-raíz) f484f68] created main_script.R file, loading data
 1 file changed, 1 insertion(+)
 create mode 100644 main_script.R

Commit changes

We can check the status again:

$ git status

1En la rama main
2nada para hacer commit, el árbol de trabajo está limpio

1

Branch

2

This has changed, now we have commits, but we don’t have anything new to track

And the cycle start again

Now we are going to modify our main script, to load some libraries and save it.

/data/iris_project/main_script.R

library(dplyr)
1library(ggplot2)

iris_data <- iris

1

These two lines are new

And the cycle continues

Check the new status.

$ git status

En la rama main
Cambios no rastreados para el commit:
  (usa "git add <archivo>..." para actualizar lo que será confirmado)
  (usa "git restore <archivo>..." para descartar los cambios en el directorio de trabajo)
    modificados:     main_script.R

sin cambios agregados al commit (usa "git add" y/o "git commit -a")

Checking the differences

We can see the differences the changes have made with git diff

$ git diff main_script.R

diff --git a/main_script.R b/main_script.R
index 65e98c9..27740cd 100644
--- a/main_script.R
+++ b/main_script.R
@@ -1 +1,4 @@
+library(dplyr)
+library(ggplot2)
+
 iris_data <- iris

And the cycle continues

Remember:

Files status lifecycle, obtained from the git book

And the cycle continues

So we need to add and commit the new changes, we can do this for modified files with a shortcut. Instead of git add FILE followed by git commit -m "MESSAGE" we can do it in one step with git commit -a -m "MESSAGE":

$ git commit -a -m 'Added needed libraries to main_script.R'

[main 8687941] Added needed libraries to main_script.R
 1 file changed, 3 insertions(+)

Check the timeline of changes

We can see at any time the timeline of changes with git log:

$ git log

commit 8687941e8cdf7819cf3c6c84b5c8a62ee7ab487e (HEAD -> main)
Author: MalditoBarbudo <victorgrandagarcia@gmail.com>
Date:   Wed Jan 31 14:54:08 2024 +0100

    Added needed libraries to main_script.R`

commit f484f687fb22990b1f3355b963814b0628748448
Author: MalditoBarbudo <victorgrandagarcia@gmail.com>
Date:   Wed Jan 31 14:14:56 2024 +0100

    created main_script.R file, loading data

Check the timeline of changes

We can simplify the log to see more clearly:

$ git log --oneline --decorate --all --graph

* 8687941 (HEAD -> main) Added needed libraries to main_script.R`
* f484f68 created main_script.R file, loading data

Summary of the workflow

%%{init: {"flowchart": {"htmlLabels": false}} }%%
flowchart LR
  Co(["`Commit`"]) --> Cr(["`Create
  or
  Modify`"])
  Cr --> A(["`Add`"]) --> Co

Clone an existing project

Let’s get a git project from the EMF GitHub to be able to work with it. For this we will use git clone:

$ git clone https://github.com/emf-creaf/meteospain.git

Clonando en 'meteospain'...
remote: Enumerating objects: 2823, done.
remote: Counting objects: 100% (625/625), done.
remote: Compressing objects: 100% (341/341), done.
remote: Total 2823 (delta 394), reused 488 (delta 275), pack-reused 2198
Recibiendo objetos: 100% (2823/2823), 23.68 MiB | 19.81 MiB/s, listo.
Resolviendo deltas: 100% (1829/1829), listo.

The cycle again

As we clone the latest available code in GitHub, the status is clean:

$ git status

En la rama main
Tu rama está actualizada con 'origin/main'.

nada para hacer commit, el árbol de trabajo está limpio

The cycle again

But the log is now a little more crowed than before:

$ git log --oneline --decorate --graph

* 63972c1 (HEAD -> main, origin/main, origin/HEAD) updated README
* 16e583f (tag: v0.1.4) updated cran-comments
* 920d094 updated version and news
* a9bdad1 (origin/devel) fixed global variables call
* 81e2448 Changes in RIA vignette
* aab768f changed style in util
* 184d687 correct version number in description and news, added Rubén F Casal to contributors
*   77a92e8 Merge pull request #21 from rubenfcasal/ria_coord
|\  
| * 53051a8 Fix bug in RIA coordinates (#20)
|/  
* e93dab4 updated readme with correct badge

....

First steps with GitHub

The first step is to create an account in the GitHub page.

Security

You probably have created a password to access your GitHub account, but for security reasons, that password can not be used when synchronizing your git projects with GitHub. For that we need to create a Personal Access Token (PAT) as our credentials.

Generating a PAT

To create a PAT, we need to go to GitHub tokens page and follow the instructions under Generate new token.

We are going to create a new folder in our project, called raw_data, and we are going to copy there the iris.csv file from the course materials.

Also, we are going to create an empty text file called .gitignore

.gitignore

After what we have done in the interlude, if we check the status of our project, we will see two new elements untracked:

$ git status

En la rama main
Archivos sin seguimiento:
  (usa "git add <archivo>..." para incluirlo a lo que será confirmado)
    .gitignore
    raw_data/

no hay nada agregado al commit pero hay archivos sin seguimiento presentes (usa "git add" para hacerles seguimiento)

.gitignore

Before anything else, we are going to modify .gitignore file to add the following:

{.git, filename="/data/iris_project/.gitignore"} raw_data

and check the status again

.gitignore

$ git status

En la rama main
Archivos sin seguimiento:
  (usa "git add <archivo>..." para incluirlo a lo que será confirmado)
    .gitignore

no hay nada agregado al commit pero hay archivos sin seguimiento presentes (usa "git add" para hacerles seguimiento)

.gitignore and sensible data

Use .gitignore to remove tracking from files and folders on your git project that is not use to have, are sensible (passwords, environmental variables files…) or just you don’t want them to be publicly available if using GitHub.

git and GitHub workflow

If you remember, the git workflow was like this:

%%{init: {"flowchart": {"htmlLabels": false}} }%%
flowchart LR
  Co(["`Commit`"]) --> Cr(["`Create
  or
  Modify`"])
  Cr --> A(["`Add`"]) --> Co

git and GitHub workflow

But now, we need to add GitHub to the workflow:

%%{init: {"flowchart": {"htmlLabels": false}} }%%
flowchart LR
  Cr(["`Create
  or
  Modify`"])
  A(["`Add`"])
  Co(["`Commit`"])
  Pus(["`Push
  **to**
  GitHub`"])
  Pul(["`Pull
  **from**
  GitHub`"])
  I(["`Init`"])
  
  Co --> Pus
  Co --> Cr
  Cr --> A(["`Add`"]) --> Co
  Pul --> Cr
  I --> Cr

So basically we need to push to GitHub (git push) and, in ocassions, we’ll need to pull from GitHub (git pull).

Create the remote repository

First we need to create a GitHub repository (project) to sync with our ´git` project. So we go there and create it.

We login to GitHub, and we click the big green New button. There we type the name of the repository (same as in our computer, iris_project) and a description and we click in the Create repository button.

Create the remote repository

A window will appear with instructions for pushing from our computer for the first time, the bit we need is:

…or push an existing repository from the command line
git remote add origin https://github.com/MalditoBarbudo/iris_project.git
git branch -M main
git push -u origin main

1

Add the remote origin to our local repo

2

Change the active branch to main

3

Push to GitHub, -u tells git is the first time pushing to GitHub.

More modifications to our project

As we have already our iris data in the raw_data folder, we can modify our main_script.R file:

/data/iris_project/main_script.R

library(dplyr)
library(ggplot2)
library(readr)

# reading data
iris_data <- read_csv('raw_data/iris.csv')

# subsetting data
setosa_subset <- iris_data |>
  filter(Species == "setosa")

And we repeat the cycle

$ git status

$ git commit -a -m 'Added readr and started to subset the data in main script'

But this time, we add a new step, for pushing to GitHub:

$ git push origin main

Enumerando objetos: 5, listo.
Contando objetos: 100% (5/5), listo.
Compresión delta usando hasta 16 hilos
Comprimiendo objetos: 100% (3/3), listo.
Escribiendo objetos: 100% (3/3), 344 bytes | 344.00 KiB/s, listo.
Total 3 (delta 1), reusados 0 (delta 0), pack-reusados 0
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
To https://github.com/MalditoBarbudo/iris_project.git
   d5b1b3a..59b52c3  main -> main

We can go to our GitHub repository to confirm is updated.

Reverting or resetting commits

Sometimes we mess things up. Is human nature. But with git we can always go back in time (commits) and restore the state of our files to an specific commit.
This can be done with git reset. There is two flavours of reset:

• git reset COMMIT_HASH_NUMBER will reset to the specified commit, but will maintain all changes in our workspace.

• git reset --hard COMMIT_HASH_NUMBER will reset to the specified commit, removing and deleting all changes since then.

Reverting or resetting commits

Reverting or resetting commits

First we look at the log to know the commit hash number to go back to:

$ git log --oneline --decorate --graph

* 59b52c3 (HEAD -> main, origin/main) added more subsetting
* d5b1b3a Added readr and started to subset the data in main script
* 6ce91f9 Added .gitignore file
* 8687941 Added needed libraries to main_script.R`
* f484f68 created main_script.R file, loading data

d5b1b3a is the target in my case (your hash will be different)

Reverting or resetting commits

We go back the hard way:

$ git reset --hard d5b1b3a

HEAD está ahora en d5b1b3a Added readr and started to subset the data in main script

And the log:

$ git log --oneline --decorate --graph

* d5b1b3a (HEAD -> main) Added readr and started to subset the data in main script
* 6ce91f9 Added .gitignore file
* 8687941 Added needed libraries to main_script.R`
* f484f68 created main_script.R file, loading data

Reverting or resetting commits

And, of course we need to push the reset to GitHub

$ git push origin main

Ups! something is not working:

To https://github.com/MalditoBarbudo/iris_project.git
 ! [rejected]        main -> main (non-fast-forward)
error: falló el empuje de algunas referencias a 'https://github.com/MalditoBarbudo/iris_project.git'
ayuda: Updates were rejected because the tip of your current branch is behind
ayuda: its remote counterpart. If you want to integrate the remote changes,
ayuda: use 'git pull' before pushing again.
ayuda: See the 'Note about fast-forwards' in 'git push --help' for details.

Reverting or resetting commits

We need to force the push to update the GitHub repository:

$ git push --force origin main

Total 0 (delta 0), reusados 0 (delta 0), pack-reusados 0
To https://github.com/MalditoBarbudo/iris_project.git
 + 59b52c3...d5b1b3a main -> main (forced update)

And… done!

Branches simplified

If you remember, at the beginning of the course we said that git works with snapshots (commits), storing only the relevant changes. So, right now, our git history looks like this:

%%{init: {"gitGraph": {"showBranches": true, "showCommitLabel": true}}}%%
gitGraph
   commit id: "commit 1"
   commit id: "commit 2"
   commit id: "commit 3"
   commit id: "commit 4"

Branches simplified

Working with snapshots means that we can branch out at any point (commit) to start a different path.
For example we had an idea for a very cool analysis, so we branch our code:

%%{init: {"gitGraph": {"showBranches": true, "showCommitLabel": true}}}%%
gitGraph
   commit id: "commit 1"
   commit id: "commit 2"
   commit id: "commit 3"
   commit id: "commit 4"
   branch "cool_analysis"
   checkout "cool_analysis"
   commit id: "cool analysis a"
   commit id: "cool analysis a fixed"
   commit id: "cool analysis a now for sure"

Branches simplified

But, at the same time, we need to prepare the poster to a congress meeting we go next week.
No time for finishing the cool analysis, but we have the boring normal one, so we continue with that one:

%%{init: {"gitGraph": {"showBranches": true, "showCommitLabel": true}}}%%
gitGraph
   commit id: "commit 1"
   commit id: "commit 2"
   commit id: "commit 3"
   commit id: "commit 4"
   branch "cool_analysis"
   checkout "cool_analysis"
   commit id: "cool analysis a"
   commit id: "cool analysis a fixed"
   commit id: "cool analysis a now for sure"
   checkout main
   commit id: "poster 1"
   commit id: "poster 2"

Branches simplified

As we said, working with snapshot allows to branch out, but also we are able to branch in, as we have all the information to merge the branches. This way we can have all the basic and the cool analysis stuff together:

%%{init: {"gitGraph": {"showBranches": true, "showCommitLabel": true}}}%%
gitGraph
   commit id: "commit 1"
   commit id: "commit 2"
   commit id: "commit 3"
   commit id: "commit 4"
   branch "cool_analysis"
   checkout "cool_analysis"
   commit id: "cool analysis a"
   commit id: "cool analysis a fixed"
   commit id: "cool analysis a now for sure"
   checkout main
   commit id: "poster 1"
   commit id: "poster 2"
   merge "cool_analysis"

Branches step by step

To work with branches, we will use git checkout. This command allows to create and change branches at any time.
First thin is to create a new branch, called cool_analysis:

$ git checkout -b cool_analysis

Cambiado a nueva rama 'cool_analysis'

Branches step by step

We can check the status and/or the log to see we are in a new branch:

$ git status

En la rama cool_analysis
nada para hacer commit, el árbol de trabajo está limpio

$ git log --oneline --decorate --graph --all

* d5b1b3a (HEAD -> cool_analysis, origin/main, main) Added readr and started to subset the data in main script
* 6ce91f9 Added .gitignore file
* 8687941 Added needed libraries to main_script.R`
* f484f68 created main_script.R file, loading data

Branches step by step

Now we made changes in our new branch, adding some code to main_script.R:

/data/iris_project/main_script.R

library(dplyr)
library(ggplot2)
library(readr)

# reading data
iris_data <- read_csv('raw_data/iris.csv')

# subsetting data
setosa_subset <- iris_data |>
  filter(Species == "setosa")
  
# cool analysis
model_setosa <- lm(Sepal Length ~ Petal.Length, data = setosa_subset)

Branches step by step

And we commit and push our new changes:

$ git commit -a -m 'started cool analysis with setosa'

1$ git push origin cool_analysis

Enumerando objetos: 5, listo.
Contando objetos: 100% (5/5), listo.
Compresión delta usando hasta 16 hilos
Comprimiendo objetos: 100% (3/3), listo.
Escribiendo objetos: 100% (3/3), 415 bytes | 415.00 KiB/s, listo.
Total 3 (delta 1), reusados 0 (delta 0), pack-reusados 0
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
remote: 
remote: Create a pull request for 'cool_analysis' on GitHub by visiting:
remote:      https://github.com/MalditoBarbudo/iris_project/pull/new/cool_analysis
remote: 
To https://github.com/MalditoBarbudo/iris_project.git
2 * [new branch]      cool_analysis -> cool_analysis

1

We push using the name of the new branch!

2

A new branch has also created in GitHub

Branches step by step

Let’s check the log now:

$ git log --oneline --decorate --graph --all

* 717d397 (HEAD -> cool_analysis, origin/cool_analysis) started cool analysis with setosa
* d5b1b3a (origin/main, main) Added readr and started to subset the data in main script
* 6ce91f9 Added .gitignore file
* 8687941 Added needed libraries to main_script.R`
* f484f68 created main_script.R file, loading data

Modifying more the cool_analysis branch

• modify main_script.R with:

/data/iris_project/main_script.R

library(dplyr)
library(ggplot2)
library(readr)

# reading data
iris_data <- read_csv('raw_data/iris.csv')

# subsetting data
setosa_subset <- iris_data |>
  filter(Species == "setosa")
  
# cool analysis
model_setosa <- lm(Sepal Length ~ Petal.Length, data = setosa_subset)

# plotting
setosa_plot <- ggplot(setosa_subset, aes(x = Petal.Length, y = Sepal.Length)) +
  geom_point(aes(size = Petal.Width)) +
  stat_smooth(method = "lm") +
  theme_bw()

Modifying more the cool_analysis branch

• Add, commit and push the new changes and take a look at the log

Branches step by step

We have our cool analysis finished. But we need to prepare our poster with the main branch, so we change to it:

$ git checkout main

Cambiado a rama 'main'
Tu rama está actualizada con 'origin/main'.

Branches step by step

We prepare our poster. For that, in the main branch, we create a new file called poster.Rmd, add it to git, commit the changes and push to main at GitHub:

$ git add poster.Rmd

$ git commit -a -m 'finished poster for international meeting'

$ git push origin main

Branches step by step

Let’s have a look at our log:

$ git log --oneline --decorate --graph --all

* 2da75fe (HEAD -> main, origin/main) finished poster for international meeting
| * 5ad9254 (origin/cool_analysis, cool_analysis) added plotting
| * 4f152cf added plotting of setosa
| * 717d397 started cool analysis with setosa
|/  
* d5b1b3a Added readr and started to subset the data in main script
* 6ce91f9 Added .gitignore file
* 8687941 Added needed libraries to main_script.R`
* f484f68 created main_script.R file, loading data

Branches step by step

Now that we are back of our congress meeting (with the best poster award, of course!), we can merge both branches, as we want to add the cool analysis to the main branch and continue from there. For that we will use git merge.
First, we make usre we are in the main branch, and then we merge:

$ git checkout main

$ git merge cool_analysis

Merge made by the 'ort' strategy.
 main_script.R | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

Branches step by step

And voilà, we have all together nicely, we can check with the log:

$ git log --oneline --decorate --all --graph

*   677910e (HEAD -> main) Merge branch 'cool_analysis'
|\  
| * 5ad9254 (origin/cool_analysis, cool_analysis) added plotting
| * 4f152cf added plotting of setosa
| * 717d397 started cool analysis with setosa
* | 2da75fe (origin/main) finished poster for international meeting
|/  
* d5b1b3a Added readr and started to subset the data in main script
* 6ce91f9 Added .gitignore file
* 8687941 Added needed libraries to main_script.R`
* f484f68 created main_script.R file, loading data

Preparation

We’ll work in pairs, OWNER and COLLABORATOR.

1. The OWNER will create a new local git repository called iris_git_course.

2. The OWNER will create a file called iris.R with the following contents:

library(dplyr)

iris_data <- iris

iris_setosa <- iris |>
  filter(Species == "setosa")

3. The OWNER will add and commit the changes.

4. The OWNER will create a GitHub repository and synchronize it with their local repo.

Working with other people - Step 1

Giving permissions

The OWNER of the repository in GitHub must give permissions to any collaborator for pulling and pushing to the iris_git_course repository. For that, we go to the Settings tab in the repository page.
There, in the Collaborators page, we click in Add people and look for the GitHub username of our COLLABORATOR.

An invitation is sent to the COLLABORATOR email and upon accepting we can configure the COLLABORATOR access.

Working with other people - Step 2

COLLABORATOR work

1. The COLLABORATOR will clone the repository locally (in their computer).

2. The COLLABORATOR will modify iris.R to add the following at the end of the script:

iris_versicolor <- iris |>
  filter(Species == "versicolor")

3. The COLLABORATOR will commit and push the changes to the repository

Working with other people - Step 3

Pulling the changes

Look at the situation now, the OWNER has work in GitHub that is ahead of their local repository. We need to synchronize both. To get the latest changes from a repository in GitHub we use git pull.

$ git pull origin main

Now, the OWNER have the latest changes.

Working with other people - Step 4

OWNER work

1. The OWNER will modify iris.R to add a plot step:

library(ggplot2)

ggplot(iris_setosa, aes(x = Petal.Length, y = Sepal.Length)) +
  geom_point() +
  geom_smooth(method = "lm")

2. The OWNER will commit and push the changes

Working with other people - Step 4

COLLABORATOR work

The last step is for the COLLABORATOR to pull the latest changes for the circle to be complete:

$ git pull origin main

%%{init: {"flowchart": {"htmlLabels": false}} }%%
flowchart LR
  Cr(["`Create
  or
  Modify`"])
  A(["`Add`"])
  Co(["`Commit`"])
  Pus(["`Push
  **to**
  GitHub`"])
  Pul(["`Pull
  **from**
  GitHub`"])
  I(["`Init`"])
  
  Co --> Pus
  Co --> Cr
  Cr --> A(["`Add`"]) --> Co
  Pul --> Cr
  I --> Cr