Jupyter et ses notebooks

# Jupyter et ses notebooks

2024

</div>
<div class="col" style="font-size: 1.2rem; text-align:right;">
	Pierre Poulain <br />
	pierre.poulain@u-paris.fr
</div>	
</div>

<div style="position: fixed; left: 0px; bottom: 20px; z-index: 31; font-size: 15px; line-height: 1; color: #c0c0c0;">
	<p>
		Except where otherwise noted, this content is licensed under 
		a Creative Commons Attribution-ShareAlike 4.0 International license 
		(CC BY-SA 4.0).
	</p>
</div>

---

## Objectifs d'apprentissage

- Décrire l'écosystème Jupyter.
- Créer un notebook Jupyter.
- Éditer des cellules en Markdown.
- Expliquer les limites d'usage d'un notebook.

---

## Notebook (en science)

- « Cahier électronique d'analyses » ou « [bloc-code](https://www.arthurperret.fr/du-notebook-au-bloc-code.html) »

</div>
<div class="fragment">

- Document :
	- code + résultats + graphiques
	- explications + images + formules mathématiques...

</div>
<div class="fragment" style="margin-top: 100px;">

Concept de *Literate programming* de Donald E. Knuth (1984) :

<cite>« Treat a program as literature understandable to human beings »</cite>

</div>

---

<div class="ref" style="text-align: left;">
	Source : <a href="https://www.wolfram.com/language/fast-introduction-for-math-students/en/notebook-documents/">Wolfram / Mathematica</a>
</div>

---

<div class="ref" style="text-align: left;">
	Source : <a href="https://rmarkdown.rstudio.com/authoring_quick_tour.html">R Markdown Quick Tour</a>
</div>

---

<div class="ref" style="margin-top: 10px;text-align: left;">
	Source : <a href="https://sinews.siam.org/Details-Page/jupyter-tools-for-the-life-cycle-of-a-computational-idea">Jupyter: Tools for the Life Cycle of a Computational Idea</a>,
	Siam News, 2018.
</div>

---

### Une idée d'hier, aujourd'hui techniquement possible...

<br />

#### The Scientific Paper Is Obsolete

</div>

<div class="ref">
	Source : <a href="https://www.theatlantic.com/science/archive/2018/04/the-scientific-paper-is-obsolete/556676/">The Atlantic</a>, 
	05/04/2018.
</div>

---

### Une idée d'hier, aujourd'hui techniquement possible...

<br />

*An open RNA-Seq data analysis pipeline tutorial with an example of reprocessing data from a recent Zika virus study* : 
- [papier](https://f1000research.com/articles/5-1574/v1), F1000 Research, Wang & Ma'ayan, **2016**.
- [notebook](http://nbviewer.jupyter.org/github/maayanlab/Zika-RNAseq-Pipeline/blob/master/Zika.ipynb)

---

### Des perspectives

<div class="fragment" style="margin-top:50px">
<a href="https://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261">Interactive notebooks: Sharing the code</a>,
Shen, Nature, 2014.
	<div style="font-style:italic;font-size:2rem;">
	"We used to speak two different languages. I would talk about the biology and she would talk about coding. 
	Now we have common ground; we can <strong>communicate to each other better</strong>. 
	This accelerates our research." she says.
	</div>
</div>

<br />

<div class="fragment" style="margin-top:50px">
<a href="https://www.nature.com/articles/d41586-018-07196-1">By Jupyter it all makes sense</a>, Perkel, Nature, 2018.
	<div style="font-style:italic;font-size:2rem;">
	Thanks to an <strong>improved interface</strong> and tools for <strong>running in the cloud</strong>, 
	take-up of the computational notebook is rocketing
	</div>
</div>
</div>

---

## Projet Jupyter

<div style="color:#b9b9b9;">
2001 - F Pérez (iPython) 
</div>

2014 - F Pérez & B E Granger
</div>

---

## Projet Jupyter

---

## Jupyter notebook

<br />

- « cahier électronique d'analyses »
- dans un navigateur web 
- Julia, Python, R, C++...  
- code + résultats + graphiques + explications + formules...
- export : py, R, md, html, pdf

<div class="fragment" style="margin-top: 150px; font-size: 1.5rem;">
	Une intro : 
	<a href="https://sinews.siam.org/Details-Page/jupyter-tools-for-the-life-cycle-of-a-computational-idea">Jupyter: Tools for the Life Cycle of a Computational Idea</a>,
	Siam News, 2018.
</div>

---

<div class="ref" style="margin-top: 10px;text-align: left;">
	Source : <a href="https://speakerdeck.com/lheagy/jupyter-meets-the-earth-from-geophysical-inversions-to-open-collaborative-geoscience">Jupyter meets the Earth: from geophysical inversions to open, collaborative geoscience</a>,
	Lindsey Heagy, 2019.
</div>

---

<div class="ref" style="margin-top: 10px;text-align: left;">
	Source : <a href="https://ieeexplore.ieee.org/document/8816763/">A Large-Scale Study About Quality and Reproducibility of Jupyter Notebooks</a>,
	Pimentel et al., 2019.
</div>

---

## Jupyter Notebooks

<div class="fragment">
 +9,7 millions notebooks sur GitHub
 <div class="ref">
	Source : <a href="http://nbviewer.jupyter.org/github/parente/nbestimate/blob/master/estimate.ipynb">Estimate of Public Jupyter Notebooks on GitHub</a>
	(01/12/2020)
</div>
</div>

<div class="fragment" style="margin-top:50px">
Utilisés en 1re année à Berkeley (1800 étudiants) <br />

<cite>
	"There would have been too many obstacles for students to easily gain access to the tools and data that they need."
</cite>

<div class="ref">
Source : <a href="https://data.berkeley.edu/news/coursefuture">The course of the future – and the technology behind it</a>
</div>
</div>

<div class="fragment" style="margin-top:50px">
Utilisés chez Google, Microsoft, IBM, Bloomberg, Nasa, SoundCloud, NetFlix, La banque mondiale <a href="https://jupyter.org/">...</a>
</div>
		
<div class="fragment" style="margin-top:50px">
<a href="https://qz.com/1417145/economics-nobel-laureate-paul-romer-is-a-python-programming-convert/">This year’s Nobel Prize in economics was awarded to a Python convert</a>
<div style="font-style:italic;font-size:2rem;">
"Instead of using Mathematica, Romer discovered that he could use a Jupyter notebook for sharing his research." (2018)
</div>
</div>

---

## Exemples

<br />

[A gallery of interesting Jupyter Notebooks](https://github.com/jupyter/jupyter/wiki/A-gallery-of-interesting-Jupyter-Notebooks)
<br />
<br />

[Bioinformatics with Python Cookbook 2nd Ed](https://github.com/PacktPublishing/Bioinformatics-with-Python-Cookbook-Second-Edition)

- [FastQ files](https://github.com/PacktPublishing/Bioinformatics-with-Python-Cookbook-Second-Edition/blob/master/Chapter02/Working_with_FASTQ.ipynb)
- [PDB files](https://github.com/PacktPublishing/Bioinformatics-with-Python-Cookbook-Second-Edition/blob/master/Chapter07/PDB.ipynb)

Génération de notebooks : [BioJupies](https://amp.pharm.mssm.edu/biojupies/) / GEO : GSE116087

---

## L'écosystème Jupyter

<div style="text-align: center;">
	Environnement « classique »
	<img src="assets/img/jupyter-classique.png"
	height="400px">
</div>
<br />

---

## L'écosystème Jupyter : Jupyter **lab**

<div style="text-align: center;">
	Environnement de développement et d'analyse
	<img src="assets/img/jupyterlab.png"
	height="400px">
<div class="ref" style="margin-top:-30px;">
	Source : <a href="https://blog.jupyter.org/jupyterlab-is-ready-for-users-5a6f039b8906">Project Jupyter</a>, CC BY-SA.
</div>
</div>
<br />
<div style="font-size:1.2rem;">
vidéo : <a href="https://www.youtube.com/watch?v=w7jq4XgwLJQ">JupyterLab: The Next-Generation Jupyter Frontend</a>, 2017.
<br />
article : <a href="https://blog.jupyter.org/jupyterlab-is-ready-for-users-5a6f039b8906">JupyterLab is Ready for Users</a>, 2018.
</div>

</div>

---

## Installation

Depuis un environnement conda :
```bash
$ conda install -c conda-forge jupyterlab
```

Lancement :
```bash
$ jupyter-lab
```

---

## Démo

[Try Jupyter](https://jupyter.org/try)

---

## Exercice 1

Notebook `intro_Python.ipynb` / `intro_R.ipynb`

<https://pierrepo.github.io/intro-jupyter/>

---

## Markdown

<br />

- Langage de balises *simples*
- Utilisé pour la partie « littérature » des notebooks

---

## Markdown : exemples

#### Google Docs, Microsoft Word, OpenOffice Writter (WYSIWYG)

Hello my name is **Pierre**.<br />
Git is *fun* and [GitHub](https://github.com/) even more!

#### LaTeX
```tex
Hello my name is \textbf{Pierre}. 
Git is \textit{fun} and \href{https://github.com/}{GitHub} even more!
```

</div>

#### HTML
```html
Hello my name is <b>Pierre</b>. 
Git is <i>fun</i> and <a href=”https://github.com/”>GitHub</a> even more!
```
	
</div>

#### Markdown
```md
Hello my name is **Pierre**. 
Git is *fun* and [GitHub](https://github.com/) even more.
```
		
</div>

---

## Markdown

Références :

- https://www.markdownguide.org/basic-syntax/
- https://www.markdownguide.org/extended-syntax/ 
- https://guides.github.com/features/mastering-markdown/
- [Markdown Syntax cheat sheet](https://guides.github.com/pdfs/markdown-cheatsheet-online.pdf)
- [The Ultimate Markdown Guide (for Jupyter Notebook)](https://medium.com/analytics-vidhya/the-ultimate-markdown-guide-for-jupyter-notebook-d5e5abf728fd)

<br />

Pour faire des tableaux :
- [Markdown Table Generator](https://www.tablesgenerator.com/markdown_tables)

</div>
---

## Markdown

Utilisation hors notebook :
- Éditeur collaboratif en Markdown : [HedgeDoc](https://demo.hedgedoc.org/)
- Fichier [README.md](https://github.com/pierrepo/autoclasswrapper) sur GitHub [✏️](https://github.com/pierrepo/autoclasswrapper/edit/master/README.md)
- Ce [cours](https://github.com/pierrepo/intro-jupyter/) 🤪
- [MyST](https://myst-nb.readthedocs.io/en/latest/) : Markdown sous stéroides pour écrire de la science.

---

---

## Exercice 2

Notebook `markdown.ipynb`

<https://pierrepo.github.io/intro-jupyter/>

---

## Exercice 3

Notebook `matplotlib.ipynb`

<https://pierrepo.github.io/intro-jupyter/>

---

## Exercice 4

Notebook `intro_Python.ipynb` / `intro_R.ipynb`

<https://pierrepo.github.io/intro-jupyter/>

---

## Conclusion

Notebooks Jupyter :

- Polyvalents (Python, R, Bash...)
- Utiles pour l'**analyse exploratoire** et le **reporting** ([exemple](https://github.com/MDverse/mdda/blob/main/notebooks/paper_stats_and_figures.ipynb))
- Excellente interface d'**échange** et de **communication** entre biologiste / bioinformaticien

On dit ce qu'on fait, pourquoi on le fait et ce qu'on obtient.<br /> *Plus de cellules Markdown que de cellules de code*.

On accompagne, on explique des résultats ou des analyses.

</div>

⚠️ Attention ⚠️

On ne développe pas un gros code Python dans un notebook.

</div>

---

## Pour aller plus loin...

Ten simple rules for writing and sharing computational analyses in Jupyter Notebooks<br />
Rule et al., *PLOS Comput. Biol.*, 2019.<br />
DOI [10.1371/journal.pcbi.1007007]( https://doi.org/10.1371/journal.pcbi.1007007)

Using interactive digital notebooks for bioscience and informatics education<br />
Davies et al., *PLOS Comput. Biol.*, 2020.<br />
DOI [10.1371/journal.pcbi.1008326](https://doi.org/10.1371/journal.pcbi.1008326)

Jupyter: Thinking and Storytelling With Code and Data<br />
Granger et Perez, *Computing in Science & Engineering*, 2021.<br />
DOI [10.1109/MCSE.2021.3059263](https://doi.org/10.1109/MCSE.2021.3059263)

Computational reproducibility of Jupyter notebooks from biomedical publications<br />
Samuel et Mietchen, *arXiv*, 2023.<br />
DOI [10.48550/arXiv.2308.07333](https://doi.org/10.48550/arXiv.2308.07333)<br />
(22 578 notebooks Jupyter Python -> 879 complètement reproductibles)

[JupyterCon](https://jupytercon.com/) 2023 : 10-12 mai ([videos](https://www.youtube.com/playlist?list=PL_1BH3ug7n1Ih_Yy2TmM7MZ2zogSLZvzE))<br />
[Fernando Pérez: Two decades of IPython and Jupyter](https://www.youtube.com/watch?v=mqeibIiLeUA) (26').

</div>

---

## Pour aller vers plus de reproductibilité

Using Jupyter for reproducible scientific workflows<br />
Beg et al., *Computing in Science & Engineering*, 2021.<br />
DOI [10.1109/MCSE.2021.3052101](https://doi.org/10.1109/MCSE.2021.3052101)

</div>

<div class="ref" style="text-align: left;margin-top:-20px;">
	Source : <a href="https://zenodo.org/record/4421040">Juliette Taka et Nicolas M. Thiéry, Publishing reproducible logbooks explainer comic strip, CC BY-SA</a>
</div>

</div>

---

## Pour aller vers plus de reproductibilité

Quelques exemples de dépôts GitHub configurés avec Binder :

- [Ce cours](https://github.com/pierrepo/intro-jupyter)
- [Python omics use cases](https://github.com/pierrepo/python-omics-use-cases)

---