[{"url":"cheatsheets/","title":"Resources","description":"","tags":["welcome"],"text":"ResourcesModern Julia WorkflowsJuliaLang DiscourseJuliaLang DocumentationJulia CommunityCheatsheetsFastrack to Julia cheatsheet.MATLAB-Julia-Python comparative cheatsheet by QuantEcon groupPlots.jl cheatsheet"},{"url":".","title":"index","description":"","tags":["homepage"],"text":""},{"url":"installation/","title":"Software installation","description":"","tags":["welcome"],"text":"First-time setup: Install Julia & PlutoText and pictures version:Step 1: Install Julia 1.12.6Go to https://julialang.org/install/ and download juliaup, juliaup will install the current stable release, Julia 1.12.6, using the correct version for your operating system (Linux x86, Mac, Windows, etc).Step 2: Run JuliaAfter installing, make sure that you can run Julia. On some systems, this means searching for the “Julia 1.12.6” program installed on your computer; in others, it means running the command julia in a terminal. Make sure that you can execute 1 + 1:Make sure that you are able to launch Julia and calculate 1+1 before proceeding!Step 3: Install PlutoNext we will install the Pluto, the notebook environment that we will be using during the course. Pluto is a Julia programming environment designed for interactivity and quick experiments.Open the Julia REPL. This is the command-line interface to Julia, similar to the previous screenshot.Here you type Julia commands, and when you press ENTER, it runs, and you see the result.To install Pluto, we want to run a package manager command. To switch from Julia mode to Pkg mode, type ] (closing square bracket) at the julia> prompt:\njulia> ]\n\n(@v1.12) pkg>\nThe line turns blue and the prompt changes to pkg>, telling you that you are now in package manager mode. This mode allows you to do operations on packages (also called libraries).To install Pluto, run the following (case sensitive) command to add (install) the package to your system by downloading it from the internet.\nYou should only need to do this once for each installation of Julia:\n(@v1.12) pkg> add Pluto\nThis might take a couple of minutes, so you can go get yourself a cup of tea!You can now close the terminal.Step 4: Use a modern browser: Mozilla Firefox or Google ChromeWe need a modern browser to view Pluto notebooks with. Firefox and Chrome work best.Second time: Running Pluto & opening a notebookRepeat the following steps whenever you want to work on a project or homework assignment.Step 1: Start PlutoStart the Julia REPL, like you did during the setup. In the REPL, type:julia> using Pluto\n\njulia> Pluto.run()\nThe terminal tells us to go to http://localhost:1234/ (or a similar URL). Let’s open Firefox or Chrome and type that into the address bar.If you’re curious about what a Pluto notebook looks like, have a look at the Featured Notebooks. These notebooks are useful for learning some basics of Julia programming.If you want to hear the story behind Pluto, have a look a the JuliaCon presentation.If nothing happens in the browser the first time, close Julia and try again. And please let us know!Step 2a: Opening a notebook from the webThis is the main menu - here you can create new notebooks, or open existing ones. Our homework assignments will always be based on a template notebook, available in this GitHub repository. To start from a template notebook on the web, you can paste the URL into the blue box and press ENTER.For example, homework 0 is available here. Go to this page, and on the top right, click on the button that says “Edit or run this notebook”. From these instructions, copy the notebook link, and paste it into the box. Press ENTER, and select OK in the confirmation box.The first thing we will want to do is to save the notebook somewhere on our own computer; see below.Step 2b: Opening an existing notebook fileWhen you launch Pluto for the second time, your recent notebooks will appear in the main menu. You can click on them to continue where you left off.If you want to run a local notebook file that you have not opened before, then you need to enter its full path into the blue box in the main menu. More on finding full paths in step 3.Step 3: Saving a notebookWe first need a folder to save our homework in. Open your file explorer and create one.Next, we need to know the absolute path of that folder. Here’s how you do that in Windows, MacOS and Ubuntu.For example, you might have:C:\\Users\\fons\\Documents\\18S191_assignments\\ on Windows/Users/fons/Documents/18S191_assignments/ on MacOS/home/fons/Documents/18S191_assignments/ on UbuntuNow that we know the absolute path, go back to your Pluto notebook, and at the top of the page, click on “Save notebook…”.This is where you type the new path+filename for your notebook:Click Choose.Step 4: Sharing a notebookAfter working on your notebook (your code is autosaved when you run it), you will find your notebook file in the folder we created in step 3. This the file that you can share with others, or submit as your homework assignment to Canvas.\nconst run = f => f();\nrun(async () => {\nconst versions = await (await fetch(`https://julialang-s3.julialang.org/bin/versions.json`)).json()\nconst sortby = v => v.split(\"-\")[0].split(\".\").map(parseFloat).reduce((a,b) => a*10000 + b)\nconst version_names = Object.keys(versions).sort((a,b) => sortby(a) - sortby(b)).reverse()\nconst stable = version_names.find(v => versions[v].stable)\nconsole.log({stable})\nconst pkg_stable = /\\d+\\.\\d+/.exec(stable)[0]\ndocument.querySelectorAll(\"auto-julia-version\").forEach(el => {\n    console.log(el)\n    el.innerText = el.getAttribute(\"short\") == null ? stable : pkg_stable\n})\n});"},{"url":"logistics/","title":"Class logistics","description":"","tags":["welcome"],"text":"main a img {\n    width: 5rem;\n    margin: 1rem;\n}\nCourse logisticsLecture datesStart: 15th of AprilEnd: 8th of JulyLecture on the 3rd of June will be moved.ProjectsChoose project by 2026-05-13Discuss idea first with me.Half-page project proposal.Project presentations: 2026-07-08Project should be relevant both to your work and topics discussed in class.Demonstrate best practices for research software development.\nTestingReproducibilityDocumentationAccessibilityDevelop “in the open” on Github.GradesIf you need a grade for this class, in addition to the project presentation, you will need to submit a short project report (2 pages) and your repository with instruction on how to reproduce your results."},{"url":"search/","title":"Search results","description":"","tags":[],"text":"window.init_search();SearchResults\nLoading..."},{"url":"assets/scripts/get_highlights/","title":"get_highlights","description":"","tags":[],"text":"if isempty get metadata \"homepage\" , \"highlights\", nothing else highlights htl \"\"\" section div class \"content\" h2 x \"name\" h2 p x \"text\" p div div class \"preview\" img src \" x \"img\" \" div section \"\"\" for x in metadata \"homepage\" \"highlights\" htl \"\"\" div class \"subjectscontainer wide\" h1 Highlights h1 div class \"contain\" highlights div div \"\"\" end"},{"url":"assets/scripts/get_schedule/","title":"get_schedule","description":"","tags":[],"text":"import Dates let today Dates.today sections metadata \"sidebar\" section htmls map sections do section id, map collections section id .pages do other page output other page.output name get output.frontmatter, \"title\", basename other page.input.relative path desc get output.frontmatter, \"description\", nothing tags get output.frontmatter, \"tags\", String date str get output.frontmatter, \"date\", nothing date str nothing && return nothing upcoming try Dates.Date string date str today catch false end class \"no decoration\", upcoming ? \"upcoming entry\" nothing, \"tag replace x, \" \" \" \" \" for x in tags ..., NOTE keep each entry on a single line with no blank whitespace only lines. This output is interpolated into a Markdown `.jlmd` file, and the CommonMark processor ends a raw HTML block on the first blank line, which would otherwise wrap the remaining cards in ` p ` and mis nest the anchors titles get hoisted out of their cards . htl \"\"\" a title desc class class href root url \" \" other page.url h3 name h3 span class \"schedule date upcoming ? \"upcoming badge\" \"\" \" date str span upcoming ? htl \"\"\" span class \"upcoming label\" Upcoming span \"\"\" nothing a \"\"\" end end isempty section htmls ? nothing htl \"\"\" div class \"wide subjectscontainer\" h1 Schedule h1 div class \"subjects\" section htmls div div \"\"\" end "},{"url":"assets/scripts/get_subjects/","title":"get_subjects","description":"","tags":[],"text":" NOTE if you re enable this in ` includes welcome.jlmd`, collapse each card to a single line with no blank whitespace only lines first — see the comment in `get schedule.jl`. The multi line ` htl` cards below break when interpolated into Markdown CommonMark ends the raw HTML block on the first blank line . let sections metadata \"sidebar\" sections htl \"\"\" let input other page.input output other page.output name get output.frontmatter, \"title\", basename input.relative path desc get output.frontmatter, \"description\", nothing tags get output.frontmatter, \"tags\", String image get output.frontmatter, \"image\", nothing class \"no decoration\", \"tag replace x, \" \" \" \" \" for x in tags ..., image nothing || isempty image ? nothing htl \"\"\" a title desc class class href root url \" \" other page.url h3 name h3 img src image a \"\"\" end for other page in collections section id .pages \"\"\" for section id, section name in sections isempty sections ? nothing htl \"\"\" div class \"wide subjectscontainer\" h1 Subjects h1 div class \"subjects\" sections div div \"\"\" end"},{"url":"exercises/exercise_10_gradient_descent/","title":"Gradient descent","description":"Implement and extend a gradient-descent training loop using automatic differentiation","tags":["module1","track_ad","exercises"],"text":" A Pluto.jl notebook v1.0.1 frontmatter order \"4.3\" exercise number \"10\" title \"Gradient descent\" tags \"module1\", \"track ad\", \"exercises\" layout \"layout.jlhtml\" license \"MIT\" description \"Implement and extend a gradient descent training loop using automatic differentiation\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils begin using PlutoTeachingTools, PlutoUI PlutoUI.TableOfContents depth 4 end ChooseDisplayMode begin using CairoMakie set theme theme latexfonts fontsize 16, Lines linewidth 2, , markersize 16 end md\"\"\" Simple gradient descent Implementing proximal gradient descent for minimization of a squared error loss function Inspiration https discourse.julialang.org t types and gradients including forward gradient 946 \"\"\" using ForwardDiff gradient, derivative import Enzyme begin using LinearAlgebra using Random end model linear regression w, b, x w x . b loss function mean squared error ŷ, y sum abs2, ŷ . y size y,2 get gradient w.r.t to `w` loss∇w model, loss, w, b, x, y gradient w loss model w, b, x , y , w get derivative w.r.t to `b` `derivative` is used instead of `gradient` because `b` is a scalar instead of an array lossdb model, loss, w, b, x, y derivative b loss model w, b, x , y , b optimization algorithm function proximal gradient descent model, loss, w, b, x, y lr .1 w lmul lr, loss∇w model, loss, w, b, x, y b lr lossdb model, loss, w, b, x, y return w, b end function main T Array, n 100000 Random.seed 0 p 25 x randn n,p ' y sum x 1 5, dims 1 . randn n ' 0.1 w 0.0001 randn 1,p b 0.0 x T x y T y w T w model linear regression loss mean squared error errs loss model w,b,x ,y time for i 1 p w, b proximal gradient descent model, loss, w, b, x, y push errs, loss model w,b,x ,y end errs end errs main Array lines errs md\"\"\" Part 1 — Custom model and loss \"\"\" question box md\"\"\" The demo above uses a linear model and mean squared error loss. 1. Define your own model `my model w, b, x ` e.g. add a non linearity like `tanh` . 2. Define your own loss `my loss ŷ, y ` e.g. mean absolute error . 3. Pass them to `proximal gradient descent` and verify the loss decreases. \"\"\" TODO Define my model and my loss let if isdefined my model func not defined my model elseif isdefined my loss func not defined my loss else try w randn 1, 3 b 0.0 x randn 3, 5 y randn 1, 5 w2, b2 proximal gradient descent my model, my loss, w, b, x, y if my loss my model w2, b2, x , y my loss my model w, b, x , y 1e 6 correct else keep working md\"One step of gradient descent should not increase the loss.\" end catch e keep working md\"Got an error sprint showerror, e \" end end end answer box hint md\"\"\" ```julia my model w, b, x tanh. w x . b my loss ŷ, y sum abs, ŷ . y size y, 2 ``` \"\"\" md\"\"\" Part 2 — Element type genericity \"\"\" md\"\"\" In machine learning `Float32` is typically preferred over `Float64` it halves memory usage and is faster on both CPUs and GPUs where most real training happens . The `main` function accepts an array constructor `T` so we can swap array types, but there is a hidden type inconsistency that prevents it from being truly generic. \"\"\" question box md\"\"\" 1. Run `main x Float32. x `. What is the element type of `b` at the end of training? 2. Find the line in `main` that forces `b` to always be `Float64`, regardless of `T`. 3. Write a corrected version `main f32` where `b` has the same element type as `w` after the arrays are converted — use `zero eltype w ` to construct the initial bias. 4. Verify that the final loss from `main f32 ` is close to the one from `main `. \"\"\" TODO Implement main f32 let if isdefined main f32 func not defined main f32 else try errs32 main f32 errs64 main Array if eltype errs32 Float32 keep working md\"The errors should be `Float32`. Check that `b` is initialised with `zero eltype w `.\" elseif last errs32 ≈ last errs64 atol 0.05 keep working md\"The final loss of `main f32 ` should be close to `main `. Got last errs32 vs last errs64 .\" else correct end catch e keep working md\"Got an error sprint showerror, e \" end end end answer box hint md\"\"\" Change `b 0.0` to `b zero eltype w ` after converting `w T w ` ```julia function main f32 T x Float32. x , n 100000 Random.seed 0 p 25 x randn n, p ' y sum x 1 5, dims 1 . randn n ' 0.1 w 0.0001 randn 1, p x T x y T y w T w b zero eltype w tracks the element type of w model linear regression loss mean squared error errs loss model w, b, x , y for i in 1 p w, b proximal gradient descent model, loss, w, b, x, y push errs, loss model w, b, x , y end errs end ``` \"\"\" md\"\"\" Part 3 — Different AD framework \"\"\" tip md\"\"\" Enzyme.jl API primer Exercise 10 showed `Enzyme.autodiff Enzyme.Forward, ... ` for scalar forward mode derivatives. Here we need gradients — derivatives of a scalar valued function w.r.t. a vector argument — and we want to use reverse mode because it needs only one backward pass regardless of the number of inputs compare forward mode needs one pass per parameter . ```julia import Enzyme gradient of f Rⁿ → R w.r.t. its vector argument x g Enzyme.gradient Enzyme.Reverse, f, x returns a vector the same shape as x for a scalar argument use autodiff directly gradient requires an array autodiff handles scalars db Enzyme.autodiff Enzyme.Reverse, f, Enzyme.Active b 1 1 ``` The key pieces `Enzyme.Reverse` — use reverse mode backpropagation . `Enzyme.gradient mode, f, x ` — high level helper returns `∇f x ` as an array. `Enzyme.Active b ` — marks a scalar as an active differentiated input when calling `autodiff` directly the derivative is returned in the first element of the first tuple. \"\"\" question box md\"\"\" `ForwardDiff.gradient` used above is forward mode AD it needs one forward pass per parameter, so cost scales with `p 25`. Reverse mode AD backpropagation computes all parameter gradients in a single backward pass. 1. Using the primer above, implement `loss∇w enzyme model, loss, w, b, x, y ` with `Enzyme.gradient Enzyme.Reverse, ... `. 2. Implement `lossdb enzyme model, loss, w, b, x, y ` for the scalar bias using `Enzyme.autodiff Enzyme.Reverse, ..., Enzyme.Active b `. 3. Verify that both return the same values as `loss∇w` and `lossdb`. \"\"\" TODO Implement loss∇w enzyme and lossdb enzyme answer box hint md\"\"\" ```julia import Enzyme loss∇w enzyme model, loss, w, b, x, y Enzyme.gradient Enzyme.Reverse, w loss model w, b, x , y , w lossdb enzyme model, loss, w, b, x, y Enzyme.gradient Enzyme.Reverse, b loss model w, b, x , y , b | only ``` \"\"\" "},{"url":"exercises/exercise_11_type_stability/","title":"Type Stability","description":"Learn how type stability affects Julia performance and how to diagnose and fix type-unstable code","tags":["module1","track_performance","exercises"],"text":" A Pluto.jl notebook v1.0.1 frontmatter order \"5.1\" exercise number \"11\" title \"Type Stability\" tags \"module1\", \"track performance\", \"exercises\" layout \"layout.jlhtml\" license \"MIT\" description \"Learn how type stability affects Julia performance and how to diagnose and fix type unstable code\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils using PlutoTeachingTools, PlutoUI using BenchmarkTools ChooseDisplayMode PlutoUI.TableOfContents depth 4 md\"\"\" Exercise Type Stability A function is type stable if the compiler can determine the return type and the types of all local variables from the types of the inputs alone, without running the code. Type unstable functions force the compiler to emit slower, boxed code that handles multiple possible types at runtime. \"\"\" md\"\"\" Motivating example `baz` returns different types depending on the branch taken `bar` always returns `Float64`. Run the benchmarks and inspect `Base.return types` to see the difference. \"\"\" function baz s rand if s 2 3 return 0.666667 Float64 elseif s 1 3 return 1 3 Rational Int64 else return 0 Int64 end end function bar s rand if s 2 3 return 0.666667 elseif s 1 3 return 0.3333333 else return 0.0 end end benchmark baz benchmark bar Base.return types baz , Base.return types bar md\"\"\" Part 1 — Fixing `my sum` The function below is type unstable because the accumulator `output` starts as an `Int` but the array elements are `Float64`. ```julia function my sum A output 0 Int for x in A output x end return output end ``` 1. Copy `my sum` into a cell and run ` code warntype my sum rand 10 `. Variables highlighted in red yellow are type unstable. 2. Write a fixed version called `my sum2` using `zero eltype A ` to initialise the accumulator. 3. Benchmark both with ` benchmark` on `rand 10^6 `. \"\"\" Paste my sum here, then write my sum2 below it let if isdefined my sum2 func not defined my sum2 else A rand 100 result my sum2 A if result isa Float64 keep working md\"`my sum2` should return a `Float64` for a `Float64` array, but got a ` typeof result ` — check your accumulator initialisation.\" elseif result ≈ sum A keep working md\"`my sum2` gives the wrong result — it should equal `sum A `.\" else correct end end end answer box hint md\"\"\" ```julia function my sum2 A output zero eltype A Float64 for a Float64 array for x in A output x end return output end ``` ` code warntype my sum rand 10 ` highlights `output` in red because the compiler infers `Union Float64, Int64 `. After the fix the inferred type is just `Float64`. \"\"\" md\"\"\" Part 2 — Newton's square root Make the following function type stable. The bug `output` is initialised to `1` an `Int` regardless of the type of `x`. ```julia function my sqrt x output 1 for i in 1 1000 output 0.5 output x output end output end ``` After fixing it, verify that `my sqrt 2.0 ≈ sqrt 2.0 ` and that `my sqrt Float32 2 isa Float32`. \"\"\" Write your type stable my sqrt here let if isdefined my sqrt func not defined my sqrt else try if my sqrt 2.0 ≈ sqrt 2.0 keep working md\"`my sqrt 2.0 ` should be approximately `sqrt 2.0 `.\" else r32 my sqrt Float32 2 if r32 isa Float32 keep working md\"`my sqrt Float32 2 ` should return a `Float32`, not a ` typeof r32 `.\" else correct end end catch e keep working md\"Your function threw an error ` sprint showerror, e `\" end end end answer box hint md\"\"\" ```julia function my sqrt x output one x same type as x for i in 1 1000 output output x output 2 end output end ``` `one x ` returns the multiplicative identity with the same type as `x`, keeping the accumulator type stable throughout the loop. \"\"\" "},{"url":"exercises/exercise_12_performance_annotations/","title":"Performance Annotations","description":"Use @inbounds, @simd and @fastmath to accelerate inner loops and understand the trade-offs","tags":["module1","track_performance","exercises"],"text":" A Pluto.jl notebook v1.0.1 frontmatter order \"5.2\" exercise number \"12\" title \"Performance Annotations\" tags \"module1\", \"track performance\", \"exercises\" layout \"layout.jlhtml\" license \"MIT\" description \"Use inbounds, simd and fastmath to accelerate inner loops and understand the trade offs\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils "},{"url":"exercises/exercise_13_memory_layout/","title":"Memory Layout & Cache Effects","description":"Understand how column-major storage affects performance, and write cache-friendly matrix loops","tags":["module1","track_performance","exercises"],"text":" A Pluto.jl notebook v1.0.1 frontmatter order \"5.3\" exercise number \"13\" title \"Memory Layout & Cache Effects\" tags \"module1\", \"track performance\", \"exercises\" layout \"layout.jlhtml\" license \"MIT\" description \"Understand how column major storage affects performance, and write cache friendly matrix loops\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils "},{"url":"exercises/exercise_14_github/","title":"Github","description":"sample exercise","tags":["module2","track_principles","exercises"],"text":" A Pluto.jl notebook v1.0.1 frontmatter order \"5.1\" exercise number \"14\" title \"Github\" tags \"module2\", \"track principles\", \"exercises\" license \"MIT\" layout \"layout.jlhtml\" description \"sample exercise\" frontmatter.author name \"Valentin Churavy\" image \"https avatars.githubusercontent.com u 145258?v 4\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils begin using PlutoTeachingTools, PlutoUI using PlutoUI Slider PlutoUI.TableOfContents depth 4 end md\"\"\" Exercise Github \"\"\" md\"\"\" Learn Git Branching Learn Git Branching https learngitbranching.js.org is a niffty little website to explore Git concepts. If you want you may spend a few minutes exploring it. \"\"\" md\"\"\" Getting started on Github 1. Create a Github account 2. On Windows I have few personal experiences, but I suspect you might want to stick to Github Desktop or the VSCode integration 3. On Mac Linux Optional Setup a SSH key https docs.github.com en authentication connecting to github with ssh generating a new ssh key and adding it to the ssh agent \"\"\" md\"\"\" Setup a repository for your Project Minimum features README.md LICENSE.md .gitignore Most likely Basic Julia project structure `src ` Project.toml Julia focused CI infrastructure Tests Documentation Benchmarking \"\"\" md\"\"\" Contribute to another Github repository Grab a friend neighbor and open a pull request to their repository \"\"\" "},{"url":"exercises/exercise_15_debugging/","title":"Debugging","description":"sample exercise","tags":["module2","track_principles","exercises"],"text":" A Pluto.jl notebook v1.0.1 frontmatter order \"7.1\" exercise number \"15\" title \"Debugging\" tags \"module2\", \"track principles\", \"exercises\" license \"MIT\" layout \"layout.jlhtml\" description \"sample exercise\" frontmatter.author name \"Valentin Churavy\" image \"https avatars.githubusercontent.com u 145258?v 4\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils import ForwardDiff using PlutoTeachingTools md\"\"\" Exercise Debugging directional derivatives \"\"\" function jvp F, u, v, ϵ sqrt eps F u ϵ . v F u . ϵ end function jacobian F, u w F u v similar u J similar u, length u , length w for i in length u v i 1 J , i jvp F, u, v end J end F1 x x 1 ^4 3 exp x 2 2 jacobian F1, 1.0, 2.0 let J jacobian F1, 1.0, 2.0 ref J ForwardDiff.jacobian F1, 1.0, 2.0 if J ≈ ref J correct elseif J 1,1 ≈ 4.0 && J 2,2 ≈ exp 2.0 keep working md\"The answer is not quite right Check your diagonal entries\" elseif J 1,2 0.0 keep working md\"The answer is not quite right Think back to one hot vectors\" else keep working end end F2 x x 1 ^4 3 exp x 2 2 log x 1 x 2 ^2 jacobian F2, 1.0, 2.0 let success false try J jacobian F2, 1.0, 2.0 success true catch end if success keep working md\"Your function is erroring \" else ref J ForwardDiff.jacobian F2, 1.0, 2.0 if J ≈ ref J correct elseif J 1,1 ≈ 4.0 && J 2,2 ≈ exp 2.0 keep working md\"The answer is not quite right Check your diagonal entries\" elseif J 3,1 ≈ 1.0 && J 3,2 ≈ 4.0 keep working md\"The new entries don't match my expectation.\" elseif J 1,2 0.0 keep working md\"The answer is not quite right Think back to one hot vectors\" else keep working end end end md\"\"\" Exercise Linked List \"\"\" mutable struct LinkedList T const value T next Union Nothing, LinkedList T LinkedList value T where T new T value, nothing end function collect ll LinkedList values ll.value while ll.next nothing push values, ll.next.value end values end ll LinkedList 1.0 collect ll md\"\"\" Let's write a function that appends values \"\"\" function append ll LinkedList, value Find tail while ll.next nothing ll ll.next end ll.next LinkedList value end let ll LinkedList 1.0 append ll, 2.0 TODO collect ll hangs ll end \"\"\" insert ll, after, value Insert a value after another \"\"\" function insert ll LinkedList, after, value while ll nothing if ll.value after ll.next LinkedList value break else ll ll.next end end end function delete ll LinkedList, value while ll nothing if ll.value value ll.next nothing break else ll ll.next end end end let ll LinkedList 1.0 append ll, 3.0 insert ll, 1.0, 2.0 OOPS where did 3.0 go? ll end let ll LinkedList 1.0 append ll, 2.0 append ll, 3.0 delete ll, 2.0 OOPS where did 3.0 go, and why is 2.0 still here? ll end "},{"url":"exercises/exercise_16_shmem_parallel/","title":"Shared memory parallelism","description":"sample exercise","tags":["module3","track_parallel","exercises"],"text":" A Pluto.jl notebook v1.0.1 frontmatter order \"8.3\" exercise number \"16\" title \"Shared memory parallelism\" tags \"module3\", \"track parallel\", \"exercises\" license \"MIT\" layout \"layout.jlhtml\" description \"sample exercise\" frontmatter.author name \"Valentin Churavy\" image \"https avatars.githubusercontent.com u 145258?v 4\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils using PlutoTeachingTools md\"\"\" Exercise Parallel PI We randomly throw \"darts\" at a circle with radius 1, contained in a square from 1 to 1 and count how many darts land within the circle. For ease we only sample the portion of the circle that is in the postive quadrant. \"\"\" question box md\"\"\" How would you parallelize the following code? Are there advantages of a \"functional\" style using mapreduce vs explicit for loops? \"\"\" using OhMyThreads function monte carlo pi N M mapreduce , 1 N do i rand ^2 rand ^2 1.0 end pi 4 M N return pi end hint md\"\"\" OhMyThreads.jl has a parallel `tmapreduce`. \"\"\" monte carlo pi 10 000 md\"\"\" Exercise Parallel Histogram Use `Base. threads` to parallelize. \"\"\" using Atomix atomic, atomicswap, atomicreplace question box md\"\"\" If you naively parallelize, is there a data race? If yes how would you fix it? Is there a risk of false sharing? Would implementing this algorithm with `tmapreduce` help? \"\"\" import StatsBase StatsBase, Histogram begin function histogram data, nbins, lb, ub buckets zeros Int, nbins width ub lb nbins 1 for x in data bucket round Int, x lb width 1 buckets bucket 1 end return Histogram lb width width ub, buckets, right end histogram data, nbins histogram data, nbins, extrema data end hint md\"\"\" ``` threads for x in data ``` \"\"\" using UnicodePlots data rand 1 64, 1024 UnicodePlots.vertical histogram StatsBase.fit Histogram, data, nbins 64, closed right , height 10 let data rand 1 64, 1024 hist histogram data, 64, 1, 64 UnicodePlots.vertical histogram hist, height 10 end md\"\"\" Exercise Parallel sorting note Implement a parallel merge sort using the skeleton below \"\"\" function merge out, left, right ll, lr length left , length right assert ll lr length out i, il, ir 1, 1, 1 inbounds while il ll && ir lr l, r left il , right ir if isless r, l out i r ir 1 else out i l il 1 end i 1 end inbounds while il ll out i left il il 1 i 1 end inbounds while ir lr out i right ir ir 1 i 1 end return out end perform a merge sort on `v` function psort v AbstractVector hi length v if hi 100 000 below some cutoff, run in serial return sort v, alg MergeSort end split the range and sort the halves in parallel recursively TODO Parallelize mid 1 hi 1 left psort view v, 1 mid right psort view v, mid 1 hi perform the merge on the result out similar v merge out, left, right return out end hint md\"\"\" Recall the parallel `fib` \"\"\" v rand 16 000 000 v sorted ours psort v v sorted theirs sort v v sorted ours v sorted theirs "},{"url":"exercises/exercise_17_gpu/","title":"Introduction to KernelAbstractions","description":"","tags":["module3","track_parallel","exercises"],"text":"Getting started on Google ColabRecently Google Colab started supporting Julia directly. One needs to switch the change the “runtime type” to “Julia”,\nand Google Colab also provides some GPU support for free.To get started with today’s exercise:"},{"url":"exercises/exercise_18_mpi/","title":"Running MPI.jl locally","description":"","tags":["module3","track_parallel","exercises"],"text":"ConfigurationThere are multiple MPI implementations:Open MPIMPICHIntel MPIMicrosoft MPIIBM Spectrum MPIMVAPICHCray MPICHFujitsu MPIHPE MPT/HMPTjulia> using MPI\n\njulia> MPI.versioninfo()\n\nMPIPreferences:\n  binary:  MPICH_jll\n  abi:     MPICH\n\nPackage versions\n  MPI.jl:             0.20.22\n  MPIPreferences.jl:  0.1.11\n  MPICH_jll:          4.3.0+1\n\nLibrary information:\n  libmpi:  /home/vchuravy/.julia/artifacts/05d8c79b270470018e9de8dd24ddb6d7954aff9d/lib/libmpi.so\n  libmpi dlpath:  /home/vchuravy/.julia/artifacts/05d8c79b270470018e9de8dd24ddb6d7954aff9d/lib/libmpi.so\n  MPI version:  4.1.0\n  Library version:\n    MPICH Version:      4.3.0\n    MPICH Release date: Mon Feb  3 09:09:47 AM CST 2025\n    MPICH ABI:          17:0:5\n    MPICH Device:       ch3:nemesis\n    MPICH configure:    --build=x86_64-linux-musl --disable-dependency-tracking --disable-doc --enable-fast=ndebug,O3 --enable-static=no --host=x86_64-linux-gnu --prefix=/workspace/destdir --with-device=ch3 --with-hwloc=/workspace/destdir\n    MPICH CC:           cc     -DNDEBUG -DNVALGRIND -O3\n    MPICH CXX:          c++   -DNDEBUG -DNVALGRIND -O3\n    MPICH F77:          gfortran   -O3\n    MPICH FC:           gfortran   -O3\n    MPICH features:\nOn Unix, MPI.jl will install and use MPICH through the MPICH_jll package.MPIPreferencesTo switch which MPI implementation MPI.jl uses you can use the package MPIPreferences.jl.For more information see: https://juliaparallel.org/MPI.jl/stable/configuration/When executing on a cluster you will likely need to configure MPI.jl to use the system provided MPI.Installing mpiexecjljulia> using MPI\njulia> MPI.install_mpiexecjl()\nBy default, it will install to ~/.julia/bin, but you can also choose to install it somewhere else.As an example, to install it in the current working directory:julia> using MPI\njulia> MPI.install_mpiexecjl(destdir=\".\")\nAfter installing it, you can use it to launch several Julia processes under MPI:mpiexecjl --project=/path/to/project -n 4 julia script.jl\n# or\n./mpiexecjl --project=/path/to/project -n 4 julia script.jl\nExercisesMPI.jl has a series of examples:Hello worldBroadcastReduceSend/receiveTo get started, copy the “Hello world” example into 01-hello.jl and run it with four processes:mpiexecjl --project=. -n 4 julia 01-hello.jl\nYou should see one line per rank, printed in a non-deterministic order — just like the @mpi hello-world from the lecture:Hello world, I have rank 0 out of 4 processors\nHello world, I have rank 2 out of 4 processors\nHello world, I have rank 1 out of 4 processors\nHello world, I have rank 3 out of 4 processors\nDiffusionIn exercise 7 we looked at a diffusion kernel.\nInstead of implementing this on the GPU you can also implement it with MPI.NoteThe “hard” part is the handling of the boundary conditions and ghost cells. So focus on that in the beginning.\nHow are you going to split the computational domain? Who needs to talk to whom?To check correctness, gather the distributed result back onto a single rank (with MPI.Gather) and compare it against the single-process version from exercise 7. Running with one rank (-n 1) should reproduce the serial result exactly, and the answer should not change as you increase the number of ranks."},{"url":"exercises/exercise_19_reverse/","title":"Reverse mode AD","description":"Implement reverse-mode AD by hand and check it against Enzyme","tags":["module4","track_ad","exercises"],"text":" A Pluto.jl notebook v1.0.1 frontmatter chapter \"4\" section \"1\" order \"11.1\" exercise number \"19\" title \"Reverse mode AD\" tags \"module4\", \"track ad\", \"exercises\" license \"MIT\" layout \"layout.jlhtml\" description \"Implement reverse mode AD by hand and check it against Enzyme\" frontmatter.author name \"Valentin Churavy\" image \"https avatars.githubusercontent.com u 145258?v 4\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils using PlutoTeachingTools using Enzyme function f1 x, y x exp y end autodiff Reverse, f1, Active 2.0 , Active 1.0 md\"\"\" Exercise Manually write the reverse mode implementation for `f1`. \"\"\" function df1 x, y ... return dx, dy end md\"\"\" Enzyme allows for a selection of arguments to be \"active\". How would your implementation change if you knew that you didn't need the gradient of an argument? \"\"\" | question box autodiff Reverse, f1, Active 2.0 , Const 1.0 autodiff Reverse, f1, Const 2.0 , Active 1.0 function f2 x Array Float64 , y Array Float64 y 1 x 1 x 1 x 2 x 1 return nothing end let x 2.0, 2.0 dx 0.0, 0.0 y 0.0 dy 1.0 Enzyme.autodiff Reverse, f2, Duplicated x, dx , Duplicated y, dy dx, dy end md\"\"\" Exercise Manually write the reverse mode implementation for `f2`. note It might be very helpful to \"desugar\" the implementation as much as possible. \"\"\" md\"\"\" Note that Enzyme zeros out the `dy` memory. What would happen if it didn't? \"\"\" | question box f3 x exp sin x ^2 md\"\"\" Exercise `f3` is a composition of several elementary functions. Manually write its reverse mode implementation `df3 x ` by applying the chain rule outward to inward, and check it against `autodiff Reverse, f3, Active x `. \"\"\" function df3 x ... return dx end "},{"url":"exercises/exercise_1_functions_unicode/","title":"Functions and Unicode","description":"Practice writing Julia functions and using Unicode identifiers","tags":["module1","track_principles","exercises"],"text":" A Pluto.jl notebook v1.0.1 frontmatter order \"2.1\" exercise number \"1\" title \"Functions and Unicode\" tags \"module1\", \"track principles\", \"exercises\" layout \"layout.jlhtml\" license \"MIT\" description \"Practice writing Julia functions and using Unicode identifiers\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils using PlutoTeachingTools, PlutoUI PlutoUI.TableOfContents depth 4 md\"\"\" Exercise Functions and Unicode Julia supports full Unicode in identifiers, making it easy to write mathematics that reads like a textbook. \"\"\" md\"\"\" Part 1 — Circle area Write a function `circle area r ` that returns the area of a circle with radius `r`. Use the built in constant `π` type `\\pi` then press Tab to get the Unicode symbol . Do not hard code an approximation like `3.14159`. \"\"\" Write circle area here let if isdefined circle area func not defined circle area elseif circle area 1.0 ≈ π keep working md\"`circle area 1.0 ` should equal `π`.\" elseif circle area 2.0 ≈ 4π keep working md\"`circle area 2.0 ` should equal `4π`.\" else correct end end answer box hint md\"\"\" ```julia circle area r π r^2 ``` \"\"\" md\"\"\" Part 2 — Quadratic formula Write a function `quadratic a, b, c ` that returns both roots of ax^2 bx c 0 as a tuple, using the quadratic formula x \\frac b \\pm \\sqrt b^2 4ac 2a Use `√` type `\\sqrt` then Tab or `sqrt` for the square root. Check `quadratic 1, 3, 2 ` should return the roots `2.0` and `1.0` in either order . \"\"\" Write quadratic here let if isdefined quadratic func not defined quadratic else r1, r2 quadratic 1, 3, 2 expected Set 1.0, 2.0 if Set r1, r2 expected keep working md\"`quadratic 1, 3, 2 ` should return the roots `1.0` and `2.0`.\" else correct end end end answer box hint md\"\"\" ```julia function quadratic a, b, c disc √ b^2 4a c b disc 2a , b disc 2a end ``` \"\"\" md\"\"\" Part 3 — Unicode variable names Rewrite the snippet below so that the variable names use proper Greek letters `α`, `β`, `γ` via `\\alpha`, `\\beta`, `\\gamma` Tab . Then define a function `linear combo α, β ` that returns `α β α^2`. ```julia alpha 0.5 beta 1.5 gamma alpha beta alpha^2 ``` \"\"\" Write linear combo here and optionally the α, β, γ variables let if isdefined linear combo func not defined linear combo elseif linear combo 0.5, 1.5 ≈ 0.5 1.5 0.5^2 keep working md\"`linear combo 0.5, 1.5 ` should equal `0.5 1.5 0.5^2 1.0`.\" else correct end end answer box hint md\"\"\" ```julia α 0.5 β 1.5 γ α β α^2 linear combo α, β α β α^2 linear combo 0.5, 1.5 1.0 ``` \"\"\" md\"\"\" Part 3 — Playing darts to estimate Pi You and some friends are playing darts. You all are very bad at darts. Each throw is guaranteed to hit the square board depicted below, but otherwise each throw will land in a completely random position within the square. To entertain yourself during this pathetic display, you decide to use this as an opportunity to estimate the irrational number π Because each throw falls randomly within the square, you realize that the probability of a dart landing within the circle is given by the ratio of the circle’s area to the square’s area ```math P circle \\frac Area circle Area square \\frac \\pi r^2 2r ^2 ``` Furthermore, we can interpret P circle as being approximated by the fraction of darts thrown that land in the circle. Thus, we find ```math \\frac N circle N total \\approx \\frac \\pi r^2 2r ^2 \\frac \\pi 4 ``` where N total is the total number of darts thrown, and is N circle the number of darts that land within the circle. Thus simply by keeping tally of where the darts land, you can begin to estimate the value of π \"\"\" md\"\"\" Write code that simulates the dart throwing and tallying process. For simplicity, you can assume that the board is centered at 0,0 , and that r 1 the radius of the circle . Use `rand` to randomly generate the positions on the board where the darts land. Do this for N darts in total. For each dart thrown determine whether or not it landed within the circle, and update your estimate of π according to the formula N circle N total \\pi 4 Keep in mind that each dart can land in x∈ 1,1 , y∈ 1,1 and that a dart that lands x,y at falls within the circle if \\sqrt x^2 y^2 1 \"\"\" write a estimate pi N function here let if isdefined estimate pi func not defined estimate pi elseif round estimate pi 100 000 ,digits 1 round π,digits 1 keep working md\"`estimate pi 100 000 ` should equal `round π,digits 1 `.\" elseif round estimate pi 100 000 000 ,digits 2 round π,digits 2 keep working md\"`estimate pi 100 000 000 ` should equal `round π,digits 2 `.\" else correct end end round π,digits 4 answer box hint md\"\"\" ```julia function estimate pi N 10 000 total 0 circle 0 for in 1 N x 2 rand 1 y 2 rand 1 if sqrt x^2 y^2 1 circle 1 end total 1 end 4 circle total end ``` \"\"\" "},{"url":"exercises/exercise_20_ml/","title":"Machine Learning","description":"Build a neural network from perceptrons and activation functions","tags":["module4","track_ad","exercises"],"text":" A Pluto.jl notebook v1.0.1 frontmatter chapter \"4\" section \"1\" order \"11.2\" exercise number \"20\" title \"Machine Learning\" tags \"module4\", \"track ad\", \"exercises\" license \"MIT\" layout \"layout.jlhtml\" description \"Build a neural network from perceptrons and activation functions\" frontmatter.author name \"Valentin Churavy\" image \"https avatars.githubusercontent.com u 145258?v 4\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils using UnicodePlots md\"\"\" x j b j \\sum i w ij x i \"\"\" function perceptron inputs, weights, bias, f f sum weights i inputs i for i in 1 length inputs bias end lineplot 3,3, identity relu x max zero x , x lineplot 3,3, relu sigmoid x 1 1 exp x lineplot 3,3, sigmoid mytanh x exp x exp x exp x exp x lineplot 3,3, mytanh md\"\"\" While one can define a perceptron like above, it is better to think of it in terms of matrix multiplication. \"\"\" Ni 10 Nn 5 x rand Ni b rand Nn W rand Nn, Ni mytanh. W x . b md\"\"\" Exercise Write your own Neural Network model with a couple of layers maybe 2 Remember https vchuravy.dev rse course 2026 exercises exercise 10 gradient descent as a starting point. \"\"\" "},{"url":"exercises/exercise_2_multiple_dispatch/","title":"Multiple Dispatch","description":"Explore Julia's multiple dispatch system by adding methods to functions","tags":["module1","track_principles","exercises"],"text":" A Pluto.jl notebook v1.0.1 frontmatter order \"2.2\" exercise number \"2\" title \"Multiple Dispatch\" tags \"module1\", \"track principles\", \"exercises\" layout \"layout.jlhtml\" license \"MIT\" description \"Explore Julia's multiple dispatch system by adding methods to functions\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils using PlutoTeachingTools, PlutoUI ChooseDisplayMode PlutoUI.TableOfContents depth 4 md\"\"\" Exercise Multiple Dispatch Multiple dispatch is Julia's central design feature a function call selects a method based on the types of all its arguments. This is what makes Julia code both fast and composable. \"\"\" md\"\"\" Part 1 — Inspecting dispatch The cell below defines a `describe` function with several methods. Run it, then experiment with `methods describe ` and ` which describe x ` for different values of `x`. \"\"\" begin describe x Int \" x is an Int\" describe x Float64 \" x is a Float64\" describe x AbstractString \"\\\" x \\\" is a String\" describe x \" x is a typeof x \" end Try methods describe , which describe 1 , which describe \"hi\" , which describe true md\"\"\" Part 2 — Writing your own dispatched function Define a function `add a, b ` with four methods | Argument types | Behaviour | | | | | `Integer, Integer` | return `Int a b ` | | `AbstractFloat, AbstractFloat` | return `Float64 a b ` | | `Integer, AbstractFloat` | convert `a` to `Float64`, then sum | | `AbstractFloat, Integer` | convert `b` to `Float64`, then sum | \"\"\" Write your add methods here let if isdefined add func not defined add elseif add 1, 2 Int 3 keep working md\"`add 1, 2 ` should return an `Int` equal to `3`.\" elseif add 1.0, 2.0 Float64 3.0 keep working md\"`add 1.0, 2.0 ` should return a `Float64` equal to `3.0`.\" elseif add 1, 2.0 Float64 3.0 keep working md\"`add 1, 2.0 ` should return a `Float64` equal to `3.0`.\" elseif add 1.0, 2 Float64 3.0 keep working md\"`add 1.0, 2 ` should return a `Float64` equal to `3.0`.\" else correct end end answer box hint md\"\"\" ```julia add a Integer, b Integer Int a b add a AbstractFloat, b AbstractFloat Float64 a b add a Integer, b AbstractFloat Float64 a b add a AbstractFloat, b Integer a Float64 b ``` Check which method is called ```julia which add 1, 2 Integer, Integer which add 1.0, 2.0 AbstractFloat, AbstractFloat which add 1, 2.0 Integer, AbstractFloat ``` \"\"\" md\"\"\" Part 3 — Dispatch on abstract types Define a function `my abs x ` with two methods `my abs x Real ` — returns the absolute value of a real number. `my abs z Complex ` — returns the modulus \\sqrt \\text re z ^2 \\text im z ^2 . Check `my abs 3 3`, `my abs 3.0 3.0`, and `my abs 3 4im 5.0`. \"\"\" Write my abs here let if isdefined my abs func not defined my abs elseif my abs 3 3 keep working md\"`my abs 3 ` should equal `3`.\" elseif my abs 3.0 3.0 keep working md\"`my abs 3.0 ` should equal `3.0`.\" elseif my abs 3 4im ≈ 5.0 keep working md\"`my abs 3 4im ` should equal `5.0`.\" else correct end end answer box hint md\"\"\" ```julia my abs x Real x 0 ? x x my abs z Complex sqrt real z ^2 imag z ^2 ``` ` which my abs 3 4im ` confirms the `Complex` method is dispatched. \"\"\" md\"\"\" Part 4 Let's play Rock Paper Scissors Define a function `play Type A , Type B ` that implements the rules of rock paper scissors. The function should return one of three strings depending on the outcome | Combination | Result | | | | | Rock vs Scissors | `\"Rock wins\"` | | Paper vs Rock | `\"Paper wins\"` | | Scissors vs Paper | `\"Scissors wins\"` | | Same vs Same | `\"Tie, try again\"` | | Reversed combinations | same result symmetric | For example `play Paper, Rock ` → `\"Paper wins\"` and `play Rock, Paper ` → `\"Paper wins\"`. note \"Question\" How many methods do you need to define? Can you use dispatch to avoid listing all 9 combinations? \"\"\" begin abstract type Shape end struct Rock Shape end struct Paper Shape end struct Scissors Shape end end Write play here let if isdefined play func not defined play else try if play Paper, Scissors \"Scissors wins\" keep working md\"`play Paper, Scissors ` should return `\\\"Scissors wins\\\"`.\" elseif play Scissors, Paper \"Scissors wins\" keep working md\"`play Scissors, Paper ` should return `\\\"Scissors wins\\\"`.\" elseif play Rock, Paper \"Paper wins\" keep working md\"`play Rock, Paper ` should return `\\\"Paper wins\\\"`.\" elseif play Paper, Rock \"Paper wins\" keep working md\"`play Paper, Rock ` should return `\\\"Paper wins\\\"`.\" elseif play Rock, Scissors \"Rock wins\" keep working md\"`play Rock, Scissors ` should return `\\\"Rock wins\\\"`.\" elseif play Scissors, Rock \"Rock wins\" keep working md\"`play Scissors, Rock ` should return `\\\"Rock wins\\\"`.\" elseif play Scissors, Scissors \"Tie, try again\" keep working md\"`play Scissors, Scissors ` should return `\\\"Tie, try again\\\"`.\" elseif play Rock, Rock \"Tie, try again\" keep working md\"`play Rock, Rock ` should return `\\\"Tie, try again\\\"`.\" elseif play Paper, Paper \"Tie, try again\" keep working md\"`play Paper, Paper ` should return `\\\"Tie, try again\\\"`.\" elseif length methods play 5 almost md\"All outcomes are correct But you defined length methods play methods — can you reduce to 5? Hint think about symmetry and a catch all for ties.\" else correct end catch e if e isa MethodError keep working md\"Your `play` function is missing a method for some combination. Make sure all 9 pairs are handled or use dispatch to cover them generically . Missing ` sprint showerror, e `\" else keep working md\"Your function threw an unexpected error ` sprint showerror, e `\" end end end end answer box hint md\"\"\" ```julia play Type Paper , Type Rock \"Paper wins\" play Type Paper , Type Scissors \"Scissors wins\" play Type Rock , Type Scissors \"Rock wins\" play Type T , Type T where T Shape \"Tie, try again\" play a Type Shape , b Type Shape play b, a symmetry ``` This uses only 5 methods 3 canonical winners, 1 tie, and 1 symmetry rule. ` which play Rock, Paper ` will show which method is selected. \"\"\" if isdefined play play Scissors, rand Scissors, Paper, Rock end "},{"url":"exercises/exercise_3_broadcasting/","title":"Broadcasting and Array Comprehensions","description":"Practice Julia's broadcasting syntax and array comprehensions for element-wise operations","tags":["module1","track_principles","exercises"],"text":" A Pluto.jl notebook v1.0.1 frontmatter order \"2.4\" exercise number \"3\" title \"Broadcasting and Array Comprehensions\" tags \"module1\", \"track principles\", \"exercises\" layout \"layout.jlhtml\" license \"MIT\" description \"Practice Julia's broadcasting syntax and array comprehensions for element wise operations\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils using PlutoTeachingTools, PlutoUI ChooseDisplayMode PlutoUI.TableOfContents depth 4 md\"\"\" Exercise Broadcasting and Array Comprehensions Julia has two idiomatic ways to apply element wise operations to arrays Broadcasting — apply any function or operator element wise using the dot `.` notation. Array comprehensions — a concise ` expr for x in collection if condition ` syntax. Both compile down to efficient single pass loops. \"\"\" md\"\"\" Part 1 — Squaring a vector Given `v 1 10`, store the vector of squares in a variable `sq broadcast` using broadcasting, and in `sq comp` using an array comprehension. Check they agree. \"\"\" v 1 10 Write sq broadcast and sq comp here let if isdefined sq broadcast func not defined sq broadcast elseif isdefined sq comp func not defined sq comp elseif sq broadcast 1,4,9,16,25,36,49,64,81,100 keep working md\"`sq broadcast` doesn't look right — it should be ` 1, 4, 9, ..., 100 `.\" elseif sq comp sq broadcast keep working md\"`sq comp` and `sq broadcast` should be equal.\" else correct end end answer box hint md\"\"\" ```julia sq broadcast v .^ 2 sq comp x^2 for x in v sq broadcast sq comp true ``` \"\"\" md\"\"\" Part 2 — Filtering with comprehensions Write a one liner that collects only the even squares from `v 1 10` into a variable `even sq`. Expected ` 4, 16, 36, 64, 100 ` \"\"\" Write even sq here let if isdefined even sq func not defined even sq elseif even sq 4, 16, 36, 64, 100 keep working md\"`even sq` should be ` 4, 16, 36, 64, 100 `.\" else correct end end answer box hint md\"\"\" ```julia Conditional comprehension even sq x^2 for x in v if iseven x Alternative filter on broadcast result even sq filter iseven, v .^ 2 ``` \"\"\" md\"\"\" Part 3 — Broadcasting a custom function Define a function `clamp01 x ` that clips a value to the interval 0, 1 i.e. returns `0.0` if `x 0`, `1.0` if `x 1`, and `x` otherwise . Then broadcast it over `w 0.5, 0.0, 0.3, 0.7, 1.0, 1.5 `. Store the result in `clamped`. \"\"\" w 0.5, 0.0, 0.3, 0.7, 1.0, 1.5 Define clamp01 and compute clamped here let if isdefined clamp01 func not defined clamp01 elseif isdefined clamped func not defined clamped elseif clamp01 1.0 0.0 && clamp01 0.5 0.5 && clamp01 2.0 1.0 keep working md\"`clamp01` doesn't clip correctly — check the boundary conditions.\" elseif clamped 0.0, 0.0, 0.3, 0.7, 1.0, 1.0 keep working md\"`clamped` should be ` 0.0, 0.0, 0.3, 0.7, 1.0, 1.0 `.\" else correct end end answer box hint md\"\"\" ```julia clamp01 x max 0.0, min 1.0, x clamped clamp01. w 0.0, 0.0, 0.3, 0.7, 1.0, 1.0 ``` Julia also has a built in `clamp. w, 0.0, 1.0 `. \"\"\" md\"\"\" Part 4 — 2D comprehension Use a nested comprehension to build the multiplication table for 1 through 5 and store it in `T` a 5×5 matrix where `T i,j i j` . \"\"\" Write T here let if isdefined T func not defined T elseif size T 5, 5 keep working md\"`T` should be a 5×5 matrix.\" elseif T i j for i in 1 5, j in 1 5 keep working md\"`T i,j ` should equal `i j`.\" else correct end end answer box hint md\"\"\" ```julia T i j for i in 1 5, j in 1 5 Alternative outer product T 1 5 1 5 ' ``` The syntax ` expr for i in rows, j in cols ` produces a 2D array directly. \"\"\" "},{"url":"exercises/exercise_4_structs/","title":"Structs and Custom Types","description":"Define custom Julia types with structs and extend them via multiple dispatch","tags":["module1","track_principles","exercises"],"text":" A Pluto.jl notebook v1.0.1 frontmatter order \"2.5\" exercise number \"4\" title \"Structs and Custom Types\" tags \"module1\", \"track principles\", \"exercises\" layout \"layout.jlhtml\" license \"MIT\" description \"Define custom Julia types with structs and extend them via multiple dispatch\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils using PlutoTeachingTools, PlutoUI PlutoUI.TableOfContents depth 4 md\"\"\" Exercise Structs and Custom Types Julia lets you define your own types with `struct` and extend any function — including those from `Base` — via multiple dispatch. This is the mechanism behind Julia's composability packages can work together without knowing about each other in advance. \"\"\" md\"\"\" Part 1 — A 2D point type Define a struct `Point2D` with two `Float64` fields, `x` and `y`. Then extend `Base. a, b ` — pointwise addition `Base. a, b ` — pointwise subtraction `Base. s, p ` where `s Real` — scalar multiplication `LinearAlgebra.norm p ` — Euclidean norm \\sqrt x^2 y^2 \"\"\" Define Point2D and its methods here let if isdefined Point2D func not defined Point2D else p1 Point2D 3.0, 0.0 p2 Point2D 0.0, 4.0 p3 try p1 p2 catch nothing end if isnothing p3 keep working md\"Addition `p1 p2` threw an error — implement `Base. ` for `Point2D`.\" elseif p3 isa Point2D && p3.x ≈ 3.0 && p3.y ≈ 4.0 keep working md\"`Point2D 3,0 Point2D 0,4 ` should give `Point2D 3.0, 4.0 `.\" elseif norm p3 ≈ 5.0 keep working md\"`norm Point2D 3.0, 4.0 ` should be `5.0`.\" elseif 2 p1 isa Point2D && 2 p1 .x ≈ 6.0 keep working md\"`2 Point2D 3.0, 0.0 ` should give `Point2D 6.0, 0.0 `.\" else correct end end end answer box hint md\"\"\" ```julia struct Point2D x Float64 y Float64 end Base. a Point2D, b Point2D Point2D a.x b.x, a.y b.y Base. a Point2D, b Point2D Point2D a.x b.x, a.y b.y Base. s Real, p Point2D Point2D s p.x, s p.y LinearAlgebra.norm p Point2D sqrt p.x^2 p.y^2 ``` \"\"\" md\"\"\" Part 2 — Parametric struct The `Point2D` above always uses `Float64`. Make it parametric so it works for any numeric element type ```julia struct Point T Real x T y T end ``` Re implement ` `, ` `, ` `, and `norm` for `Point T `. Check that `Point 3, 4 ` integers and `Point 3.0f0, 4.0f0 ` Float32 both work and that the norm result type follows from the element type. \"\"\" Define Point T and its methods here let if isdefined Point func not defined Point else p int try Point 3, 4 catch nothing end p f32 try Point 3.0f0, 4.0f0 catch nothing end if isnothing p int keep working md\"`Point 3, 4 ` threw an error.\" elseif isnothing p f32 keep working md\"`Point 3.0f0, 4.0f0 ` threw an error.\" elseif norm p int ≈ 5.0 keep working md\"`norm Point 3, 4 ` should be `5.0`.\" elseif norm p f32 isa Float32 keep working md\"`norm Point 3.0f0, 4.0f0 ` should return a `Float32`.\" else correct end end end answer box hint md\"\"\" ```julia struct Point T Real x T y T end Base. a Point, b Point Point a.x b.x, a.y b.y Base. a Point, b Point Point a.x b.x, a.y b.y Base. s Real, p Point Point s p.x, s p.y LinearAlgebra.norm p Point sqrt p.x^2 p.y^2 ``` `sqrt` preserves element type `sqrt 3^2 4^2 ` → `Float64`, `sqrt 3.0f0^2 4.0f0^2 ` → `Float32`. \"\"\" md\"\"\" Part 3 — Custom `show` Implement `Base.show io IO, p Point ` so that a point displays as ` 3.0, 4.0 ` rather than the default `Point Float64 3.0, 4.0 `. \"\"\" Implement show here let if isdefined Point func not defined Point else p Point 3.0, 4.0 s sprint show, p if s \" 3.0, 4.0 \" correct else keep working md\"`show` should display the point as ` x, y `, but got ` s `.\" end end end answer box hint md\"\"\" ```julia Base.show io IO, p Point print io, \" \", p.x, \", \", p.y, \" \" ``` After defining this, `Point 3.0, 4.0 ` will render as ` 3.0, 4.0 ` in Pluto cells and in the REPL. \"\"\" "},{"url":"exercises/exercise_5_statistics/","title":"Statistics","description":"Explore a real dataset: filter DataFrames, compute summary statistics, and create labelled plots","tags":["module1","track_principles","exercises"],"text":" A Pluto.jl notebook v1.0.1 frontmatter order \"2.6\" exercise number \"5\" title \"Statistics\" tags \"module1\", \"track principles\", \"exercises\" layout \"layout.jlhtml\" license \"MIT\" description \"Explore a real dataset filter DataFrames, compute summary statistics, and create labelled plots\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils using PlutoTeachingTools, PlutoUI using RDatasets using CairoMakie using AlgebraOfGraphics using Statistics ChooseDisplayMode PlutoUI.TableOfContents depth 4 md\"\"\" Exercise Statistics and Visualization Julia's ecosystem makes it easy to load real datasets, compute statistics, and create publication quality plots — often in just a few lines. In this exercise you will work with the classic iris dataset 150 flower measurements across three species to practise DataFrame filtering, summary statistics, and plotting with Makie and AlgebraOfGraphics. \"\"\" iris dataset \"datasets\", \"iris\" md\"\"\" Part 1 — Filtering a DataFrame The `iris` dataset has 150 rows and five columns `SepalLength`, `SepalWidth`, `PetalLength`, `PetalWidth`, and `Species`. Filter `iris` to keep only the rows where `Species \"setosa\"` and store the result in a variable called `setosa`. Use logical indexing `df condition, ` where `condition` is a Boolean vector produced by broadcasting `. ` over a column. \"\"\" Write setosa here let if isdefined setosa func not defined setosa elseif size setosa, 1 50 keep working md\"`setosa` should have 50 rows one per setosa flower .\" elseif all setosa.Species . \"setosa\" keep working md\"Not all rows in `setosa` have `Species \\\"setosa\\\"`.\" else correct end end answer box hint md\"\"\" ```julia setosa iris iris.Species . \"setosa\", ``` or equivalently ```julia setosa iris iris , Species . \"setosa\", ``` \"\"\" md\"\"\" Part 2 — Summary statistics Using `mean` from `Statistics`, compute the mean sepal length for each species and store the results in three variables `mean setosa` — mean `SepalLength` for setosa `mean versicolor` — mean `SepalLength` for versicolor `mean virginica` — mean `SepalLength` for virginica Expected values to 3 d.p. `5.006`, `5.936`, `6.588`. \"\"\" Write mean setosa, mean versicolor, mean virginica here let if isdefined mean setosa func not defined mean setosa elseif isdefined mean versicolor func not defined mean versicolor elseif isdefined mean virginica func not defined mean virginica elseif mean setosa ≈ 5.006 keep working md\"`mean setosa` should be approximately `5.006`.\" elseif mean versicolor ≈ 5.936 keep working md\"`mean versicolor` should be approximately `5.936`.\" elseif mean virginica ≈ 6.588 keep working md\"`mean virginica` should be approximately `6.588`.\" else correct end end answer box hint md\"\"\" ```julia mean setosa mean iris iris.Species . \"setosa\", SepalLength mean versicolor mean iris iris.Species . \"versicolor\", SepalLength mean virginica mean iris iris.Species . \"virginica\", SepalLength ``` \"\"\" md\"\"\" Part 3 — Basic scatter plot Write a function `my scatter ` that creates and returns a `Figure` with a scatter plot of `iris.SepalLength` x axis, labelled `\"Sepal length\"` vs `iris.PetalWidth` y axis, labelled `\"Petal width\"` . ```julia fig Figure ax Axis fig 1,1 , xlabel ..., ylabel ... scatter ax, xs, ys fig ``` \"\"\" Write my scatter here let if isdefined my scatter func not defined my scatter else result try my scatter catch e e end if result isa Exception keep working md\"`my scatter ` threw an error.\" elseif result isa Figure keep working md\"`my scatter ` should return a `Figure`.\" else correct end end end answer box hint md\"\"\" ```julia function my scatter fig Figure ax Axis fig 1,1 , xlabel \"Sepal length\", ylabel \"Petal width\" scatter ax, iris.SepalLength, iris.PetalWidth fig end ``` \"\"\" md\"\"\" Part 4 — Grouped scatter plot The plain scatter above mixes all three species. Write a function `species scatter ` that draws one scatter series per species so they are visually distinct, with a legend. Use `groupby iris, Species ` to iterate over species subsets. For each subset, call `scatter ax, ... label ... `. Finish with `axislegend ax `. \"\"\" Write species scatter here let if isdefined species scatter func not defined species scatter else result try species scatter catch e e end if result isa Exception keep working md\"`species scatter ` threw an error.\" elseif result isa Figure keep working md\"`species scatter ` should return a `Figure`.\" else correct end end end answer box hint md\"\"\" ```julia function species scatter fig Figure ax Axis fig 1,1 , xlabel \"Sepal length\", ylabel \"Petal width\" for subdf in groupby iris, Species scatter ax, subdf.SepalLength, subdf.PetalWidth label string first subdf.Species end axislegend ax fig end ``` \"\"\" md\"\"\" Part 5 — AlgebraOfGraphics AlgebraOfGraphics AoG provides a higher level, declarative interface for plots. The same grouped scatter from Part 4 can be expressed in a single pipeline using ` ` to combine a data source, variable mappings, and visual marks \"\"\" data iris mapping SepalLength, PetalWidth mapping color Species | draw md\"\"\" AoG also makes faceted plots easy. The penguins dataset below is split by `sex` rows and `island` columns in a single spec, with a linear trend overlaid \"\"\" let spec data AlgebraOfGraphics.penguins mapping bill length mm \"Bill length mm \", bill depth mm \"Bill depth mm \", color species \"Species\", row sex, col island, visual Scatter, alpha 0.3 linear draw spec end "},{"url":"exercises/exercise_6_multithreading/","title":"Shared-memory parallelism","description":"Practice shared-memory parallelism in Julia using Threads.@spawn for recursive parallelism and concurrent map","tags":["module1","track_parallel","exercises"],"text":" A Pluto.jl notebook v1.0.1 frontmatter order \"3.1\" exercise number \"6\" title \"Shared memory parallelism\" tags \"module1\", \"track parallel\", \"exercises\" layout \"layout.jlhtml\" license \"MIT\" description \"Practice shared memory parallelism in Julia using Threads. spawn for recursive parallelism and concurrent map\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils using PlutoTeachingTools, PlutoUI ChooseDisplayMode PlutoUI.TableOfContents depth 4 md\"\"\" Exercise Shared memory parallelism Julia's multi threading model lets you spawn lightweight tasks that the scheduler maps onto available CPU cores. In this exercise you will use `Threads. spawn` to parallelise a recursive algorithm and a map operation. \"\"\" md\"\"\" This notebook uses Threads.nthreads threads \"\"\" md\"\"\" Part 1 — Parallel fibonacci Remember ```julia t Threads. spawn begin ` spawn` returns right away 3 3 end fetch t `fetch` waits for the task to finish ``` \"\"\" function fib n if n 1 return n end return fib n 1 fib n 2 end fib 12 TODO Implement pfib let if isdefined pfib func not defined pfib elseif pfib 12 fib 12 keep working md\"Your solution and the reference solution disagree \" else correct end end answer box hint md\"\"\" ```julia function pfib n if n 1 return n end t Threads. spawn pfib n 2 return pfib n 1 fetch t Int end ``` \"\"\" md\"\"\" Part 2 — Multi threaded map \"\"\" using LinearAlgebra, Random using BenchmarkTools function tmap fn, itr for each i ∈ itr, spawn a task to compute fn i tasks map i Threads. spawn fn i , itr fetch and return all the results return fetch. tasks end Ms rand 100,100 for i in 1 8 Threads.nthreads begin BLAS.set num threads Sys.CPU THREADS Fix number of BLAS threads BLAS.set num threads 1 blas edge nothing end begin blas edge serial map svdals b benchmark map svdvals, Ms samples 10 evals 3 end begin blas edge threaded map svdals b benchmark tmap svdvals, Ms samples 10 evals 3 end minimum serial map svdals b.times minimum threaded map svdals b.times Threads.nthreads 100 parallel efficiency md\"\"\" note Vary the number of threads the BLAS library uses. See the cell above with `BLAS.set num threads ` \"\"\" md\"\"\" Task — chunked map `tmap` above spawns one task per element. For large arrays with cheap per element work the task spawning overhead can dominate. Implement `tmap chunked fn, itr, chunk size ` that splits `itr` into chunks of `chunk size` elements and spawns one task per chunk. Hint `Iterators.partition itr, chunk size ` splits an iterable into fixed size pieces. \"\"\" TODO Implement tmap chunked let if isdefined tmap chunked func not defined tmap chunked elseif tmap chunked x x^2, 1 10, 3 map x x^2, collect 1 10 keep working md\"`tmap chunked x x^2, 1 10, 3 ` should equal ` 1, 4, 9, 16, 25, 36, 49, 64, 81, 100 `.\" else correct end end answer box hint md\"\"\" ```julia function tmap chunked fn, itr, chunk size chunks Iterators.partition itr, chunk size tasks map chunk Threads. spawn map fn, collect chunk , chunks vcat fetch. tasks ... end ``` \"\"\" "},{"url":"exercises/exercise_7_accelerated/","title":"Introduction to accelerated computing","description":"Write portable GPU kernels using KernelAbstractions.jl, targeting CUDA, Metal, oneAPI, and CPU backends","tags":["module1","track_parallel","exercises"],"text":" A Pluto.jl notebook v1.0.1 frontmatter order \"3.2\" exercise number \"7\" title \"Introduction to accelerated computing\" tags \"module1\", \"track parallel\", \"exercises\" layout \"layout.jlhtml\" license \"MIT\" description \"Write portable GPU kernels using KernelAbstractions.jl, targeting CUDA, Metal, oneAPI, and CPU backends\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils This Pluto notebook uses bind for interactivity. When running this notebook outside of Pluto, the following 'mock version' of bind gives bound variables a default value instead of an error . macro bind def, element format off return quote local iv try Base.loaded modules Base.PkgId Base.UUID \"6e696c72 6542 2067 7265 42206c756150\" , \"AbstractPlutoDingetjes\" .Bonds.initial value catch b missing end local el esc element global esc def Core.applicable Base.get, el ? Base.get el iv el el end format on end using PlutoTeachingTools, PlutoUI PlutoUI.TableOfContents depth 4 md\"\"\" Exercise Introduction to accelerated computing KernelAbstractions.jl https github.com JuliaGPU KernelAbstractions.jl is a lightweight DSL for writing portable GPU kernels in Julia. A single kernel runs unchanged on any supported backend CUDA.jl — NVIDIA GPUs Metal.jl — Apple Silicon GPUs oneAPI.jl — Intel GPUs AMDGPU.jl — AMD GPUs CPU — always available as a fallback \"\"\" using LinearAlgebra, Random using BenchmarkTools using KernelAbstractions begin import CUDA import Metal Work around https github.com JuliaGPU oneAPI.jl issues 445 . ENV \"SYCL PI LEVEL ZERO BATCH SIZE\" \"1\" import oneAPI sigh Currently crashes inside Pluto import AMDGPU end md\"\"\" Choose your own backend \"\"\" let available backends KernelAbstractions.Backend CPU CI get ENV, \"GITHUB ACTIONS\", \"false\" \"true\" Always shows all backend when rendering the notebook on GitHub Actions, notebooks will be static anyway. if CUDA.functional || CI push available backends, CUDA.CUDABackend end if oneAPI.functional || CI push available backends, oneAPI.oneAPIBackend end if Metal.functional || CI push available backends, Metal.MetalBackend end if AMDGPU.functional || CI push available backends, AMDGPU.ROCBackend end bind backend Select available backends end md\"\"\" Part 1 — AXPY serial implementation The AXPY operation is defined as Y \\leftarrow Y a \\cdot X note Vary `M` below to explore performance at different problem sizes. \"\"\" M 1024 function axpy serial Y, a, X for i in eachindex Y, X Write the AXPY update for Y i here end return nothing end let Y randn Float32, M X randn Float32, M a randn Float32 reference Y . a . X axpy serial Y, a, X if reference Y correct else keep working md\"`axpy serial ` did not produce the expected result — make sure you update `Y i ` in the loop body.\" end end answer box hint md\"\"\" ```julia function axpy serial Y, a, X for i in eachindex Y, X Y i a X i end return nothing end ``` \"\"\" md\"\"\" Part 2 — AXPY with KernelAbstractions Now implement the same operation as a KernelAbstractions kernel. The ` index Global ` macro returns the global thread index — use it to address a single element of `Y` and `X`. \"\"\" kernel function axpy kernel Y, a, X Write the single element AXPY update here use index Global end Creating a wrapper kernel for launching with error checks function axpy ka Y, a, X if length Y length X error \"Arrays disagree in length\" return nothing end backend KernelAbstractions.get backend Y kernel axpy kernel backend kernel Y, a, X, ndrange size Y KernelAbstractions.synchronize backend return end let Y randn allocate backend, Float32, M X randn allocate backend, Float32, M a randn Float32 reference Y . a . X axpy ka Y, a, X if reference Y correct else keep working md\"`axpy kernel ` did not produce the expected result — use ` index Global ` to get `i`, then update `Y i `.\" end end answer box hint md\"\"\" ```julia function axpy kernel Y, a, X i index Global Y i a X i end ``` \"\"\" question box md\"Benchmark the KernelAbstraction implementation against a high level array implementation at different problem sizes\" md\"\"\" Part 3 — 2D diffusion kernel \"\"\" begin using OffsetArrays using CairoMakie end begin a 0.01 dx 0.01 x grid spacing dy 0.01 y grid spacing end dt dx^2 dy^2 2.0 a dx^2 dy^2 Largest stable time step N 64 md\"\"\" Implement a kernel that solves the 2D Diffusion Equation. \\frac \\partial U \\partial t a \\frac \\partial^2 U \\partial x^2 \\frac \\partial^2 U \\partial y^2 Using a finite difference approximation dU a dt \\frac U i 1, j 2U i,j U i 1,j dx^2 \\frac U i, j 1 2U i,j U i,j 1 dy^2 note The code below implements the harness using OffsetArrays and wrap around boundary conditions to make your life easier. \"\"\" kernel function diffuse dU, Const U , a, dt, dx, dy Use index Global, NTuple to get i, j , then write the finite difference update into dU i, j end function diffuse U, a, dt, dx, dy dU zero U diffuse get backend U dU, U, a, dt, dx, dy ndrange N,N U . dU update boundary condition wrap around U 0, . U N, U N 1, . U 1, U , 0 . U , N U , N 1 . U , 0 U end let xs 0 N 1 ys 0 N 1 domain OffsetArray KernelAbstractions.zeros CPU , Float32, N 2, N 2 , xs, ys parent domain 16 32, 16 32 . 5.0f0 expected sum sum parent domain diffuse domain, a, dt, dx, dy peak maximum parent domain actual sum sum parent domain if peak ≥ 5.0f0 keep working md\"After one step the peak value should decrease — check your finite difference stencil.\" elseif actual sum ≈ expected sum keep working md\"Diffusion should conserve the total — check that you are writing into `dU i,j `, not `U i,j `.\" else correct end end answer box hint md\"\"\" ```julia kernel function diffuse dU, Const U , a, dt, dx, dy i, j index Global, NTuple dU i, j a dt U i 1, j 2 U i, j U i 1, j dx^2 U i, j 1 2 U i, j U i, j 1 dy^2 end ``` \"\"\" let xs 0 N 1 ys 0 N 1 domain OffsetArray KernelAbstractions.zeros backend, Float32, N 2, N 2 , xs, ys TODO Split out into initalize function parent domain 16 32, 16 32 . 5 fig, ax, hm heatmap xs, ys, Array parent domain Makie.Record fig, 1 250 do i diffuse domain, a, dt, dx, dy update data autolimits ax update limits end end "},{"url":"exercises/exercise_8_optimization/","title":"Optimization of parameters","description":"Use ForwardDiff to compute gradients and fit model parameters with Optim.jl","tags":["module1","track_ad","exercises"],"text":" A Pluto.jl notebook v1.0.1 frontmatter order \"4.1\" exercise number \"8\" title \"Optimization of parameters\" tags \"module1\", \"track ad\", \"exercises\" layout \"layout.jlhtml\" license \"MIT\" description \"Use ForwardDiff to compute gradients and fit model parameters with Optim.jl\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils using PlutoTeachingTools, PlutoUI ChooseDisplayMode begin using CairoMakie set theme theme latexfonts fontsize 16, Lines linewidth 2, , markersize 16 end md\"\"\" Exercise Optimization of parameters \"\"\" using ForwardDiff md\"\"\" Example Rosenbrock function f x,y 1 x ^2 100 y x^2 ```julia rosenbrock x, y 1 x ^ 2 100 y x ^ 2 ^ 2 ``` \"\"\" rosenbrock x, y 1 x ^ 2 100 y x ^ 2 ^ 2 begin xs 6 0.04 6 ys 2 0.04 5 zs log10 rosenbrock x, y for x in xs, y in ys end surface xs, ys, zs, axis type Axis3, contourf xs, ys, zs using Optim let f x rosenbrock x... sol optimize f, 0.0, 0.0 show Optim.minimizer sol sol end ∂f ∂x f, x, y ForwardDiff.derivative x f x, y , x ∂f ∂y f, x, y ForwardDiff.derivative y f x, y , y contourf xs, ys, x,y ∂f ∂x rosenbrock, x, y 1 . xs, ys' contourf xs, ys, x,y ∂f ∂y rosenbrock, x, y . xs, ys' let f x rosenbrock x 1 , x 2 function g dx, x dx 1 ∂f ∂x rosenbrock, x 1 , x 2 dx 2 ∂f ∂y rosenbrock, x 1 , x 2 nothing end sol optimize f, g , 0.0, 0.0 show Optim.minimizer sol sol end md\"\"\" Exercise Fitting our parameters \"\"\" function m x, a, b, c, d return sin a x b cos c x d end xs1 0.0 0.01 2π ys1 m. xs1, 0.3, 1.2, 0.5, 0.7 Mean squared error function mse ŷ, y sum ŷ . y .^2 length y end sol2 let f coeffs mse ys1, m. xs1, coeffs... function g dc, c dc 1 ForwardDiff.derivative a f a, c 2 , c 3 , c 4 , c 1 dc 2 0 TODO dc 3 0 TODO dc 4 0 TODO nothing end c₀ 0.9456253412871252, 0.5060263144193908, 0.7313087724591087, 0.5401913661922946 TODO What happens with c₀ rand 2.0 0.1 2.0, 4 optimize f, g , c₀ end Optim.minimizer sol2 ys2 m. xs1, Optim.minimizer sol2 ... let fig Figure ax Axis fig 1,1 lines ax, xs1, ys1, label \"Target curve\" lines ax, xs1, ys2, label \"Your curve, mse mse ys1, ys2 \" axislegend ax fig end answer box hint md\"\"\" ```julia function g dc, c dc 1 ForwardDiff.derivative a f a, c 2 , c 3 , c 4 , c 1 dc 2 ForwardDiff.derivative a f c 1 , a, c 3 , c 4 , c 2 dc 3 ForwardDiff.derivative a f c 1 , c 2 , a, c 4 , c 3 dc 4 ForwardDiff.derivative a f c 1 , c 2 , c 3 , a , c 4 nothing end ``` \"\"\" danger md\"\"\" It is not gurantueed that the optimizer converges to the correct global minimum. Change the initialization of `c₀`. \"\"\" using LinearAlgebra PlutoUI.TableOfContents depth 4 let target 0.3, 1.2, 0.5, 0.7 result Optim.minimizer sol2 if norm result . target 0.1 keep working md\"The optimizer didn't converge to the right parameters. Check your gradient computation.\" else correct md\"The optimizer found the correct parameters \" end end "},{"url":"exercises/exercise_9_pitfalls/","title":"Pitfalls of AD","description":"Understand and fix common pitfalls in automatic differentiation: type constraints, mixing of types, and perturbation confusion","tags":["module1","track_ad","exercises"],"text":" A Pluto.jl notebook v1.0.1 frontmatter order \"4.2\" exercise number \"9\" title \"Pitfalls of AD\" tags \"module1\", \"track ad\", \"exercises\" layout \"layout.jlhtml\" license \"MIT\" description \"Understand and fix common pitfalls in automatic differentiation type constraints, mixing of types, and perturbation confusion\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils begin using PlutoTeachingTools, PlutoUI using PlutoUI Slider PlutoUI.TableOfContents depth 4 end ChooseDisplayMode import ForwardDiff using LinearAlgebra import Enzyme using HypertextLiteral function expected failure f try f Markdown.MD Markdown.Admonition \"warning\", \"Expected failure\", Markdown.Paragraph \"The code was expected to fail, but it evaluated successfully.\" catch e str sprint do io showerror IOContext io, color true , e end htl \"\"\" div jlerror div class \"error header\" secret h1 Expected error message secret h1 div header style 'overflow auto ' embed display Text str header jlerror div \"\"\" end end function derivative enzyme f, x Enzyme.autodiff Enzyme.Forward, f, Enzyme.Duplicated x, one x | only end md\"\"\" Exercise Pitfalls of AD Automatic differentiation is powerful, but it can fail in subtle ways. In this exercise you will identify and fix three common pitfalls. \"\"\" md\"\"\" Part 1 — Type constraints \"\"\" f x Float64 x^2 expected failure do ForwardDiff.derivative f, 2.0 end md\"\"\" note ForwardDiff fails but Enzyme cell below succeeds. \"\"\" derivative enzyme f, 2.0 question box md\"\"\" Implement `f fixed` without a type constraint so that `ForwardDiff.derivative f fixed, 2.0 ` also returns `4.0`. \"\"\" TODO Implement f fixed let if isdefined f fixed func not defined f fixed else try result ForwardDiff.derivative f fixed, 2.0 if result ≈ 4.0 correct else keep working md\"`f fixed' 2.0 ` should be `4.0`, got result .\" end catch e keep working md\"Your `f fixed` still causes an error with ForwardDiff.\" end end end answer box hint md\"\"\" ```julia f fixed x x^2 ``` \"\"\" tip md\"\"\" ForwardDiff works by operator overloading it wraps the input in a `Dual Float64 ` type and relies on Julia's dispatch to propagate derivatives. A ` Float64` annotation rejects the `Dual` number — hence the error. Enzyme works at the compiler IR level it differentiates the compiled machine code directly, so source level type annotations are irrelevant to how derivatives are propagated. \"\"\" md\"\"\" Part 2 — Mixing of types \"\"\" function g x A zeros 2, 2 A 1,1 x A 2,2 x det A end expected failure do ForwardDiff.derivative g, 2.0 end md\"\"\" note ForwardDiff fails but Enzyme cell below succeeds. \"\"\" derivative enzyme g, 2.0 question box md\"\"\" Implement `g fixed` using `zeros typeof x , 2, 2 ` so that `ForwardDiff.derivative g fixed, 2.0 ` also returns `4.0`. \"\"\" TODO Implement g fixed let if isdefined g fixed func not defined g fixed else try result ForwardDiff.derivative g fixed, 2.0 if result ≈ 4.0 correct else keep working md\"`g fixed' 2.0 ` should be `4.0`, got result .\" end catch e keep working md\"Your `g fixed` still causes an error with ForwardDiff.\" end end end answer box hint md\"\"\" ```julia function g fixed x A zeros typeof x , 2, 2 A 1,1 x A 2,2 x det A end ``` \"\"\" md\"\"\" Part 3 Higher order derivatives A somewhat natural question is How would we obtain higher order derivatives . Interstingly we can define higher order derivatives as a recursive application of AD. \"\"\" function nth derivative forwarddiff f F, x, Val N where F, N if N 0 return f x else return ForwardDiff.derivative y nth derivative forwarddiff f, y, Val N 1 , x end end nth derivative forwarddiff sin, 1.0, Val 1 cos 1.0 nth derivative forwarddiff sin, 1.0, Val 2 sin 1.0 nth derivative forwarddiff sin, 1.0, Val 3 cos 1.0 nth derivative forwarddiff sin, 1.0, Val 4 sin 1.0 md\"\"\" Simple AD system Recall our small AD system from the lecture. \"\"\" begin struct Dual T Number Number value T deriv T end Dual x Real, y Real Dual promote type typeof x , typeof y promote x, y ... Base.convert Type Dual T , x Real where T Real Dual x, zero T Base.promote rule Type Dual T , Type Real where T Real Dual T Base. x Dual, y Dual Dual x.value y.value, x.deriv y.deriv Base. x Dual, y Dual Dual x.value y.value, x.deriv y.deriv Base. x Dual, y Dual Dual x.value y.value, x.value y.deriv x.deriv y.value Base. x Dual, y Dual Dual x.value y.value, x.deriv y.value x.value y.deriv y.value^2 Base.sin x Dual Dual sin x.value , cos x.value x.deriv Base.cos x Dual Dual cos x.value , sin x.value x.deriv Base.log x Dual Dual log x.value , x.deriv x.value Base.exp x Dual Dual exp x.value , exp x.value x.deriv NEW Base.one x Dual Dual one x.value , zero x.deriv Base. x Dual Dual x.value, x.deriv end begin derivative f, x f Dual x, one x .deriv derivative f x derivative f, x end function nth derivative f F, x, Val N where F, N if N 0 return f x else return derivative y nth derivative f, y, Val N 1 , x end end nth derivative sin, 1.0, Val 0 sin 1.0 nth derivative sin, 1.0, Val 1 cos 1.0 nth derivative sin, 1.0, Val 2 sin 1.0 nth derivative sin, 1.0, Val 3 cos 1.0 md\"\"\" Exercise Derivation \"\"\" md\"\"\" Thinking about this in the language of dual numbers. D f, x f x ϵ .deriv \"\"\" md\"\"\" We can evaluate D symbolically using substitution replace x with x \\epsilon , apply the function, then read off the dual part. Example ```julia D x x^2, 3 3 ϵ ^2 .deriv 9 6ϵ ϵ^2 .deriv Remember ϵ^2 0 6 ``` \"\"\" question box md\"\"\" Using the definitions D f, x f x \\epsilon .\\text deriv \\sin a b\\epsilon \\sin a \\cos a \\,b\\epsilon \\cos a b\\epsilon \\cos a \\sin a \\,b\\epsilon Evaluate step by step D \\sin, x D \\cos, x D x \\mapsto D \\sin, x , x \"\"\" md\"\"\" ```julia D sin, x TODO ``` \"\"\" answer box hint md\"\"\" ```julia D sin, x sin x ϵ .deriv Apply rule sin x cos x ϵ .deriv cos x ``` \"\"\" md\"\"\" ```julia D cos, x TODO ``` \"\"\" answer box hint md\"\"\" ```julia D cos, x cos x ϵ .deriv Apply rule cos x sin x ϵ .deriv sin x ``` \"\"\" md\"\"\" ```julia D x D sin, x , x TODO ``` \"\"\" answer box hint md\"\"\" ```julia D x D sin, x , x substitute D sin,x ϵ .deriv sin x ϵ ϵ .deriv .deriv apply rule for sin sin x ϵ cos x ϵ ϵ .deriv .deriv apply rule for sin sin x cos x ϵ cos x ϵ ϵ .deriv .deriv apply rule for cos sin x cos x ϵ cos x ϵ sin x ϵ^2 .deriv .deriv apply .deriv 2cos x sin x ϵ .deriv apply .deriv sin x ``` \"\"\" md\"\"\" Exercise Perturbation confusion \"\"\" md\"\"\" If we have derivatives on different \"levels\" interact with each other we have to be careful. \\frac d dx x \\frac d dy x y or ``` f x, y D x x D y x y, y , x ``` \"\"\" md\"\"\" Manually differentiating \\frac d dx x \\frac d dy x y \\frac d dy x y 1 \\frac d dx x 1 1 \"\"\" md\"\"\" warning Yet when we use our `derivative` function, we get a wrong answer of 2 . \"\"\" let D derivative D x x D y x y, 3. , 5. end md\"\"\" ForwardDiff.jl and Enzyme.jl get this right \"\"\" let D ForwardDiff.derivative D x x D y x y, 3. , 5. end let D derivative enzyme D x x D y x y, 3.0 , 5.0 end question box md\"\"\" Evaluate the expression below step by step through substitution. Start by substituting x with 5 \\epsilon . D x \\mapsto x \\cdot D y \\mapsto x y,\\ 3 ,\\ 5 Why does the result differ from the correct answer of 1 ? \"\"\" md\"\"\" ```julia D x x D y x y, 3 , 5 substitute x with 5 ϵ TODO ``` \"\"\" answer box hint md\"\"\" ```julia D x x D y x y, 3 , 5 substitute x with 5 ϵ 5 ϵ D y 5 ϵ y, 3 .deriv substitute y with 3 ϵ 5 ϵ 5 ϵ 3 ϵ .deriv .deriv apply rule for 5 ϵ 8 2ϵ .deriv .deriv simplify 5 ϵ 2 .deriv extract inner deriv 10 2ϵ .deriv apply rule for 2 extract outer deriv ``` The inner D sees x as 5 \\epsilon not just 5 , so D y \\mapsto 5 \\epsilon y,\\ 3 5 \\epsilon 3 \\epsilon .\\text deriv 2 instead of 1 . The outer and inner \\epsilon tags interact — this is perturbation confusion . \"\"\" md\"\"\" We see an interaction between the inner and the outer derivative. \"\"\" md\"\"\" We fix this by introducing tagged \\epsilon \\epsilon 1^2 0 \\epsilon 2^2 0 \\epsilon 1 \\neq \\epsilon 2 \\neq \\epsilon 1\\epsilon 2 \\neq 0 But better just to use ForwardDiff or Enymze. \"\"\" "},{"url":"mod1_introduction/automatic_differentiation/","title":"Automatic Differentiation","description":"","tags":["module1","track_ad"],"text":" A Pluto.jl notebook v1.0.1 frontmatter chapter \"1\" section \"4\" order \"4\" title \"Automatic Differentiation\" date \"2026 05 06\" tags \"module1\", \"track ad\" layout \"layout.jlhtml\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils This Pluto notebook uses bind for interactivity. When running this notebook outside of Pluto, the following 'mock version' of bind gives bound variables a default value instead of an error . macro bind def, element format off return quote local iv try Base.loaded modules Base.PkgId Base.UUID \"6e696c72 6542 2067 7265 42206c756150\" , \"AbstractPlutoDingetjes\" .Bonds.initial value catch b missing end local el esc element global esc def Core.applicable Base.get, el ? Base.get el iv el el end format on end begin using PlutoUI, PlutoTeachingTools using PlutoUI Slider PlutoUI.TableOfContents depth 4 end ChooseDisplayMode begin using CairoMakie set theme theme latexfonts fontsize 16, Lines linewidth 2, , markersize 16 end md\"\"\" Introduction to AD with material from Hendrick Ranocha, Adrian Hill, and Alan Edelman \"\"\" md\"\"\" What is a derivative? From your calculus classes, you might recall that a function f \\mathbb R \\rightarrow \\mathbb R is differentiable at \\tilde x if there is a number f' \\tilde x such that \\lim h \\rightarrow 0 \\frac f \\tilde x h f \\tilde x h f' \\tilde x \\quad . This number f' \\tilde x is called the derivative of f at \\tilde x . Let's visualize this on a simple scalar function \"\"\" f x x^2 5 sin x 10 you can change this function bind x̂ Slider 5 0.2 5, default 1.5, show value true let fig Figure ax Axis fig 1,1 , xlabel L\"x\" Plot function xs range 5, 5, 50 ymin, ymax extrema f. xs ylims ax, ymin 5, ymax 5 lines ax, xs, f, label L\"Function f x \" Obtain the function f′ f′ derivative f Plot f′ x lines ax, xs, f′ label L\"Derivative f′ x \" Plot 1st order Taylor series approximation taylor approx x f x̂ f′ x̂ x x̂ f x ≈ f x̃ f′ x̃ x x̃ lines ax, xs, taylor approx label L\"Taylor approx. around \\tilde x \" Show point of linearization vlines ax, x̂ color grey, linestyle dash, label L\"\\tilde x \" axislegend ax, position ct fig end md\"\"\" Partial derivatives For a multi variable function like `f x, y ` we define \\frac \\partial \\partial x f as the rate of change in the x direction, and likewise \\frac \\partial \\partial y f as the rate of change in the y direction. \"\"\" f2 x, y x^2 y^3 5 sin x 5 cos y x 20 you can change this function bind x̂₁ Slider 5 0.2 5, default 1.5, show value true bind ŷ₁ Slider 5 0.2 5, default 1.5, show value true let fig Figure ax Axis3 fig 1,1 , xlabel L\"x\", ylabel L\"y\", zlabel L\"z\", title \"f x,y \" ax4 Axis3 fig 2,1 , xlabel L\"x\", ylabel L\"y\", zlabel L\"z\", title \"Taylor approx. around x̂ x̂₁, ŷ ŷ₁\" ax2 Axis fig 1, 2 , title \"Partial deriv. in x at ŷ ŷ₁\", xlabel L\"x\" ax3 Axis fig 2, 2 , title \"Partial deriv. in y at x̂\", xlabel L\"y\" Plot function xs range 5, 5, 50 ys range 5, 5, 50 zmin, zmax extrema f2. xs, ys' zlims ax, zmin 5, zmax 5 surface ax, xs, ys, f2 lines ax, xs, f, label L\"Function f x \" Obtain the partial derivative functions f′ f x′ derivative x f2 x, ŷ₁ f y′ derivative y f2 x̂₁, y Plot partial f′ x lines ax2, xs, f x′ label L\"Derivative f x′ x \" lines ax3, ys, f y′ label L\"Derivative f y′ y \" Plot 1st order Taylor series approximation taylor approx x, y f2 x̂₁, ŷ₁ f x′ x̂₁ x x̂₁ f y′ ŷ₁ y ŷ₁ surface ax4, xs, ys, taylor approx label L\"Taylor approx. around \\tilde x ,\\tilde y \" fig end md\"\"\" Motivating example In many areas of science we encounter models with unkown parameters Curve fitting Machine learning f x p \\cdots We are often interested in learning these parameters given a training dataset or real world observation. Below we study a model that is the combination of a `sin` and `cos` function with four parameters. \"\"\" function m x, a, b, c, d return sin a x b cos c x d end xs 0.0 0.01 2π function multi slider names, values return PlutoUI.combine do Child inputs md\"\"\" name Child name, Slider vals show value true, default first vals last vals 2 \"\"\" for name, vals in zip names, values md\"\"\" inputs \"\"\" end end bind coeffs multi slider \"a\", \"b\", \"c\", \"d\" , 1.0 0.1 2.0, 1.0 0.1 2.0, 1.0 0.1 2.0, 1.0 0.1 2.0, let fig Figure ax Axis fig 1,1 lines ax, xs, sin, label \"sin\" lines ax, xs, cos, label \"cos\" lines ax, xs, m. xs, coeffs... , label \"model\" axislegend ax fig end question box md\"\"\" Given an \"observed\" evaluation of `m` can we \"learn\" the values of `a`, `b`, `c`, `d`? \"\"\" ys m. xs, 0.3, 1.2, 0.5, 0.7 lines xs, ys md\"\"\" We could try some random values \"\"\" coeffs guess rand 2.0 0.1 2.0, 4 ys guess m. xs, coeffs guess... md\"\"\" We need to define a \"loss\" a function that measures how far away we are from our solution. The Mean Squared Error is a common choice. \"\"\" Mean squared error function mse ŷ, y sum ŷ . y .^2 length y end mse ys, ys guess let fig Figure ax Axis fig 1,1 , title \"MSE mse ys, ys guess \" lines ax, xs, ys lines ax, xs, ys guess fig end md\"\"\" So how do we improve our guess systematically? \"\"\" let neighborhood 0.3 0.1 0.3 fig Figure ax Axis fig 1,1 for offset in neighborhood ys m. xs, coeffs guess 1 offset, coeffs guess 2 end ... lines ax, xs, ys, label \"Offset a offset , MSE mse ys, ys \" end axislegend ax fig end tip md\"\"\" We want to change a parameter, such that we are improving. Randomly trying certainly would get us \"somewhere\" Monte carlo methods Evolutionary algorithms But what we are after is the \"rate of change\" in the error given for a parameter or all parameters . \"\"\" let fig Figure ax Axis fig 1,1 , xlabel L\"a\" Plot function as range 2, 2, 100 f a mse ys, m. xs, a, coeffs guess 2 end ... ymin, ymax extrema f. as ylims ax, ymin 5, ymax 5 lines ax, as, f, label L\"Function f mse ys, m x a, b, c, d \" Obtain the function f′ f′ derivative f Plot f′ x lines ax, as, f′ label L\"Derivative f′ a \" â coeffs guess 1 Plot 1st order Taylor series approximation taylor approx a f â f′ â a â f x ≈ f x̃ f′ x̃ x x̃ lines ax, as, taylor approx label L\"Taylor approx. around \\tilde a \" Show point of linearization vlines ax, â color grey, linestyle dash, label L\"\\tilde a \" axislegend ax, position ct fig end tip md\"\"\" Today we are focusing on \"forward mode\" automatic differentiation. This means that we conceptualize calculating derivatives by applying an infinitesimal perturbation to an argument of a function. As we will see for a multi variable problem like ours this means we need to calculate the partial derivative for each parameter \"seperatly\". In machine learning one often encounters \"reverse mode\" automatic differentiation, which conceptually applies a perturbation to the \"return value\" of a function. We will revisit this in the Lecture \"Automatic Differentiation and Machine Learning\" \"\"\" md\"\"\" Finite differences There are several ways to compute a function and its derivative on a computer. If you use a computer algebra system CAS , you can compute derivatives analytically, e.g., using Wolfram Alpha https www.wolframalpha.com . Another option you should have seen in the introduction to numerical analysis are finite differences. Since the derivative is defined as f' x \\lim h \\to 0 \\frac f x h f x h , it makes sense to use the forward difference \\frac f x h f x h \\approx f' x and approximate the limit by taking a small h 0 . However, this leads to round off error since we typically represent real numbers via floating point numbers with fixed precision . \"\"\" nextfloat 1.0 1.0 eps 1.0 eps 1.0f0 nextfloat 1.234e5 1.234e5 eps 1.234e5 md\"\"\" Thus, there will be two regimes if h is too big, the error of the limit approximation dominates if h is too small, the floating point error dominates We illustrate this for different functions f at x 1 . We use different types of floating point numbers and compute the error of the finite difference approximation. \"\"\" using DoubleFloats bind f diff Select sin \"f x sin x \", cos \"f x cos x \", exp \"f x exp x \", x sin 100 x \"f x sin 100 x \", x sin x 100 \"f x sin x 100 \", bind FloatType Select Float32, Float64, Double64 default Float64 let fig Figure ax Axis fig 1, 1 xlabel L\"Step size h \", ylabel \"Error of the forward differences\", xscale log10, yscale log10 f f diff x one FloatType f′x derivative f, Float64 x h FloatType. 10.0 .^ range 20, 0, length 500 fd error h max abs f x h f x h f′x , eps x 100 lines ax, h, fd error. h label \"\" h def sqrt eps x scatter ax, h def , fd error h def color gray text ax, \"sqrt eps x \" position 5 h def, fd error h def , space data fig end md\"\"\" Next, we use the central difference \\frac f x h f x h 2 h \\approx f' x . \"\"\" let fig Figure ax Axis fig 1, 1 xlabel L\"Step size h \", ylabel \"Error of the central differences\", xscale log10, yscale log10 f f diff x one FloatType f′x, derivative f, Float64 x h FloatType. 10.0 .^ range 20, 0, length 500 fd error h max abs f x h f x h 2 h f′x , eps x 100 lines ax, h, fd error. h label \"\" h def cbrt eps x scatter ax, h def , fd error h def color gray text ax, \"cbrt eps x \" position 5 h def, fd error h def , space data fig end md\"\"\" Forward mode AD for scalars There is a well know proverb Differentiation is mechanics, integration is art Luckily, we are just interested in differentiation for now. Thus, all we need to do is to implement the basic rules of calculus like the product rule and the chain rule. Before doing that, let's consider an example. \"\"\" g x log x^2 exp sin x md\"\"\"We can compute the derivative by hand using the chain rule.\"\"\" g′ x 1 x^2 exp sin x 2 x exp sin x cos x md\"We can think of the function as a kind of computational graph obtained by dividing it into steps.\" function g graph x c1 x^2 c2 sin x c3 exp c2 c4 c1 c3 c5 log c4 return c5 end g 1.0 ≈ g graph 1.0 md\"To compute the derivative, we have to apply the chain rule multiple times.\" function g graph derivative x c1 x^2 c1 ε 2 x c2 sin x c2 ε cos x c3 exp c2 c3 ε exp c2 c2 ε c4 c1 c3 c4 ε c1 ε c3 ε c5 log c4 c5 ε c4 ε c4 return c5, c5 ε end g graph derivative 1.0 g 1.0 , g′ 1.0 let x 1.0, h sqrt eps g x h g x h end md\"\"\" We would like to automate this To do so, we introduce so called dual numbers https en.wikipedia.org wiki Dual number . They carry both a `value` and derivative called `deriv`, the ε part above . Formally, a dual number can be written as x \\varepsilon y, \\qquad x, y \\in \\mathbb R , quite similar to a complex number z x \\mathrm i y, \\qquad x, y \\in \\mathbb R . However, the new basis element \\varepsilon satisfies \\varepsilon^2 0 instead of \\mathrm i ^2 1 . Thus, the dual number have the algebraic structure of an algebra instead of a field like the complex numbers \\mathbb C . In our applications, the \\varepsilon part contains the derivative. Indeed, the rule \\varepsilon^2 0 yields a \\varepsilon b c \\varepsilon d a c \\varepsilon a d b c , which is just the product rule of calculus. You can code this as follows. \"\"\" begin struct Dual T Real Number value T deriv T end Dual x Real, y Real Dual promote x, y ... end md\"Now, we can create such dual numbers.\" Dual 5, 2.0 md\"Next, we need to implement the required interface methods for numbers.\" Base. x Dual, y Dual Dual x.value y.value, x.deriv y.deriv Dual 1, 2 Dual 2.0, 3 Base. x Dual, y Dual Dual x.value y.value, x.deriv y.deriv Dual 1, 2 Dual 2.0, 3 Base. x Dual, y Dual Dual x.value y.value, x.value y.deriv x.deriv y.value Dual 1, 2 Dual 2.0, 3 Base. x Dual, y Dual Dual x.value y.value, x.deriv y.value x.value y.deriv y.value^2 Dual 1, 2 Dual 2.0, 3 md\"We also need to tell Julia how to convert and promote our dual numbers.\" Base.convert Type Dual T , x Real where T Real Dual x, zero T Base.promote rule Type Dual T , Type Real where T Real Dual T Dual 1, 2 3.0 "},{"url":"mod1_introduction/julia_introduction/","title":"Introduction to Julia","description":"","tags":["module1","track_principles"],"text":" A Pluto.jl notebook v1.0.1 frontmatter chapter \"1\" section \"2\" order \"2\" title \"Introduction to Julia\" date \"2026 04 22\" tags \"module1\", \"track principles\" layout \"layout.jlhtml\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils This Pluto notebook uses bind for interactivity. When running this notebook outside of Pluto, the following 'mock version' of bind gives bound variables a default value instead of an error . macro bind def, element format off return quote local iv try Base.loaded modules Base.PkgId Base.UUID \"6e696c72 6542 2067 7265 42206c756150\" , \"AbstractPlutoDingetjes\" .Bonds.initial value catch b missing end local el esc element global esc def Core.applicable Base.get, el ? Base.get el iv el el end format on end using PlutoUI, PlutoTeachingTools html\" button onclick 'present ' Toggle presentation mode button \" PlutoUI.TableOfContents depth 4 md\"\"\" Introduction to Julia Julia is a high level, high performance, dynamic programming language designed for numerical and scientific computing. Key design goals Fast LLVM compiled, approaching C speeds General not just for numerics Dynamic interactive, like Python or MATLAB Open source MIT license, community driven \"\"\" md\"\"\" Why Julia? The classic \"two language problem\" in scientific computing | Language | Prototyping | Performance | | | | | | Python MATLAB R | ✅ Easy | ❌ Slow | | C C Fortran | ❌ Hard | ✅ Fast | | Julia | ✅ Easy | ✅ Fast | Julia solves this by being both productive and performant. \"\"\" md\"\"\" Basic Syntax \"\"\" md\"\"\" Arithmetic \"\"\" 1 2, 3 4, 10 3, 10 ÷ 3, 2^10, sqrt 2 md\"\"\" Strings and string interpolation \"\"\" let name \"Julia\" \"Hello, name 1 1 1 1 \" end md\"\"\" ```julia f x x 2 ``` sin 3 \"\"\" md\"\"\" String concatination \"\"\" \"ab\" \"cd\" \"b\"^4 md\"\"\" Variables Julia is dynamically typed — variables don't need type declarations. \"\"\" let x 42 y 3.14 z x y end md\"\"\" Unicode identifiers Julia supports full Unicode in variable names, making math notation natural. \"\"\" let α 0.01 β 0.99 γ α β end md\"\"\" Types \"\"\" md\"\"\" Every value in Julia has a type. You can inspect types with `typeof`. \"\"\" typeof 1 , typeof 1.0 , typeof \"hello\" , typeof true md\"\"\" Julia has a rich type hierarchy `Number` `Real` `Integer` `Int64`, `Int32`, … `AbstractFloat` `Float64`, `Float32`, … `Complex` Use `supertype` and `subtypes` to explore. \"\"\" supertype Float64 typeof 1.0 2.0im md\"\"\" Type annotations You can annotate types for clarity or to restrict dispatch \"\"\" \"\"\" double x Number Doubles all numbers \"\"\" function double x Number return 2 x end double 3 , double 3.14 doc double In the REPL use ?double md\"\"\" Functions \"\"\" md\"\"\" Functions are first class citizens in Julia. There are several ways to define them. \"\"\" function greet name return \"Hello, name \" end greet \"World\" Short form one liner square x x^2 square 5 Anonymous functions cube x x^3 if rand 0.5 other x x^2 else other x x^4 end other 3.0 cube 3 md\"\"\" Positional and Keyword Arguments Julia function arguments come in two flavours, separated by ` ` in the definition ``` function f pos1, pos2 default kw1, kw2 default ``` Before the ` ` — positional arguments matched by position, may have defaults. After the ` ` — keyword arguments always passed by name at the call site. \"\"\" "},{"url":"mod1_introduction/matrices/","title":"Matrix multiply","description":"","tags":["module1","track_performance","indepth"],"text":" A Pluto.jl notebook v1.0.1 frontmatter chapter \"1\" section \"5\" order \"5.1\" title \"Matrix multiply\" tags \"module1\", \"track performance\", \"indepth\" layout \"layout.jlhtml\" indepth number \"1\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils using PlutoUI, PlutoTeachingTools md\"\"\" Experiments with Memory Access and Matrices based on https github.com stevengj 18a01 blob master memory matrices.ipynb In many problems, especially problems accessing lots of data and doing relatively simple computations on each datum, the performance bottleneck is memory rather than computational speed. Because memory is arranged into a memory hierarchy of larger slower and smaller faster memories, it turns out that changing the order of memory access can have a huge impact on performance. \"\"\" md\"\"\" Benchmarking Matrix Multiplication One of the most basic building blocks of numerical linear algebra is the computation of matrix multiplication given an m \\times n matrix A and an n \\times p matrix B , compute the m \\times p matrix C AB . The entries of C are given by the exact formula C ik \\\\sum j 1 ^n A ij B jk but there are many ways to implement this computation. \\approx 2mnp flops floating point additions and multiplications are required, but they can re ordered arbitrarily, leading to \\sim mnp possible orderings. It turns out that the ordering of the operations in the matrix multiplication has a huge impact on performance, along with low level details of the inner loops. Basically, three factors make the implementation of efficient matrix multiplication highly nontrivial Caches https en.wikipedia.org wiki CPU cache the matrix accesses must be reordered to obtain temporal locality https en.wikipedia.org wiki Locality of reference and hence efficient memory cache usage. Registers https en.wikipedia.org wiki Processor register at the lowest level, the CPU registers form a kind of ideal cache. The innermost loops of the matrix multiplication need to be unrolled in order to load many values into registers and perform as much work with them as possible essentially a small submatrix multiplication . It turns out that a lot of tricks http cscads.rice.edu workshops july2007 autotune slides 07 Frigo.pdf are required to do this well. SIMD instructions https en.wikipedia.org wiki SIMD modern CPUs include special instructions that can perform several arithmetic operations at once e.g. 2, 4, or even 8 `Float64` operations , and to get the full benefit of these operations typically requires hand coding. As a consequence, there is a huge performance gap between the most obvious three loop matrix multiplication code and highly optimized code. This gap has become the central factor in the design of dense linear algebra libraries for several decades, especially the industry standard free open source the LAPACK https en.wikipedia.org wiki LAPACK library nearly all dense linear algebra is now organized around highly optimized BLAS https en.wikipedia.org wiki Basic Linear Algebra Subprograms libraries Because Julia benefits from fast compilers, we can illustrate this performance gap fairly with simple Julia code. In contrast, similar implementation in Matlab or Python would be orders of magnitude slower, and would demonstrate mostly language rather than the algorithmic effects. \"\"\" md\" Naive algorithm The following is the simplest, most obvious, matrix multiplication algorithm just three nested loops, implementing a dot product for each output C ik The only concessions we have made to performance concerns here are 1 we implement an in place matmul variant that operates on a pre existing C array, to avoid benchmarking the memory allocation deallocation and 2 we use the ` inbounds` macro to turn off array bounds checking in Julia for the inner loop. Together, these make less than a factor of two difference in speed. \" compute C A B, using naive matrix multiplication algorithm, with a pre allocated output array C. \" \" is a Julia convention for functions that modify their arguments. function matmul C, A, B m,n size A n,p size B size C m,p || error \"incorrect dimensions \", size C , \" ≠ p\" for i 1 m for k 1 p c zero eltype C for j 1 n inbounds c A i,j B j,k end inbounds C i,k c end end return C end a wrapper that allocates C of an appropriate type matmul A, B matmul Array promote type eltype A , eltype B undef, size A,1 , size B,2 , A, B using LinearAlgebra, Random begin correctness check A rand 5,6 B rand 6,7 norm matmul A,B A B end md\"\"\" Benchmarking naive `matmul` Here, we will benchmark our `naive` matmul implementation against the highly optimized OpenBLAS library that Julia uses for its built in matrix multiplication. Like `matmul `, we will call OpenBLAS with pre allocated output via `mul C, A, B ` instead of the simpler `A B`. By default, OpenBLAS uses multiple CPU cores, which gives it an \"unfair\" parallel speedup, but we can disable this for benchmarking purposes \"\"\" for benchmarking, use only single threaded BLAS begin BLAS.set num threads 1 blas threads nothing end function logspace start, stop, length exp10. range start stop, length end begin blas threads N round. Int, logspace 1, log10 3000 , 60 60 sizes from 10 to 3000 alternatively, use N 10 1000 to see some interesting patterns due to cache associativity etc. t Float64 t0 Float64 for n in N local A zeros n,n local B zeros n,n preallocate output C so that allocation is not included in timing C zeros n,n push t, elapsed matmul C,A,B push t0, elapsed LinearAlgebra.mul C,A,B println \"finished n n slowdown of \", t end t0 end end end md\"\"\" Now, we will plot the results. Since the number of flops is 2n^3 , we will plot 2n^3 t for time t in microseconds in order to plot the gigaflops rate billions of flops per second . If you naively think of a CPU as a box that performs floating point instructions at a fixed rate, with all other instructions being negligible a picture that may have been true circa 1985 , this would be a flat horizontal line independent of n , but we will see that reality is quite different. The OpenBLAS library gets an \"unfair\" factor of 8 speedup on typical modern Intel processors thanks to hand coded support for AVX 512 https en.wikipedia.org wiki Advanced Vector Extensions SIMD instructions, which perform 8 double precision floating point operations simultaneously. \"\"\" using CairoMakie begin blas threads peakflops 1e 9 end let fig Figure ax Axis fig 1, 1 , title \"\", xlabel \"matrix size n\", ylabel L\"\\text gigaflops~ \\frac 2n^3 t \", yscale Makie.pseudolog10, yticks 0, 1, 2, 3, 4, 5, 25, 50, 75 lines ax, N, 2N.^3 . t . 1e 9, label \"naive matmul\" lines ax, N, 2N.^3 . t0 . 1e 9, label \"BLAS matmul\" axislegend ax, position rc fig end md\"\"\" Cache oblivious matrix multiplication As a first step in the right direction, we'll implement a cache oblivious algorithm https en.wikipedia.org wiki Cache oblivious algorithm for matrix multiplication divide the matrices into four submatrices which are multiplied recursively until a sufficiently large base case is reached large enough to amortize the recursion overhead . This strategy erases the steep performance drop off that occurs for large n where the matrix goes out of cache, at the cost of ~25 lines of code rather than ~10 for the naive loops. It still doesn't match the OpenBLAS performance because it fails to address the other two problems unrolling and optimizing the base cases to optimize register utilization, and coding for SIMD instructions. \"\"\" function add matmul rec m,n,p, i0,j0,k0, C,A,B if m n p 64 base case naive matmult for sufficiently large matrices for i 1 m for k 1 p c zero eltype C for j 1 n inbounds c A i0 i,j0 j B j0 j,k0 k end inbounds C i0 i,k0 k c end end else m2 m ÷ 2 n2 n ÷ 2 p2 p ÷ 2 add matmul rec m2, n2, p2, i0, j0, k0, C, A, B add matmul rec m m2, n2, p2, i0 m2, j0, k0, C, A, B add matmul rec m2, n n2, p2, i0, j0 n2, k0, C, A, B add matmul rec m2, n2, p p2, i0, j0, k0 p2, C, A, B add matmul rec m m2, n n2, p2, i0 m2, j0 n2, k0, C, A, B add matmul rec m2, n n2, p p2, i0, j0 n2, k0 p2, C, A, B add matmul rec m m2, n2, p p2, i0 m2, j0, k0 p2, C, A, B add matmul rec m m2, n n2, p p2, i0 m2, j0 n2, k0 p2, C, A, B end return C end function matmul rec C, A, B m,n size A n,p size B size C m,p || error \"incorrect dimensions \", size C , \" ≠ p\" fill C, 0 return add matmul rec m,n,p, 0,0,0, C,A,B end matmul rec A, B matmul rec Array promote type eltype A , eltype B undef, size A,1 , size B,2 , A, B let correctness check A rand 50,60 B rand 60,70 norm matmul rec A,B A B end begin tco Float64 for n in N local A zeros n,n local B zeros n,n preallocate output C so that allocation is not included in timing C zeros n,n push tco, elapsed matmul rec C,A,B println \"finished n n slowdown of \", tco end t0 length tco end end let fig Figure ax Axis fig 1, 1 , title \"\", xlabel \"matrix size n\", ylabel L\"\\text gigaflops~ \\frac 2n^3 t \", yscale Makie.pseudolog10, yticks 0, 1, 2, 3, 4, 5, 25, 50, 75 lines ax, N, 2N.^3 . t . 1e 9, label \"naive matmul\" lines ax, N, 2N.^3 . t0 . 1e 9, label \"BLAS matmul\" lines ax, N, 2N.^3 . tco . 1e 9, label \"Cache Oblivious matmul\" axislegend ax, position rc fig end "},{"url":"mod1_introduction/parallelism/","title":"Parallelism","description":"","tags":["module1","track_parallel"],"text":" A Pluto.jl notebook v1.0.1 frontmatter chapter \"1\" section \"3\" order \"3\" title \"Parallelism\" date \"2026 04 28\" tags \"module1\", \"track parallel\" layout \"layout.jlhtml\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils using PlutoUI, PlutoTeachingTools using CairoMakie ChooseDisplayMode PlutoUI.TableOfContents depth 4 md\"\"\" Parallelism \"\"\" md\"\"\" Why do we need parallelism in the first place? \"\"\" md\"\"\" What has your processor done for you recently As programmers we have the mental model that a processor executes our program in linear order Processors are Out of order Superscalar Predictive Micro ops \"\"\" md\"\"\" An idealized processor RobustLocalResource \"https s3.amazonaws.com media p.slid.es uploads 779853 images 4327544 pasted from clipboard.png\", \"arch.png\" \"\"\" function my dot a, b acc zero eltype a inbounds for i in eachindex a,b acc a i b i end return acc end with terminal do code native debuginfo none my dot zeros 3 , zeros 3 end md\"\"\" End of Moore's Law Roughly Processors keep getting faster, but until when? \"\"\" RobustLocalResource \"https www.alleywatch.com wp content uploads 2023 01 screen shot 2017 03 03 at 1 59 21 pm.png\", \"moore.png\" md\"\"\" Notions of scalability \"\"\" md\"\"\" Speedup Strong scalability Given a program that can execute with some compute units N , we define t N as the total time the program takes. note Computers have many different ways of measure time, total time or wall clock time is the 'human' experienced time. \\text Speedup \\frac t 1 t N The time it takes for a program to execute with one compute unit, vs N compute units. \"\"\" md\"\"\" Amdahl's law Given a function f that consists of a parallel portion p and a serial portion s . In 1967, Amdahl pointed out that the speedup is limited by the fraction of the serial part of the software that is not amenable to parallelization. where s p 1 the serial and parallel fractions sum to 1 . \\text Speedup \\frac 1 s p N Amdahl’s law states that, for a fixed problem, the upper limit of speedup is determined by the serial fraction of the code. \"\"\" speedup s, p, N 1 s p N let fig, ax lines 1 16, N speedup 0, 1, N , label \"0%\" lines ax, 1 16, N speedup .01, .99, N , label \"1%\" lines ax, 1 16, N speedup .05, .95, N , label \"5%\" lines ax, 1 16, N speedup .1, .9, N , label \"10%\" lines ax, 1 16, N speedup .2, .8, N , label \"20%\" lines ax, 1 16, N speedup .4, .6, N , label \"40%\" lines ax, 1 16, N speedup .6, .4, N , label \"60%\" lines ax, 1 16, N speedup .8, .2, N , label \"80%\" lines ax, 1 16, N speedup 1, 0, N , label \"100%\" fig 1, 2 Legend fig, ax, \"Serial fraction\", framevisible false fig end md\"\"\" note For strong scalability the problem size stays constant, and we only vary the amount of compute available. \"\"\" md\"\"\" Weak Scaling Efficiency Given a program that can do N work with N compute units , we define t N as the total time the program takes. \\text Efficiency \\frac t 1 t N note The definition of t N includes the amount of work being scaled up \"\"\" RobustLocalResource \"https raw.githubusercontent.com eth cscs ImplicitGlobalGrid.jl master docs src assets images fig parEff HM3D Julia CUDA all Daint extrapol.png\", \"parallel efficiency.png\" md\"\"\" Node level This notebook uses Threads.nthreads threads \"\"\" with terminal do Sys.cpu summary end using Hwloc with terminal do Hwloc.topology end md\"\"\" note \"Hyper threading SMT \" Modern CPUs expose more logical cores than physical cores via Simultaneous Multi Threading SMT , marketed as Hyper Threading by Intel. Each physical core runs two hardware threads that share the core's execution units, caches, and memory bandwidth. Useful for memory bound workloads while one thread stalls waiting for data, the other can make progress. Little or no benefit for compute bound workloads both threads compete for the same functional units. Julia and the OS see the logical hyperthreaded count `Hwloc.topology ` above distinguishes physical vs. logical cores. For performance sensitive parallel code, consider pinning threads to physical cores only e.g. `JULIA EXCLUSIVE 1` or limiting ` threads` to the physical core count . \"\"\" md\"\"\" Shared memory parallelism Julia is task based `M N` threading, green threads `Channel`, locks and atomics ```julia function pfib n Int if n 1 return n end t Threads. spawn pfib n 2 return pfib n 1 fetch t Int end ``` ```julia using Base.Threads threads function prefix threads ⊕, y AbstractVector l length y k ceil Int, log2 l do reduce phase for j 1 k threads for i 2^j 2^j min l, 2^k inbounds y i y i 2^ j 1 ⊕ y i end end do expand phase for j k 1 1 1 threads for i 3 2^ j 1 2^j min l, 2^k inbounds y i y i 2^ j 1 ⊕ y i end end return y end A fill 1, 500 000 prefix threads , A ``` From Nash et al., 2021 . Basic Threading Examples in JuliaLang v1.3. JuliaCon Proceedings, 1 1 , 54, https doi.org 10.21105 jcon.00054 \"\"\" function myfun s 0.0 N 10000 for i in 1 N s det randn 3,3 end s N end function bench f, N 10 a zeros N Threads. threads for i in eachindex a a i f end a end using BenchmarkTools, LinearAlgebra function bench serial f, N 10 a zeros N for i in eachindex a a i f end a end time bench serial myfun, 1000 time bench myfun, 1000 benchmark bench myfun, 1000 samples 10 evals 3 question box md\"What are potential issues with `myfun`\" answer box md\"\"\" Lots of memory allocation in the hot loop How is `det` implemented? \\ ` which det zeros 3,3 ` `det lufact A ` `det` calls into BLAS \"\"\" md\"\"\" Composable parallelism \"\"\" question box md\"Who is in charge of parallelism?\" md\"\"\" Potential answers The user? The library? Are they all written in Julia? \"The system\" \"\"\" md\"\"\" As an example for linear algebra Julia uses OpenBLAS. OpenBLAS manages its own thread pool via `BLAS.set num threads`. We can run into contention issues when we use Julia's own parallelism using tasks and system libraries. When Julia launches N tasks and each task calls into OpenBLAS, OpenBLAS will in turn spin up its own M threads — resulting in N \\times M threads competing for the same physical cores. This over subscription leads to excessive context switching and can make a parallel program slower than its serial counterpart. One mitigation is to tell OpenBLAS to use a single thread when you are already parallelising at a higher level ```julia using LinearAlgebra BLAS.set num threads 1 ``` note When we work with small matrices, the overhead of calling BLAS can be avoided. `myfun improved` below uses `StaticArrays` for small, fixed size arrays. Since the compiler knows the size at compile time, it can inline and unroll the computation. \"\"\" using StaticArrays function myfun improved s 0.0 N 10000 for i in 1 N s det randn SMatrix 3,3 end s N end benchmark bench myfun improved, 1000 samples 10 evals 3 benchmark bench serial myfun improved, 1000 samples 10 evals 3 md\"\"\" Accelerated \"\"\" md\"\"\" tip \"Trying GPU code without local hardware\" Google Colab https colab.research.google.com now supports Julia natively. Change the runtime type to Julia Runtime → Change runtime type → Julia and enable a free GPU. No shell setup required. \"\"\" md\"\"\" A GPU has many \"lightweight\" threads \"\"\" md\"\"\" CPU die shot RobustLocalResource \"https s3.amazonaws.com media p.slid.es uploads 779853 images 4399827 800px coffee lake die quad core annotated .png\", \"quad core.png\" \"\"\" md\"\"\" GPU block diagram RobustLocalResource \"https devblogs.nvidia.com parallelforall wp content uploads 2016 04 gp100 block diagram 1 624x368.png\", \"p100.png\" \"\"\" md\"\"\" Bottlenecks RobustLocalResource \"https s3.amazonaws.com media p.slid.es uploads 779853 images 4399848 pasted from clipboard.png\", \"gpu system.png\" \"\"\" md\"\"\" Composable infrastructure Core GPUCompiler.jl Takes native Julia code and compiles it directly to GPUs GPUArrays.jl High level array based common functionality KernelAbstractions.jl Vendor agnostic kernel programming language Adapt.jl Translate complex structs across the host device boundary Vendor specific CUDA.jl AMDGPU.jl oneAPI.jl Metal.jl \"\"\" md\"\"\" Different layers of abstraction Vendor specific ```julia using CUDA function saxpy a,X,Y i blockIdx .x 1 blockDim .x threadIdx .x if i length Y inbounds Y i a X i Y i end return nothing end cuda threads 32 blocks cld length Y , 32 saxpy a, X, Y ``` KernelAbstractions ```julia using KernelAbstractions using CUDA kernel function saxpy a, Const X , Y I index Global inbounds Y I a X I Y I end saxpy CUDABackend a, X, Y, ndrange length Y ``` Array abstractions ```julia Y . a . X . Y ``` \"\"\" md\"\"\" How to use KernelAbstractions Use ` kernel function mykernel args... end` to write a GPU style program Instantiate kernel for a backend `kernel mykernel backend ` Backends come from Vendor specific libraries `KA.allocate backend, ... ` to obtain memory Launch kernel `kernel args..., ndrange ... ` while specifying the grid to execute over. \"\"\" TwoColumn md\"\"\" ```julia function vadd a, b, c for i in eachindex c c i a i b i end end a rand Float32, N b rand Float32, N c similar a vadd a, b, c ``` \"\"\", md\"\"\" ```julia import KernelAbstractions as KA kernel function vadd a, b, c i index Global c i a i b i end backend CUDABackend a KA.allocate backend, Float32, N b KA.allocate backend, Float32, N c similar a vadd kernel vadd backend vadd kernel a, b, c ndrange size c ``` \"\"\" md\"\"\" note GPU execution is asynchronous We will discuss the details in the later GPU lecture. When benchmarking you need to synchronize the device ```julia benchmark begin vadd kernel a, b, c ndrange size c KA.synchronize backend end ``` Otherwise you are only measuring the launch of the kernel. \"\"\" md\"\"\" High level array based programming Julia and GPUArrays.jl provide support for an efficient GPU programming environment build around array abstractions and higher order functions. Vocabulary of operations `map`, `broadcast`, `scan`, `reduce`, ... Map naturally onto GPU execution models Compiled to efficient code multiple dispatch, specialization Write generic, reusable applications BLAS matrix multiply, ... , and other libraries like FFT Array operators using multiple dispatch a design methodology for array implementations in dynamic languages doi 10.1145 2627373.2627383 Rapid software prototyping for heterogeneous and distributed platforms doi 10.1016 j.advengsoft.2019.02.002 \"\"\" md\"\"\" Array types where memory resides and how code is executed. | | | | | | | `A Matrix Float64 undef, 64, 32 ` | CPU | | `A CuMatrix Float64 undef, 64, 32 ` | Nvidia GPU | | `A ROCMatrix Float64 undef, 64, 32 ` | AMD GPU | info Data movement is explicit. \"\"\" md\"\"\" What makes an application portable? 1. Can I run it on a different compute architecture 1. Different CPU architectures 2. We live in a mult GPU vendor world 2. Does it compute the same thing? 1. Can I develop on one platform and move to another later? 3. Does it achieve the same performance ? 4. Can I take advantage of platform specific capabilities? Productivity meets Performance Julia on A64FX doi 10.1109 CLUSTER51413.2022.00072 \"\"\" md\"\"\" Adapt.jl Adapt.jl https github.com JuliaGPU Adapt.jl is a lightweight dependency that you can use to convert complex structures from CPU to GPU. ```julia using Adapt adapt CuArray, Adjoint Array Adjoint CuArray ``` ```julia struct Model T Number, AT AbstractArray T data AT end Adapt.adapt structure to, x Model Model adapt to, x.data cpu Model rand 64, 64 using CUDA gpu adapt CuArray, cpu Model Float64, CuArray Float64, 2, CUDA.Mem.DeviceBuffer ... ``` \"\"\" md\"\"\" Multi Node Distributed \"\"\" md\"\"\" Explicit communication between processes using a library like `MPI.jl` \"\"\" md\"\"\" ```julia using MPI MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm size MPI.Comm size comm dst mod rank 1, size src mod rank 1, size N 4 send mesg Array Float64 undef, N recv mesg Array Float64 undef, N fill send mesg, Float64 rank rreq MPI.Irecv recv mesg, comm source src, tag src 32 sreq MPI.Isend send mesg, comm dest dst, tag rank 32 stats MPI.Waitall rreq, sreq MPI.Barrier comm ``` \"\"\" md\"\"\" note Weak scaling is often a more interesting measurement on clusters. \"\"\" "},{"url":"mod1_introduction/performance_engineering/","title":"Performance Engineering","description":"","tags":["module1","track_performance"],"text":" A Pluto.jl notebook v1.0.1 frontmatter chapter \"1\" section \"5\" order \"5\" title \"Performance Engineering\" date \"2026 05 13\" tags \"module1\", \"track performance\" layout \"layout.jlhtml\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils This Pluto notebook uses bind for interactivity. When running this notebook outside of Pluto, the following 'mock version' of bind gives bound variables a default value instead of an error . macro bind def, element format off return quote local iv try Base.loaded modules Base.PkgId Base.UUID \"6e696c72 6542 2067 7265 42206c756150\" , \"AbstractPlutoDingetjes\" .Bonds.initial value catch b missing end local el esc element global esc def Core.applicable Base.get, el ? Base.get el iv el el end format on end using PlutoUI, PlutoTeachingTools ChooseDisplayMode PlutoUI.TableOfContents depth 4 md\"\"\" Performance Engineering \"\"\" md\"\"\" Goals 1. Understanding the performance characteristics of your program What is the hot path Where is time being spent 2. Define metrics Benchmarks allow us to understand \"is a change good\" 3. How does my program get executed? Language of choice? Compilation Hardware architecture \"\"\" TwoColumn md\"\"\" Benchmarking Focusing on the \"hot loop\" Allows for comparison Different algorithms Different hardware \"\"\", md\"\"\" Profiling Analyse where time is being spent Many tools with different trade offs Different perspectives Language profiler System profiler \"\"\" md\"\"\" Benchmarking \"\"\" md\"\"\" note Premature optimization is the root of all evil & If you don't measure you won't improve \"\"\" md\"\"\" BenchmarkTools.jl Solid package that tries to eliminate common pitfalls in performance measurement. ` benchmark` macro that will repeatedly evaluate your code to gain enough samples Caveat You probably want to escape your input data \"\"\" using BenchmarkTools bind N PlutoUI.Slider 5 12 md\"\"\" N N \"\"\" data rand 10^N function sum X acc 0 for x in X acc x end acc end elapsed sum data Huh Is pluto lying to us belapsed sum data samples 10 evals 3 benchmark sum data samples 10 evals 3 md\"\"\" note We will talk about the Julia compiler in detail in a future lecture. One important thing to know, that Julia performs type inference based on the argument types, and quality of type inference determines performance characteristics. \"\"\" md\"\"\" ```julia function sum X Vector Float64 acc 0 Int64 for x in X acc x Float64 end acc Union Int64, Float64 end ``` \"\"\" with terminal do code warntype sum data end md\"\"\" Caveats of micro benchmarking BenchmarkTools tries to approximate a function execution a top level. It thus measures the cost of accessing global variables. Use the interpolation syntax ` ` to avoid that cost for very cheap functions. \"\"\" begin a 3.0 b 4.0 end benchmark sin a b benchmark sin a b benchmark sin a b md\"\"\" warning Did we get too fast? \"\"\" code typed do sin 3.0 4.0 end | only md\"\"\" Julia can constant fold expressions. \"\"\" benchmark sin Ref a Ref b md\"\"\" warning Be careful with benchmarks whose time doesn't change with an increase in complexity Likely that the compiler got clever and turned it into a constant time expression or constant folded it. \"\"\" function count N acc 0 for i in 1 N acc 1 end return acc end benchmark count 10 benchmark count 1000000 md\"\"\" warning How would you benchmark `sort `? \"\"\" let v rand Int, 1024 benchmark sort v end md\"\"\" `sort ` is mutating its input. Therefore we need to set `evals 1` and provide a `setup` to re initialize the data everytime \"\"\" benchmark sort v setup v rand Int, 1024 evals 1 md\"\"\" Sources of noise Computers are noisy systems. The operating system manages resources and distribute them to programs. 1. Heat The temperature of your processor influences the frequency it is targetting When benchmarking a function may be faster in the beginning and slower afterwards 2. Other programs The OS is splitting the CPU time into slices So a program may be descheduled 3. Input Output IO Disk Network access is variable The OS will also put you to sleep, when waiting for data 4. The CPU CPU are \"learning\" \"predictive\" systems. See https discourse.julialang.org t psa microbenchmarks remember branch history 17436 \"\"\" md\"\"\" Profiling \"\"\" md\"\"\" Most profilers we will use are stochastic profilers. They sample the running program at a fixed interval. Julia uses a default of `0.001s` and it also uses a fixed size buffer. See `Profile.init`. note Sampling artifacts can occur when profiling multi threaded applications. During a sample the thread we are sampling from is paused. If the thread is in a critical section this may introduce artifacts. \"\"\" Profile.init md\"\"\" Profiler https docs.julialang.org en latest manual profile ProfileView.jl https github.com timholy ProfileView.jl ProfileCanvas.jl https github.com pfitzseb ProfileCanvas.jl PProf.jl https github.com JuliaPerf PProf.jl \"\"\" using Profile using ProfileCanvas md\"\"\" The Julia profiler focuses on the execution of Julia code. This leads to two limitations 1. By default it does not show time spent in C functions 2. It does not measure time spent on \"external\" threads BLAS threads GC worker threads \"\"\" function profile test n for i 1 n A randn 100,100,20 m maximum A Am mapslices sum, A dims 2 B A , ,5 Bsort mapslices sort, B dims 1 b rand 100 C B. b end end profview profile test 1 run once to trigger compilation ignore this one profview profile test 10 md\"\"\" note The profiler shows all Julia worker threads. The time spent in `task done hook` is a worker thread idiling. \"\"\" function pfib n Int if n 1 return n end t Threads. spawn pfib n 2 return pfib n 1 fetch t Int end function profile pfib n, k for i 1 n pfib k end end profview profile pfib 1, 4 profview profile pfib 10, 16 md\"\"\" To gain insight into runtime behavior we can turn on C frames \"\"\" profview profile pfib 10, 16 C true md\"\"\" Runtime functions you might see `jl gc alloc` Memory allocations `jl gc collect` Garbage collection `jl safepoint wait gc` Garbage collection waiting for all threads to reach it. \"\"\" md\"\"\" Native profilers \"\"\" md\"\"\" System profilers allow you to gain even more insight into how your program is performing, but come with usability down sides. \"\"\" md\"\"\" VTune https github.com JuliaPerf IntelITT.jl?tab readme ov file running julia under vtune Perf https docs.julialang.org en v1 manual profile External Profiling NSight Systems https cuda.juliagpu.org stable development profiling External profilers `CUDA.jl` has it's own inbuilt ` profile` \"\"\" md\"\"\" Instrumentation It can be hard to correlate profiles with our programs, instrumentation makes it easier to define semantically important portions. TimerOutputs https github.com KristofferC TimerOutputs.jl NVTX https github.com JuliaGPU NVTX.jl Tracy https github.com topolarity Tracy.jl IntelITT.jl https github.com JuliaPerf IntelITT.jl \"\"\" using TimerOutputs using Base.Threads threads const to TimerOutput timeit to function prefix threads ⊕, y AbstractVector l length y k ceil Int, log2 l do reduce phase timeit to \"reduce\" for j 1 k threads for i 2^j 2^j min l, 2^k inbounds y i y i 2^ j 1 ⊕ y i end end do expand phase timeit to \"expand\" for j k 1 1 1 threads for i 3 2^ j 1 2^j min l, 2^k inbounds y i y i 2^ j 1 ⊕ y i end end return y end let TimerOutputs.reset timer to for i in 1 10 A fill 1, 500 000 prefix threads , A end to end md\"\"\" Allocation profiler The allocation profiler will collect backtraces at allocation sites with a default `sample rate 0.0001` So about 1 10 000 allocations . PProf.jl https github.com JuliaPerf PProf.jl also has a callgraph view instead of just the \"icile\" view. \"\"\" profview allocs profile test 10 profview allocs profile test 10 sample rate 1.0 md\"\"\" Where does time go? \"\"\" TwoColumn md\"\"\" Your code Arithmetic operations Special functions Memory accesses Memory layout Type instabilities Bad algorithm choices Lack of parallelism \"\"\", md\"\"\" Runtime Memory allocation Garbage collection finding unused memory Waiting for the OS Network Filesystem ... Concurrency Lock conflicts Function dispatch Compiling code \"\"\" md\"\"\" Performance annotation in Julia https docs.julialang.org en v1 manual performance tips Julia does bounds checking by default `ones 10 11 ` is an error ` inbounds` Turns of bounds checking locally ` fastmath` Turns of strict IEEE 754 locally – be very careful this might not do what you want ` simd` and ` simd ivdep` provide stronger gurantuees to encourage LLVM to use SIMD operations \"\"\" function my sum X acc zero eltype X for x in X acc x end return acc end benchmark my sum w setup w rand 2048 function my sum2 X acc zero eltype X simd for x in X acc x end return acc end benchmark my sum2 w setup w rand 2048 md\"\"\" note ` simd` allows for re ordering of reduction operations. \"\"\" using CairoMakie md\"\"\" Example Matrix addition Matrix addition is an interesting case because it has no data re use, so there is no possible temporal locality, but depending on what order you use for the loops and how matrices are stored in memory, you may or may not get spatial locality that takes advantage of cache lines . Here let's implement matrix addition in two different ways. We'll use a pre allocated output array so that our benchmark does not include the time for memory allocation \"\"\" function matadd1 C, A, B size C size A size B || throw DimensionMismatch m,n size A for i 1 m simd for j 1 n inbounds C i,j A i,j B i,j end end return C end matadd1 A, B matadd1 similar A, promote type eltype A , eltype B , A, B function matadd2 C, A, B size C size A size B || throw DimensionMismatch m,n size A for j 1 n simd for i 1 m inbounds C i,j A i,j B i,j end end return C end matadd2 A, B matadd2 similar A, promote type eltype A , eltype B , A, B let A rand 5,6 B rand 5,6 A B ≈ matadd1 A,B ≈ matadd2 A,B end function logspace start, stop, length exp10. range start stop, length end begin Na round. Int, logspace 1, log10 3000 , 60 60 sizes from 10 to 3000 alternatively, use N 10 1000 to see some interesting patterns due to cache associativity etc. t1 Float64 t2 Float64 for n in Na local A zeros n,n local B zeros n,n preallocate output C so that allocation is not included in timing C zeros n,n matadd1 C,A,B add once just to make sure we are in cache if A and B are small push t1, elapsed matadd1 C,A,B push t2, elapsed matadd2 C,A,B println \"finished n n ratio t1 t2 of \", t1 end t2 end end end let fig Figure ax Axis fig 1, 1 , title \"\", xlabel \"matrix size n\", ylabel L\"\\text gigaflops~ \\frac n^2 t \" lines ax, Na, Na.^2 . t1 . 1e 9, label \"by row\" lines ax, Na, Na.^2 . t2 . 1e 9, label \"by column\" axislegend ax, position rt fig end let fig Figure ax Axis fig 1, 1 , title \"Ratio of matrix addition algorithms\", xlabel \"matrix size n\", ylabel \"by row time by column time\" lines ax, Na, t1 . t2 fig end md\"\"\" note The reason for this is that Julia stores matrices with consecutive columns , which is known as column major storage format. \"\"\" let A zeros Int, 10, 3 for i in 1 length A A i i end A end md\"\"\" Memory layout of Objects \"\"\" using About md\"\"\" Julia has two types of struct types. 1. Immutable 2. Mutable Immutable datatypes have fields that can't be mutated and they do not have object identity . Think numbers, and similar objects. Mutable datatypes can be updated in place and they posses object identity. \"\"\" struct MyImmutable a Float64 end mutable struct MyMutable a Float64 end let x MyImmutable 1.0 y MyImmutable 1.0 x y end let x MyMutable 1.0 y MyMutable 1.0 x y end with terminal do about Float64 ℯ end with terminal do about 1.0 0.0im end with terminal do about MyImmutable end with terminal do about MyMutable end begin x1 fill MyImmutable 0.0 , 10 x1 1 MyImmutable 1.0 x1 end begin x2 fill MyMutable 0.0 , 10 x2 1 .a 1.0 x2 end md\"\"\" warning `fill` with a mutable object will lead to aliasing. \"\"\" begin x3 MyMutable 0.0 for in 1 10 x3 1 .a 1.0 x3 end with terminal do about 1.0, 2.0 end with terminal do about 1.0 0.0im, 2.0 1.0im end with terminal do about MyMutable 0.0 , MyMutable 1.0 end md\"\"\" info Mutable objects are stored in fields and arrays as references pointers . Immutable objects may be stored inline . \"\"\" "},{"url":"mod1_introduction/research_software_engineering/","title":"Research Software Engineering","description":"","tags":["module1","track_principles"],"text":" A Pluto.jl notebook v1.0.1 frontmatter chapter \"1\" section \"1\" order \"1\" title \"Research Software Engineering\" date \"2026 04 15\" tags \"module1\", \"track principles\" layout \"layout.jlhtml\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils This Pluto notebook uses bind for interactivity. When running this notebook outside of Pluto, the following 'mock version' of bind gives bound variables a default value instead of an error . macro bind def, element format off return quote local iv try Base.loaded modules Base.PkgId Base.UUID \"6e696c72 6542 2067 7265 42206c756150\" , \"AbstractPlutoDingetjes\" .Bonds.initial value catch b missing end local el esc element global esc def Core.applicable Base.get, el ? Base.get el iv el el end format on end html\" button onclick 'present ' Toggle presentation mode button \" using PlutoUI, PlutoTeachingTools PlutoUI.TableOfContents depth 4 md\"\"\" Who am I? B.Sc in Cognitive Science University of Osnabrueck 2011 2014 Applied ML research Okinawa Institute of Technology, Japan 2015 2017 M.Sc & PhD in Computer Science Massachussetts Institute of Technology, USA 2017 2024 Research Software Engineer PostDoc University of Mainz University of Augsburg Working on GPU computing in Julia since 2014 – First Julia version 0.3 dev I work on the Julia compiler and runtime. My goal is to make HPC and GPU based programming easier and more accessible, I work with science teams to help them maximise their usage of Julia. \"\"\" md\"\"\" What is reasearch software engineering Software development is an essential, integral part of research activity. Research software increasingly supports the acquisition, processing and analysis of empirical data, but also the modeling and the simulation of complex processes. Thus, software has a significant influence on the quality of research results. Resource \"https www.software.ac.uk sites default files images content BetterSoftwareBetterResearchImage.jpg\" Data analysis Classical data science statistics Visualization Data generation Modelling Why does it matter Code is science Writing programms is akin to writing paper a way of scientific communication Academic recognition of this is still a challenge Science is built on \"trust but verify\" Reproduction of code is a challenge Too often \"works only on one system\" Building scientific software together Leads to building communities Reuse \"\"\" md\"\"\" Community https researchsoftware.org https society rse.org https de rse.org https fg rse.gi.de Conferences JuliaCon 2026 Here in Mainz https juliacon.org 2026 RSECon25 https rsecon25.society rse.org https de rse.org de events \"\"\" md\"\"\" Different kind of projects Small 1 user developer 1 developer developer user Most likely what you will build during this class or for your thesis Reproducibility for papers our future selves Medium Small number of developers Equal number of users Slightly more users than developers Most research projects Large Small number of developers Many users Multiple organization invested Hopefully self sustaining note In the end we have a choice between letting a small project grow if it is useful to not just us , or Maintainership Bus factor Crudly How many people could be hit by a bus, before the project becomes umaintainble Often a one to many relationship It can be very easy to become overwhelmed Many things vying for attention Slack Github Issues Discourse Going from small to medium Giving up some measure of control Communities User community Developer community Open Source Necessary for Open Science All to often people are afraid of releasing their code It's not ready yet1 It's ugly I don't want someone to scoop me. Often Release the source code that was used to produce results in a paper But It is worthwhile to think about converting something from a \"project\" to \"a package\" Encourage reuse Separate concerns Then release a project that uses a package to do something The role of Git Github Git is a version control system Keeps track of previous state of the project Don't email tarballs \"Branches\" can be used to keep track of concurrent developments GitHub is a collaboration platform Supports \"pull request\" Keeping track of \"issues\" Software releases Continous integration Documentation hosting Very much building a community \"\"\" md\"\"\" What are you interested in? \"\"\" \"\"\" 1. Why are you in this class? 2. What do you want to write code for? 3. What do you want to learn about? \"\"\" | question box md\"\"\" The use of LLMs in programming \"\"\" md\"\"\" LLMs are tools that are more and more relevant for programming, but the brings some issues with them. 1. Do you understand what your LLM is doing? Does it generate code you can read maintain stand behind? 2. LLM usage feels a bit like a slot machine Just one more prompt Prompt until it \"works\" 3. LLMs are sycophant programmed to please the user. Principled LLM usage can be helpful~ But you remain the author of the piece of code, you should review the output of your LLM. warning \"Responsible usage of LLMs\" Read through Using LLMs at Oxide https rfd.shared.oxide.computer rfd 0576 LLVM AI tool policy human in the loop https discourse.llvm.org t rfc llvm ai tool policy human in the loop 89159 note \"LLMs in the classroom\" You are here to learn something. Using an LLM to solve exercises or to write your project will stop you from deeply engaging with the material and learn the necessary skills. I can't stop you from using LLMs, but I will be grumpy if I start feeling like I am talking to your LLM through you... \"\"\" md\"\"\" Topics of this Course \"\"\" md\"\"\" Principles of research software engineering Best practices of research software engineering, how to develop scientific software in an reproducible, collaborative way. Sub topics 1. Reproducibility 2. Testing 3. Documentation 4. Accessibility note The goal is to enable you , to build the software you need for your research. \"\"\" md\"\"\" Parallelism Computers are becoming evermore parallel. Multi core Multi node Cluster Accelerated computing GPUs ML accelerators TPU Scientific software often has the characteristics that it can scale. 1. Strong scaling \\ More compute resources will enable a faster time to solution constant problem size 2. Weak scaling \\ Increasing the problem size and compute resources, provides a better solution in the same time note Writing parallel programs takes more consideration than writing a serial problem, but we must to solve problems efficiently Efficiency Time efficiency Power efficiency Cost efficiency \"\"\" md\"\"\" Performance Engineering \"\"\" md\"\"\" Automatic differentiation Why do we want gradients derivatives Derivatives compute the rate of change of a function’s output with respect to input s ```math f' x \\lim h\\rightarrow 0 \\frac f a h f a h ``` Used in many places Optimization problems Parameter estimation Uncertainity quantification Machine learning note Automatic differentiation is the art of taking derivatives not of mathematical functions, but of computer programs that implement mathematical functions. It is a corner stone of both scientfici computing and machine learning. \"\"\" md\"\"\" What's Julia? 🟢 🟣 🔴 Julia is a modern, dynamic, general purpose, compiled programming language. It's interactive \"like Python\" , can be used in a REPL or notebooks, like Jupyter it's the \"Ju\" or Pluto this one🎈 . Julia has a runtime which includes a just in time JIT compiler and a garbage collector GC , for automatic memory management. Julia is mainly used for technical computing, and addresses a gap in the programming language landscape for numerical computing. Main paradigm of Julia is multiple dispatch, what functions do depend on type and number of all arguments. \"\"\" md\"\"\" Why Julia? 😍 From \" My Target Audience https scientificcoder.com my target audience \" by Matthijs Cox Resource \"https cdn.hashnode.com res hashnode image upload v1681735971356 91b6e886 7ce1 41a3 9d9f 29b7b096e7f2.png\" Resource \"https cdn.hashnode.com res hashnode image upload v1681735992315 62fdd58f 4630 4120 8eb4 5238740543e8.png\" Explorable & Understandable Composability thanks to multiple dispatch ask me more about this at the end User defined types are as fast and compact as built ins Code that is close to the mathematics No need to switch languages for performance... ...but you can still call C like shared libraries with simple Foreign Function Interface FFI if you want to MIT licensed free and open source \"\"\" md\"\"\" What is the 2 language problem? You start out proto typing in one language high level, dynamic , but performance forces you to switch to a different one low level, static . For convinience use a scripting language Python, R, Matlab, ... but do all the hard stuff in a systems language C, C , Fortran Pragmatic for many applications, but has drawbacks aren't the hard parts exactly where you need an easier language creates a social barrier a wall between users and developers \"sandwich problem\" layering of system & user code is expensive prohibits full stack optimisations Julia for RSEs? Tearing down barriers of collaboration Fostering collaboration Low barrier from package user to package developer One codebase to rule them all Understandable and explorable performance Julia now Recently released v1.9, comming soon v1.10 Focus on solving latency and infrastructure issues Stable language foundation Vibrant package ecosystem Yearly developer conference, all talks and workshops on Youtube. Excellent native GPU computing support NVIDIA AMD Intel Apple Experimental support for accelerators like Graphcore IPU. \"\"\" md\"\"\" Getting started with Julia info Modern Julia Workflows https modernjuliaworkflows.org is an excellent resource to get started with. Installation In order of preference 1. Use `juliaup` ```shell curl fsSL https install.julialang.org | sh ``` 2. Use the binaries from from https julialang.org downloads 3. ... 4. Use whatever version of Julia is on your cluster 5. Don't use `julia` from your package manager Resources https modernjuliaworkflows.org https discourse.julialang.org https docs.julialang.org https julialang.org community events \"\"\" md\"\"\" A first Julia function \"\"\" function mandel z c z maxiter 80 for n 1 maxiter if abs z 2 return n 1 end z z^2 c end return maxiter end function mandel String \"Hello world\" end mandel \"\" function f a Int, b return \"first method\" end function f a, b Int return \"second method\" end f 1, \"\" f \"\", 1 f 1, 1 function f Int64, Int64 \"third method\" end mandel complex .3, .6 dump 0.3 0.6im π md\"\"\" Real component bind x real PlutoUI.Slider 2 0.01 1 Imaginary component bind x img PlutoUI.Slider 1 0.01 1 \"\"\" mandel complex x real, x img using CairoMakie md\"\"\" Resolution bind resolution PlutoUI.Slider 0.1, 0.01, 0.001 , show value true Real shift bind real shift PlutoUI.Slider 3 resolution 3, show value true, default 0 Img shift bind img shift PlutoUI.Slider 3 resolution 3, show value true, default 0 Zoom bind zoom PlutoUI.Slider 1 100, show value true, default 1 \"\"\" evaluate the mandelbrot set over a complex plane begin reals 2 resolution 1 . real shift . zoom imgs 1 resolution 1 . img shift . zoom plane complex re, img for re, img in Iterators.product reals, imgs end heatmap mandel. plane "},{"url":"mod1_introduction/types/","title":"Types, Structs, and Mutable Structs","description":"","tags":["module1","track_principles","indepth"],"text":" A Pluto.jl notebook v1.0.1 frontmatter chapter \"1\" section \"2\" order \"2.7\" title \"Types, Structs, and Mutable Structs\" tags \"module1\", \"track principles\", \"indepth\" layout \"layout.jlhtml\" indepth number \"1\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils "},{"url":"mod2_github_reproducibility/debugging/","title":"Debugging","description":"","tags":["module2","track_principles"],"text":" A Pluto.jl notebook v1.0.1 frontmatter chapter \"2\" section \"3\" order \"7\" title \"Debugging\" date \"2026 05 28\" tags \"module2\", \"track principles\" layout \"layout.jlhtml\" indepth number \"2\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils begin using PlutoUI, PlutoTeachingTools using PlutoUI Slider PlutoUI.TableOfContents depth 4 end begin using CairoMakie set theme theme latexfonts fontsize 16, Lines linewidth 2, , markersize 16 end ChooseDisplayMode using Asciicast using HypertextLiteral md\"\"\" Debugging \"\"\" md\"\"\" Strategies Ideas \"Scientific method\" Form a hypothesis Aim to invalidate it One change at a time Reduce, reduce, reduce The smaller the section of code, the easier it is to have a mental model of what is happeneing Create a git repository \\ Goal Keep track of changes we have so that we don't lose a bug. Is it \"randomly\" happening? What do we do after we fixed a bug? Turn our MWE into a test The bug likely happened due to some confusion of assumptions Does this pattern repeat elsewhere? \"\"\" md\"\"\" The importance of an MWE Remember last weeks lecture on reproducibility You might need to ask a friend or stranger for help What version of Julia are you version Which packages Project.toml Manifest.toml note Make it as easy as possible for another person to reproduce your setup \"\"\" md\"\"\" Common causes Noteworthy Differences https docs.julialang.org en v1 manual noteworthy differences Undefined memory Julia allows for memory to be unitialized ```julia x Array Float64 undef, 10 or similar x ``` \"\"\" sum Array Float64 undef, 10 let x ones 10 similar x end md\"\"\" this can \"poison\" your calculations \"\"\" md\"\"\" Aliasing of memory and variable scope ```julia a 0, 1, 0 b a b 2 0 ??? a ``` ```julia function f a a 3 end a 2 f a ?a ``` \"\"\" md\"\"\" \"Printf\" debugging When we write code we should check our assumptions and raise appropriate errors \"\"\" 1÷0 1 0 md\"\"\" You can write your own exception types and use `throw` to raise them. \"\"\" function safe div x, y if y zero y throw DivideError end x y end safe div 1, 0 md\"\"\" Or you can use `error` to raise a generic `ErrorException`. \"\"\" error \"Should have thought about this\" md\"\"\" Or ` assert` to check conditions. \"\"\" assert 1 0 md\"\"\" Revise Revise https github.com timholy Revise.jl is a core component of my development workflow. It watches source files and packages for changes and applies them to my running Julia session note If you use `VSCode` it is automatically used when you use the inbuilt Julia REPL. note If the script you are working on is split across multiple files, you need to use `includet` for Revise to track included files. \"\"\" md\"\"\" Julia's logging infrastructure Julia has a rich logging infrastructure https docs.julialang.org en v1 stdlib Logging man logging with different levels ` debug` Hidden by default, can be turned on by setting the environment variable `JULIA DEBUG \"Package\"` ` info` General information, great for passing structured information to the user. ` warn` Something is wrong, but the program is likely continuing ` error` Something is really wrong note ` error` does not imply `error ` it in itself is not an exception. You can use the Logging infrastructure to redirect and manipulate structured log messages. \"\"\" info \"Wow so useful \" lecture true md\"\"\" You can used the reserved field `exception` to recover from errors and pass along the backtrace \"\"\" try error \"Ye who enters, abandons all hope \" catch err info \"Caught error\" exception err end try error \"Ye who enters, abandons all hope \" catch err bt catch backtrace info \"Caught error\" exception err, bt end md\"\"\" This can be useful if you want to figure out in which context a log message was fired \"\"\" function fib x if x 0 || x 1 info \"Basecase\" exception ErrorException \"\" , backtrace return x end fib x 1 fib x 2 end fib 1 fib 3 md\"\"\" Callstacks and backtraces Profilers and debuggers think in backtraces When we execute a program, each function needs a little bit of space on the stack. note Heap and Stack are regions in memory, the `heap` contains all allocated data and the stack contains the local variables & co of the currently executing functions. given the execution of `fib 3 ` ``` fib 3 fib 2 fib 1 fib 0 fib 1 ``` Each function may execute more functions, but we only execute one function at a time. So we visit the call graph in order and thus linearize the execution. ``` fib 3 ``` ``` fib 3 fib 2 ``` ``` fib 3 fib 2 fib 1 ``` ``` fib 3 fib 2 fib 0 ``` ``` fib 3 fib 1 ``` \"\"\" md\"\"\" When we collect a profile, or receive an exception, or are in a debugger at a breakpoint, we almost always will collect the current backtrace stacktrace. Letting us know \"how\" did we get here. Information that is not in a backtrace is as an example loops \"\"\" md\"\"\" Tools \"\"\" md\"\"\" Debugger.jl Julia's native debugger. Built upon JuliaInterpreter.jl https github.com JuliaDebug JuliaInterpreter.jl Pro's Close to \"your code\" Full step through debugging with variable inspection Con's Only as fast as JuliaInterpreter and thus can struggle with large codebases Example ```julia using Debugger function foo n x n 1 BigInt 1 1 1 0 ^x 2,1 end enter foo 20 ``` \"\"\" asciicinema LocalResource \"recordings debugger.cast\" .src speed 2.0 md\"\"\" Infiltrator.jl Since Debugger.jl needs to interpret all code to ensure breakpoints are triggered it can be slow on a very large code to reach the region of interest ```julia function f x out for i in x push out, 2i infiltrate end out end ``` Tricks Combine with Revise.jl and use `Main. infiltrate` to work on package code \"\"\" asciicinema LocalResource \"recordings infiltrator.cast\" .src speed 2.0 md\"\"\" VS Code Debugger.jl is integrated into VS Code https www.julia vscode.org docs dev userguide debugging it has similar caveats as before, but you may prefer it over the pure text interface. \"\"\" md\"\"\" GDB LLDB GDB and LLDB are native debugger, they are closest to the true execution of Julia. They are very powerful but they require are deeper understanding on how programs work. Pro's Faster than `Debugger.jl` since it debug's native code Faithful Debugging the code doesn't change it's behavior Con's Low level view Doesn't know much about Julia code Tricks `ENABLE GDBLISTENER 1 julia g2` and potentially a debug build of Julia ` ccall jl breakpoint val Any Cvoid` `jl ... ` Breakpoints Watchpoitns Execute commands on breakpoint https docs.julialang.org en v1 devdocs debuggingtips ```julia function example r Ref 0 ccall jl breakpoint r Any Cvoid for i in 1 10 r i end return r end ``` \"\"\" asciicinema LocalResource \"recordings gdb.cast\" .src speed 1.5 md\"\"\" ``` ENABLE GDBLISTENER 1 gdb args julia g2 gdb break jl breakpoint gdb run julia function example r Ref 0 ccall jl breakpoint r Any Cvoid for i in 1 10 r i end return r end example generic function with 1 method julia example Thread 1 \"julia\" hit Breakpoint 1, jl breakpoint v 0x7fffeec5c6a0 at cache build tester amdci5 12 julialang julia release 1 dot 11 src rtutils.c 1475 warning 1475 cache build tester amdci5 12 julialang julia release 1 dot 11 src rtutils.c No such file or directory gdb p jl v Base.RefValue Int64 x 0 gdb bt 3 gdb p int64 t v gdb p x v 3 0x7fffeec5c6a0 gdb watch int64 t 0x7fffeec5c6a0 Hardware watchpoint 2 int64 t 0x7fffeec5c6a0 gdb continue Thread 1 \"julia\" hit Hardware watchpoint 2 int64 t 0x7fffeec5c6a0 Old value 0 New value 55 0x00007fffebcfc518 in julia example 43 at REPL 1 5 ``` \"\"\" md\"\"\" rr note A key challenge when debugging is that we often ask the question \"how did we get here\", but debuggers are really good at answering the question \"where are we going from here\". RR https rr project.org is a time traveling debugger. Using `rr record` we record the execution of a program, and it's children. Using `rr replay` we can re execute the program faith fully. We can then use `gdb` to debug a program being replayed and do things like `reverse continue`, e.g. execute a program backwards in time to answer questions like When did this value in memory get set? Caveat Linux only Works most reliably on Intel CPU boooh Doesn't work with GPU codes. \"\"\" md\"\"\" Cthulhu.jl Cthulhu.jl https github.com JuliaDebug Cthulhu.jl is an unusal debugger in the sense that it doesn't debug values, but it debugs types, and thus what the Julia compiler was able to infer about your program. It takes a static walk though dynamic programs , it works on multiple level of representation, but at it's core it operates on typed intermediate representation . \"\"\" md\"\"\" Motivation Have you ever done ` code typed` Search for a `call g, .... Any` `code typed g, .... ` repeat? \"\"\" md\"\"\" Examples ```julia import NamedDims g NamedDims.unify names a, b , a, ``` ```julia h A,B A . B . A h zeros 3 , zeros 3,3 ``` ```julia struct Interval Number a Float64 b Float64 end Base.eltype Interval Float64 contains I Interval, x Float64 I.a x I.b f I Interval, xs Vector Float64 contains. I, xs f Interval 0.5, 0.6 , rand 4 ``` \"\"\" begin struct Interval Number a Float64 b Float64 end Base.eltype Interval Float64 contains I Interval, x Float64 I.a x I.b f I Interval, xs Vector Float64 contains. I, xs end f Interval 0.5, 0.6 , rand 4 with terminal do code warntype f Interval 0.5, 0.6 , rand 4 end begin function asciicinema src speed 1.0 htl \"\"\" link rel \"stylesheet\" type \"text css\" href \"https cdn.jsdelivr.net npm asciinema player 3.9.0 dist bundle asciinema player.css\" script src \"https cdn.jsdelivr.net npm asciinema player 3.9.0 dist bundle asciinema player.min.js\" script div style \"margin 20px\" script const div document.createElement \"div\" AsciinemaPlayer.create src, div, fit \"width\", cols 80, rows 24, speed speed, preload true, return div script div \"\"\" end function asciicinema cast Asciicast.Cast kwargs... str Asciicast.base64encode Asciicast.collect bytes cast asciicinema \"data text plain base64,\" str kwargs... end end "},{"url":"mod2_github_reproducibility/github_and_package_manager/","title":"Github & Package Manager","description":"","tags":["module2","track_principles"],"text":" A Pluto.jl notebook v1.0.1 frontmatter chapter \"2\" section \"1\" order \"5\" title \"Github & Package Manager\" date \"2026 05 14\" tags \"module2\", \"track principles\" layout \"layout.jlhtml\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils using PlutoUI, PlutoTeachingTools ChooseDisplayMode TableOfContents "},{"url":"mod2_github_reproducibility/numerical_reproducibility/","title":"Numerical Reproducibility","description":"","tags":["module2","track_principles"],"text":" A Pluto.jl notebook v1.0.1 frontmatter chapter \"2\" section \"2\" order \"6.1\" title \"Numerical Reproducibility\" date \"2026 05 21\" tags \"module2\", \"track principles\" layout \"layout.jlhtml\" indepth number \"1\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils begin using PlutoUI, PlutoTeachingTools using PlutoUI Slider PlutoUI.TableOfContents depth 4 end begin using CairoMakie set theme theme latexfonts fontsize 16, Lines linewidth 2, , markersize 16 end ChooseDisplayMode md\"\"\" Numerical reproducibility What every Computer Scientist should know about Floating Point arithmetic https dl.acm.org doi 10.1145 103162.103163 \"\"\" md\"\"\" Randomness We sometimes \"desire\" randomness, but true randomness is hard in a computer. In computer programs we talk about \"pseudo random number generators\", given a seed they generate a \"unpredicatble\" sequence of numbers with a very large period. \"\"\" using Random rand rand let Random.seed 182 rand end let Random.seed 182 rand end walk x rand x 1, x 1 function walk nsteps N steps 0 for in 2 N push steps, walk steps end end return steps end let fig Figure ax Axis fig 1, 1 , xlabel \"Position\", ylabel \"Steps\", title \"Unseeded\" N 100 lines ax, walk nsteps N , 1 N lines ax, walk nsteps N , 1 N lines ax, walk nsteps N , 1 N lines ax, walk nsteps N , 1 N fig end let fig Figure ax Axis fig 1, 1 , xlabel \"Position\", ylabel \"Steps\", title \"Seeded\" N 100 Random.seed 12 lines ax, walk nsteps N , 1 N lines ax, walk nsteps N , 1 N lines ax, walk nsteps N , 1 N lines ax, walk nsteps N , 1 N fig end md\"\"\" Order of summing Kahan summation https en.wikipedia.org wiki Kahan summation algorithm Pairwise summation used in Julia https github.com JuliaLang julia pull 4039 \"\"\" 0.1 0.2 0.3 0.1 0.2 0.3 md\"\"\" note My first \"silly question\" https stackoverflow.com questions 21872854 floating point math in different programming languages \"\"\" md\"\"\" Floating point arithmetic doesn't quite follow the rules of real arithmetic. Instead the rules are defined in a standard called `IEEE 754`. Because summation in different order will give different answers, the compiler has no lease to reorder operations for us. This is in particular relevant for vectorization and parallel computation. Code may give different answers due to different execution order. \"\"\" md\"\"\" 2046 floating pointer numbers that sum to almost anything Thanks to Stefan Karpinski https discourse.julialang.org t array ordering and naive summation 1929 \"\"\" md\"\"\" The `sumsto` function always returns the same 2046 floating point numbers but returns them in a different order based on `x` for any 64 bit float value of `x` from 0 up to but not including 2^970, the naive left to right sum of the vector returned by `sumsto x ` is precisely `x` \"\"\" function sumsto x Float64 0 x exp2 970 || throw ArgumentError \"sum must be in 0,2^970 \" n, p₀ Base.decompose x integers such that `n exp2 p₀ x` floatmax exp2 p for p in 1074 969 if iseven n p p₀ floatmax exp2 p for p in 1074 969 if isodd n p p₀ end foldl , sumsto 0.0 foldl , sumsto eps 0.0 foldl , sumsto 1.23 foldl , sumsto pi 0 foldl , sumsto 6.0221409e23 foldl , sumsto 9.979201547673598e291 md\"\"\" When adding the values from left to right, all the powers of two after `floatmax ` but before ` floatmax ` have no effect on the sum. Then ` floatmax ` cancels `floatmax ` out, bringing the sum back to zero so the remaining powers of two are added up as expected, giving the final result, `x`. There are 2044 powers of two that can be represented as 64 bit floating point value below 2^970 since the smallest representable power of 2 is `eps 0.0 exp2 1074 5.0e 324`. So the number of values we’re summing is 1074 970 2, one for each power of two and two more for `floatmax ` and ` floatmax `. \"\"\" md\"\"\" Fast math Simon Byrne notes on Fast Math https simonbyrne.github.io notes fastmath Towards Useful Fast Math https llvm.org devmtg 2024 10 slides techtalk Kaylor Towards Useful Fast Math.pdf \"\"\" function foo A 1.0f0 C 1.0f0 Find the smallest value A 2^k for which A 1 A 1 while C 1.0f0 A 2.0f0 C A 1.0f0 A end return A end foo md\"\"\" warning Do not execute `foo fast` It will loop forever. \"\"\" function foo fast A 1.0f0 C 1.0f0 Find the smallest value A 2^k for which A 1 A 1 fastmath while C 1.0f0 A 2.0f0 C A 1.0f0 A end return A end with terminal do code llvm foo fast end md\"\"\" Precision of mathematical implementations CPU vs GPU? `muladd` can given different answer expansion of precsion... \"\"\" md\"\"\" Extending precision, can change the answer, as an example in Swift, operations may be executed in extended precision and one might get different answers under transformations. Take the code below and run it in https swiftwasm.org ```swift func foo x Float16 Float16 let s Float16 1024 let t Float16 3 s let rx Float16 x x t t return rx print foo x 0.5 0.0 func bar x Float16 Float16 let s Float16 1024 let t Float16 3 s let rx Float16 x bar plus a x, b t t return rx inline never func bar plus a Float16, b Float16 Float16 return a b print bar x 0.5 0.5 ``` \"\"\" md\"\"\" Our goal in Julia for `Float16` is to emulate what hardware with real Float16 support would do. \"\"\" md\"\"\" This behavior is important when we implement our own trigonometic functions https github.com JuliaLang julia blob 8987f7ac6d284cb512d336279a74b5414c635757 base special trig.jl L764 If we look at `foo` we would think that `x x t t ` should always return `0.0`, but we choose `t` specially for that to not be the case. \"\"\" maxintfloat Float16 2 Float16 1024 function foo x T where T s maxintfloat T 2 t 3 s rx x x t t end foo Float32 0.5 foo Float64 0.5 foo Float16 0.5 function bar x Float16 s Float16 1024 t 3 s rx x bar plus x,t t end noinline function bar plus a, b return a b end bar Float16 0.5 md\"\"\" So what happens is that Swift and Julia in olden times internally expanded `Float16` operations to `Float32` \"\"\" begin extend x Float16 Float32 x extend x Float32 Float64 x end function foo extended x T where T s maxintfloat T 2 t extend 3 s extend precision rx T x x t t end foo extended Float16 0.5 foo extended Float32 0.5 md\"\"\" This answer is seemingly \"more\" correct since it calculates our \"real\" understanding of x x t t , but it is more useless since it does not accuratly implement floating point semantics. \"\"\" md\"\"\" Implementation of Float16 \"\"\" md\"\"\" ```julia abstract type Number end abstract type Real Number end abstract type AbstractFloat Real end primitive type Float64 AbstractFloat 64 end primitive type Float32 AbstractFloat 32 end primitive type Float16 AbstractFloat 16 end ``` \"\"\" methods cbrt md\"\"\" First attempt Naively lowering Float16 to LLVM’s half type. What to do on platforms with no limited hardware support Extended precision thanks x87 rears it’s ugly head Lesson In order to implement numerical routines that are portable we must be very careful in what semantics we promise. Solution On targets without hardware support for `Float16`, truncate after each operation. GCC 12 supports this as ` fexcess precision 16` \"\"\" md\"\"\" On x86 \"\"\" md\"\"\" ```llvm define half julia muladd half %0, half %1, half %2 top %3 fmul half %0, %1 %4 fadd half %3, %2 ret half %4 ``` \"\"\" md\"\"\" turns into \"\"\" md\"\"\" ``` define half julia muladd half %0, half %1, half %2 top %3 fpext half %0 to float %4 fpext half %1 to float %5 fmul float %3, %4 %6 fptrunc float %5 to half %7 fpext half %6 to float %8 fpext half %2 to float %9 fadd float %7, %8 %10 fptrunc float %9 to half ret half %10 ``` \"\"\" md\"\"\" On your machine? \"\"\" with terminal do code llvm muladd, Float16, Float16, Float16 , optimize false end with terminal do code llvm muladd, Float16, Float16, Float16 end md\"\"\" Stochastic rounding 🎲 Sometimes randomness can be quite useful Real numbers constitue a continuous set \\mathbb R , but finite precision numbers used in computers are a part of a discrete set F \\subset \\mathbb R . When computers do operations involving floating point numbers in F , the true result x \\in \\mathbb R will be approximated by a number \\hat x \\in F , which is typically chosen deterministically to be the nearest number in F this is called \"nearest rounding\". Stochastic rounding is an alternative rounding mode to classic deterministic rounding, which randomly rounds a number x \\in \\mathbb R to either of the two nearest floating point numbers of the result \\lfloor x \\rfloor previous number in F or \\lceil x \\rceil following number in F with the following rule ```math \\mathrm round x \\begin cases \\lfloor x \\rfloor & \\text with probability P x \\\\ 6pt \\lceil x \\rceil & \\text with probability 1 P x \\end cases ``` Common choices are P x 1 2 or, more interestingly, ```math P x \\frac x \\lfloor x \\rfloor \\lceil x \\rceil \\lfloor x \\rfloor ``` In the following we'll always talk about the latter probability function P x . Resource \"https nickhigham.files.wordpress.com 2020 06 stoch round fig2.jpg\" source \" What Is Stochastic Rounding? https nhigham.com 2020 07 07 what is stochastic rounding \" by Nick Higham Stochastic rounding is useful because the average result of operations matches the expected mathematical result. In a statistical sense, it retains some of the information that is discarded by a deterministic rounding scheme, smoothing out numerical rounding errors due to limited precisions. This is particularly important when using low precision floating point numbers like `Float16`. For contrast, deterministic rounding modes like nearest rounding introduce a bias, which is more severe as the precision of the numbers is lower. The IPU is one of the very few processors available with hardware support for stochastic rounding. Let's do an exercise on the CPU with classical nearest rounding. We define a function to do the naive sequential sum of a vector of numbers, because the `sum` function in Julia uses pairwise summation https en.wikipedia.org wiki Pairwise summation , which would have better accuracy. From Mosè Giordano talk on using Julia on IPU https giordano.github.io talks 2024 06 20 julia ipu ornl b11731af d8f3 4714 839d 1535f459a278 \"\"\" naive sum v foldl , v x fill Float16 0.9 , 3000 naive sum x eps Float16 2048 naive sum x ≈ x 1 length x using StochasticRounding using Statistics x sr Float16sr. x naive sum x sr sums sr map naive sum x sr , 1 10000 extrema sums sr mean Float64. sums sr std Float64. sums sr median sums sr let fig Figure ax Axis fig 1,1 stephist ax, sums sr, label \"Stochastic Rounding\", color green vlines ax, naive sum x , , label \"Nearest Rounding\" vlines ax, x 1 length x , label \"True value\" axislegend ax position ct fig end "},{"url":"mod3_parallelism/accelerated/","title":"GPU Computing","description":"","tags":["module3","track_parallel"],"text":" A Pluto.jl notebook v1.0.1 frontmatter chapter \"3\" section \"2\" order \"9\" title \"GPU Computing\" date \"2026 06 10\" tags \"module3\", \"track parallel\" layout \"layout.jlhtml\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils This Pluto notebook uses bind for interactivity. When running this notebook outside of Pluto, the following 'mock version' of bind gives bound variables a default value instead of an error . macro bind def, element format off return quote local iv try Base.loaded modules Base.PkgId Base.UUID \"6e696c72 6542 2067 7265 42206c756150\" , \"AbstractPlutoDingetjes\" .Bonds.initial value catch b missing end local el esc element global esc def Core.applicable Base.get, el ? Base.get el iv el el end format on end using PlutoUI, PlutoTeachingTools import PlutoUI Slider using CairoMakie ChooseDisplayMode PlutoUI.TableOfContents depth 4 md\"\"\" Accelerated Computing \"\"\" md\"\"\" Accelerated \"\"\" md\"\"\" A GPU has many \"lightweight\" threads \"\"\" md\"\"\" CPU die shot RobustLocalResource \"https s3.amazonaws.com media p.slid.es uploads 779853 images 4399827 800px coffee lake die quad core annotated .png\", \"quad core.png\" \"\"\" md\"\"\" GPU block diagram RobustLocalResource \"https devblogs.nvidia.com parallelforall wp content uploads 2016 04 gp100 block diagram 1 624x368.png\", \"p100.png\" \"\"\" md\"\"\" Bottlenecks RobustLocalResource \"https s3.amazonaws.com media p.slid.es uploads 779853 images 4399848 pasted from clipboard.png\", \"gpu system.png\" \"\"\" md\"\"\" Composable infrastructure Core GPUCompiler.jl https github.com JuliaGPU GPUCompiler.jl Takes native Julia code and compiles it directly to GPUs GPUArrays.jl https github.com JuliaGPU GPUArrays.jl High level array based common functionality KernelAbstractions.jl https github.com JuliaGPU KernelAbstractions.jl Vendor agnostic kernel programming language Adapt.jl https github.com JuliaGPU Adapt.jl Translate complex structs across the host device boundary Vendor specific CUDA.jl https github.com JuliaGPU CUDA.jl AMDGPU.jl https github.com JuliaGPU AMDGPU.jl oneAPI.jl https github.com JuliaGPU oneAPI.jl Metal.jl https github.com JuliaGPU Metal.jl General backends OpenCL.jl https github.com JuliaGPU OpenCL.jl \"\"\" md\"\"\" Different layers of abstraction Vendor specific ```julia using CUDA function saxpy a,X,Y i blockIdx .x 1 blockDim .x threadIdx .x if i length Y inbounds Y i a X i Y i end return nothing end cuda threads 32 blocks cld length Y , 32 saxpy a, X, Y ``` KernelAbstractions ```julia using KernelAbstractions using CUDA kernel function saxpy a, Const X , Y I index Global inbounds Y I a X I Y I end saxpy CUDABackend a, X, Y, ndrange length Y ``` Array abstractions ```julia Y . a . X . Y ``` \"\"\" md\"\"\" Asynchronous operations warn GPU operations are asynchronous with regards to the host They are ordered with respect to each other, but special care must be taken when using Julia's task based programming together with GPU programming. The JuliaGPU ecosystem synchronizes the GPU on access, so when you move data from and to the GPU we wait for all the kernels to finish See the exercise \"Introduction to KernelAbstractions\" for more details on how to measure GPU kernels and performance. \"\"\" md\"\"\" How to use KernelAbstractions Use ` kernel function mykernel args... end` to write a GPU style program Instantiate kernel for a backend `kernel mykernel backend ` Backends come from Vendor specific libraries `KA.allocate backend, ... ` to obtain memory Launch kernel `kernel args..., ndrange ... ` while specifying the grid to execute over. \"\"\" TwoColumn md\"\"\" ```julia function vadd a, b, c for i in eachindex c c i a i b i end end a rand N b rand N c similar a vadd a, b, c ``` \"\"\", md\"\"\" ```julia import KernelAbstractions as KA kernel function vadd a, b, c i index Global c i a i b i end backend CUDABackend a KA.allocate backend, Float32, N b KA.allocate backend, Float32, N c similar a vadd kernel vadd backend vadd kernel a, b, c ndrange size c ``` \"\"\" md\"\"\" note When benchmarking you need to synchronize the device ```julia benchmark begin vadd kernel a, b, c ndrange size c KA.synchronize backend end ``` Otherwise you are only measuring the launch of the kernel. \"\"\" md\"\"\" High level array based programming Julia and GPUArrays.jl provide support for an efficient GPU programming environment build around array abstractions and higher order functions. Vocabulary of operations `map`, `broadcast`, `scan`, `reduce`, ... Map naturally onto GPU execution models Compiled to efficient code multiple dispatch, specialization Write generic, reusable applications BLAS matrix multiply, ... , and other libraries like FFT Array operators using multiple dispatch a design methodology for array implementations in dynamic languages doi 10.1145 2627373.2627383 Rapid software prototyping for heterogeneous and distributed platforms doi 10.1016 j.advengsoft.2019.02.002 \"\"\" md\"\"\" Array types where memory resides and how code is executed. | | | | | | | `A Matrix Float64 undef, 64, 32 ` | CPU | | `A CuMatrix Float64 undef, 64, 32 ` | Nvidia GPU | | `A ROCMatrix Float64 undef, 64, 32 ` | AMD GPU | info Data movement is explicit. \"\"\" md\"\"\" Guiding principle info \"Compute follows data\" By using types to differentiate where data is being stored, computation can be dispatched to the right compute device, and the user is protected from expensive data transfers Even in the era of unified virtual memory . \"\"\" md\"\"\" What makes an application portable? 1. Can I run it on a different compute architecture 1. Different CPU architectures 2. We live in a multi GPU vendor world 2. Does it compute the same thing? 1. Can I develop on one platform and move to another later? 3. Does it achieve the same performance ? 4. Can I take advantage of platform specific capabilities? Productivity meets Performance Julia on A64FX doi 10.1109 CLUSTER51413.2022.00072 \"\"\" md\"\"\" Adapt.jl Adapt.jl https github.com JuliaGPU Adapt.jl is a lightweight dependency that you can use to convert complex structures from CPU to GPU. ```julia using Adapt adapt CuArray, Adjoint Array Adjoint CuArray ``` ```julia struct Model T Number, AT AbstractArray T data AT end Adapt.adapt structure to, x Model Model adapt to, x.data cpu Model rand 64, 64 using CUDA gpu adapt CuArray, cpu Model Float64, CuArray Float64, 2, CUDA.Mem.DeviceBuffer ... ``` \"\"\" md\"\"\" Many backends \"\"\" using ThreadPinning pinthreads, threadinfo begin pinthreads cores threadinfo tp marker nothing end using KernelAbstractions, BenchmarkTools begin import CUDA import Metal Work around https github.com JuliaGPU oneAPI.jl issues 445 . ENV \"SYCL PI LEVEL ZERO BATCH SIZE\" \"1\" import oneAPI import AMDGPU import OpenCL import pocl jll end available backends let tp marker available backends KernelAbstractions.Backend CPU , OpenCL.OpenCLBackend CI get ENV, \"GITHUB ACTIONS\", \"false\" \"true\" Always shows all backend when rendering the notebook on GitHub Actions, notebooks will be static anyway. if CUDA.functional || CI push available backends, CUDA.CUDABackend end if oneAPI.functional || CI push available backends, oneAPI.oneAPIBackend end if Metal.functional || CI push available backends, Metal.MetalBackend end if AMDGPU.functional || CI push available backends, AMDGPU.ROCBackend end available backends end md\"\"\" Vendor agnostic kernel programming with `KernelAbstractions.jl` Computing π based on material from Mosè Giordano \"\"\" using MultiFloats kernel function pi kernel results AbstractVector FT , step, work per thread where FT rank index Global lower rank 1 work per thread upper lower work per thread 1 local pi zero FT for idx in lower upper x idx FT 0.5 step local pi 4 1 x x end inbounds results rank local pi end function picalc backend, blocks, threads per block, num steps, FT Float64 step 1 FT num steps tot threads blocks threads per block work per thread num steps ÷ tot threads results KernelAbstractions.zeros backend, FT, tot threads pi kernel backend results, step, num steps ÷ tot threads workgroupsize threads per block, ndrange tot threads KernelAbstractions.synchronize backend s sum results step return s end Run the benchmark. function picalc backend, blocks, threads per block, numsteps, FT Float64 println \"Calculating PI using \" println \" \", numsteps, \" slices\" if backend isa CPU println \" \", Threads.nthreads , \" CPU threads s \" else GPU threads are either SM Cores or threads per block blocks println \" \", threads per block, \" threads × \", blocks, \" blocks \" end start time mypi picalc backend, blocks, threads per block, numsteps, FT elapsed time start println \"Obtained value of PI in \", typeof mypi , \" \", mypi, \" relative error \", round Float64 abs 1 big mypi pi sigdigits 2 , \" \" println \"Time taken \", round elapsed, digits 3 , \" seconds\" end md\"pi\\ backend bind pi backend Select available backends \" let available types Float32 if pi backend isa Metal.MetalBackend push available types, Float64 end append available types, Float32x2, Float32x4 if pi backend isa Metal.MetalBackend append available types, Float64x2, Float64x4 end md\"float\\ type bind float type Select available types \" end let info \"warming up...\" blocks 256 threads per block 256 num steps blocks threads per block 2 time picalc pi backend, blocks, threads per block, num steps, float type nothing end let values filter blocks threads per block , 2 .^ 20 42 default max minimum values , 2 ^ 28 md\"num\\ steps bind num steps Slider values default, show value true \" end blocks 4096 threads per block 1024 with terminal do picalc pi backend, blocks, threads per block, num steps, float type end with terminal do tp marker picalc CPU , blocks, threads per block, num steps, float type end md\"\"\" Writing a custom sum function with KernelAbstractions \"\"\" md\"backend bind backend Select available backends \" md\"\"\" The below code implements a custom eigenvalue solver. We are interested in implementing porting it to the GPU. For a momement let's forget about `GPUArrays` and all the high level primitives it provides. \"\"\" md\"\"\" ```julia f A MyMatrix λ 1 mapreduce v v^2 v λ , , A.v f′ A MyMatrix λ mapreduce v v^2 v λ ^2, , A.v function LinearAlgebra.eigmax A MyMatrix tol eps 2.0 , debug false x0 maximum A.v maximum A.v ^2 δ f A x0 f′ A x0 while abs δ x0 tol x0 δ δ f A x0 f′ A x0 debug && println \"x x0, δ δ\" Debugging end x0 end ``` \"\"\" question box md\"\"\" 1. What fundamental operations do we need to implement? 2. Is there a synchronization point within this code? \"\"\" hint md\"\"\" 1. `maximum v ` can also be written as `reduce max, v, init typemin eltype v ` 2. `reduce op, v ` can also be written as `mapreduce identity, op, f ` So the only operation we need to implement is a `mapreduce` Yes, there are several synchronization points, most notably both `f` and `f′` have to wait for the GPU kernel to finish to return a scalar GPU value. An interesting optimization might be to better pipeline this code and try to avoid a bubble between `f` and `f′`. Or perhaps write a fully custom kernel \"\"\" md\"\"\" We can only communicate with the GPU through Global Memory. A GPU kernel does not return anything. Thus we allocate a single \"value\" storage array. note \"Shared Virtual Memory Unified Virtual Memory\" SVM UVM make memory on the GPU CPU mutually accesibly, we could thus not use an array, but perhaps a `RefValue`, though this is not portable and would mean we miss the implicit syncronization semantics. For our custom eigenvalue solver we need `mapreduce`, but for now let's just implement a non generic `sum`. \"\"\" kernel function naive sum val, data I index Global x data I val 1 x end data KernelAbstractions.ones backend, Int, 1024^2 sum data let x KernelAbstractions.zeros backend, eltype data , 1 naive sum backend x, data, ndrange length data Array x 1 end md\"\"\" note \"Data race\" This looks similar to our issues from last week Data races galore Let's use Atomix again to avoid these issues. \"\"\" using Atomix atomic, atomicswap, atomicreplace kernel function naive atomic sum val, data I index Global x data I atomic val 1 x end let x KernelAbstractions.zeros backend, eltype data , 1 oops I used similar here... naive atomic sum backend x, data, ndrange length data Array x 1 end let x KernelAbstractions.zeros backend, eltype data , 1 benchmark begin naive atomic sum backend x, data, ndrange length data KernelAbstractions.synchronize backend end end benchmark sum data if backend isa CUDA.CUDABackend CUDA. profile sum data end md\"\"\" Currently we have a big bottle neck. The atomic summation on the global data KernelAbstraction provides a ` localmem` region that implements a memory region that is local to a work group So we can load all the data in parallel, then sum it up, and perform much fewer atomic stores. \"\"\" kernel function naive shmem sum val, data AbstractArray T where T I index Global i index Local N uniform prod groupsize tile localmem T N, tile i data I synchronize if i 1 acc tile 1 for i in 2 N acc tile i end atomic val 1 acc end end let x KernelAbstractions.zeros backend, eltype data , 1 naive shmem sum backend, 64 x, data, ndrange length data Array x 1 end let x KernelAbstractions.zeros backend, eltype data , 1 benchmark begin naive shmem sum backend, 64 x, data, ndrange length data KernelAbstractions.synchronize backend end end md\"\"\" info There are improvements we can do on this algorithm As an example one thread is doing all the \"sum\" work per compute group. We probably want to share that work. In CUDA there are also shuffle operations that can be used instead of the `localmem`. For now let's write a \"mapreduce\" code that applies an operation `f` to each element. \"\"\" md\"\"\" warn The code below contains an loop that uses atomic operations to update a value... Developing this is a pain since it can easily hang your system. \"\"\" md\"\"\" note Sometimes things are broken. Originally the plan was to implement a generic `mapreduce` kernel using atomic operations, but this did not work out generally. Instead we implement only `mapsum` ```julia kernel function naive shmem mapreduce val, data AbstractArray T , f F, op O where T, F, O I index Global i index Local N uniform prod groupsize tile localmem T N, tile i f data I synchronize if i 1 acc tile 1 for i in 2 N acc op acc, tile i end TODO Generic atomic breaks atomic val 1 op acc val 1 op val 1 , acc v atomic val 1 success false while success v, success atomicreplace val 1 v op v, acc end end end ``` \"\"\" kernel function naive shmem mapsum val, data AbstractArray T , f F where T, F I index Global i index Local N uniform prod groupsize tile localmem T N, tile i f data I synchronize if i 1 acc tile 1 for i in 2 N acc tile i end atomic val 1 acc end end using GPUArraysCore function my mapsum f, data backend get backend data x KernelAbstractions.zeros backend, eltype data , 1 naive shmem mapsum backend, 64 x, data, f, ndrange length data GPUArraysCore. allowscalar x 1 end my mapsum x x^2, data mapreduce x x^2, , data md\"\"\" Reuse `AcceleratedKernels.jl` Just like last week, where after hand writing a handful of parallel reductions we could simply reach for `OhMyThreads.treduce`, we do not have to write the ` localmem` reduction by hand on the GPU either. `AcceleratedKernels.jl` https github.com JuliaGPU AcceleratedKernels.jl provides portable, backend agnostic high level primitives built directly on top of `KernelAbstractions`. They mirror the `OhMyThreads.jl` vocabulary from last week, but run on any backend the CPU as well as every GPU vendor | `OhMyThreads.jl` CPU | `AcceleratedKernels.jl` portable | | | | | `tforeach` | `foreachindex` | | `tmap` | `map` | | `treduce` | `reduce` | | `tmapreduce` | `mapreduce` | So our hand written `naive shmem mapsum` `my mapsum` is really just a special case of `AK.mapreduce` and the library takes care of the local memory reduction, the atomics, and the work group sizing for us. \"\"\" import AcceleratedKernels as AK md\"ak\\ backend bind ak backend Select available backends \" ak data KernelAbstractions.ones ak backend, Int, 1024^2 AK.reduce , ak data init zero eltype ak data AK.mapreduce x x^2, , ak data init zero eltype ak data let benchmark begin AK.mapreduce x x^2, , ak data init zero eltype ak data KernelAbstractions.synchronize ak backend end end md\"\"\" Compute follows the backend , not the data The high level array operations from `GPUArrays` are implicitly parallel only because they dispatch on the array type `mapreduce f, , CuArray ` runs on the GPU because the data is a `CuArray`. But the very same operation over a lazy host iterator like `1 N` is just an ordinary, sequential CPU reduction there is no array to dispatch on, so nothing moves to the device \"\"\" mapreduce x x^2, , 1 length ak data plain sequential CPU reduction md\"\"\" `AcceleratedKernels.jl` decouples where the computation runs from where the data lives the backend is just an argument. A `1 N` range is `isbits`, so it is captured straight into the kernel and indexed on the device no `CuArray`, no explicit data movement required. ```julia AK.mapreduce x x^2, , 1 length data , backend init zero Int ``` Run on the selected backend ak backend , this evaluates the reduction over a host range on the GPU \"\"\" AK.mapreduce x x^2, , 1 length ak data , ak backend init zero Int md\"\"\" Custom array types \"\"\" md\"adapt\\ backend bind adapt backend Select available backends \" using Adapt using LinearAlgebra struct MyMatrix A v A end md\"\"\" note \"Adapt.jl\" Defining `Adapt.adapt structure` allows us to easily move data from device to host, even if it is complicated. \"\"\" Adapt.adapt structure to, x MyMatrix MyMatrix adapt to, x.v adapt adapt backend, MyMatrix X X randn 1024 md\"\"\" \"\"\" f A MyMatrix λ 1 my mapsum v v^2 v λ , A.v f′ A MyMatrix λ my mapsum v v^2 v λ ^2, A.v function LinearAlgebra.eigmax A MyMatrix tol eps 2.0 , debug false x0 my mapreduce identity, max, A.v my mapreduce identity, max, A.v ^2 x0 maximum A.v maximum A.v ^2 Cheating by using GPUArrays δ f A x0 f′ A x0 while abs δ x0 tol x0 δ δ f A x0 f′ A x0 debug && println \"x x0, δ δ\" Debugging end x0 end md\"\"\" note \"High Level Operations\" The code above uses only high level operations like `mapreduce` or `maximum` map reduce in disguise . \"\"\" eigmax diagm X X X' eigmax MyMatrix X eigmax adapt adapt backend, MyMatrix X backend md\"\"\" note \"Custom kernels\" For more on writing custom kernels https vchuravy.dev rse course exercises exercise\\ 17\\ gpu https vchuravy.dev rse course exercises exercise 17 gpu Open in Colab https colab.research.google.com assets colab badge.svg https colab.research.google.com github vchuravy rse course blob main src mod3 parallelism Introduction to KernelAbstractions.ipynb \"\"\" "},{"url":"mod3_parallelism/distributed/","title":"Distributed programming with MPI.jl","description":"","tags":["module3","track_parallel"],"text":" A Pluto.jl notebook v1.0.1 frontmatter chapter \"3\" section \"3\" order \"10\" title \"Distributed programming with MPI.jl\" date \"2026 06 17\" tags \"module3\", \"track parallel\" layout \"layout.jlhtml\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils This Pluto notebook uses bind for interactivity. When running this notebook outside of Pluto, the following 'mock version' of bind gives bound variables a default value instead of an error . macro bind def, element format off return quote local iv try Base.loaded modules Base.PkgId Base.UUID \"6e696c72 6542 2067 7265 42206c756150\" , \"AbstractPlutoDingetjes\" .Bonds.initial value catch b missing end local el esc element global esc def Core.applicable Base.get, el ? Base.get el iv el el end format on end begin using PlutoUI, PlutoTeachingTools using PlutoUI Slider PlutoUI.TableOfContents depth 4 end ChooseDisplayMode md\"\"\" Distributed programming with MPI.jl \"\"\" using MPI, Serialization, Statistics, StaticArrays md\"\"\" MPI stands for \"Message Passing Interface\" and is one of the oldest and most successful distributed programming paradigms. We normally start an MPI program with `mpiexec` see exercises , but this doesn't play nicely with Pluto. Here I use a ` mpi` macro to execute a block of code as a standalone MPI. `mpiexec` starts a number of independent processes that execute the same program. We can use API calls like `MPI.Comm rank` and `MPI.Comm size` to query which rank we are. \"\"\" md\"\"\" The ` mpi` macro executes a block as an MPI program. note Each block is isolated from another, and as such you need to setup state independently. warning The ` mpi` macro is purely to make MPI work in Pluto for teaching, but should be not used for any real uses. Furthermore, always wrap your blocks in `let` and not `begin` to not confuse Pluto. \"\"\" np 4 macro mpi np, expr path, io mktemp control io path path \".ji\" println io, \"using MPI, Serialization\" println io, \" mpi begin\" println io, expr println io, \"end\" println io, \"\"\" mpi MPI.gather mpi, MPI.COMM WORLD root 0 if MPI.Comm rank MPI.COMM WORLD 0 Serialization.serialize \" control io path\", mpi end \"\"\" close io quote let np esc np path path run ` mpiexec np np Base.julia cmd project Base.active project path ` end v Serialization.deserialize control io path rm control io path all isnothing, v ? nothing v end end md\"\"\" When using MPI we are going to execute N np independent copies of a program. \"\"\" mpi np let MPI.Init n rand println n end md\"\"\" As we can see the output of the program is mangled. note Most often we only print from `rank 0`. \"\"\" mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm if rank 0 println \"Comm size MPI.Comm size comm \" end end md\"\"\" It can also help to print in \"one shot\" \"\"\" mpi np let MPI.Init comm MPI.COMM WORLD print \"\"\" Hello world, I am rank MPI.Comm rank comm of MPI.Comm size comm \"\"\" MPI.Barrier comm wait for all processes to reach this point. end md\"\"\" Initialization and Finalization As you may have noticed when we start an MPI program we have to explicitly call `MPI.Init` In other programming languages you will have to use a corresponding `MPI.Finalize`, this is not necessary in Julia since we automatically execute `MPI.Finalize` when the program shuts down. \"\"\" md\"\"\" Two sided communication \"\"\" md\"\"\" Blocking \"\"\" mpi 2 let function pingpong T, sz, iters buff zeros T, sz comm MPI.COMM WORLD rank MPI.Comm rank comm MPI.Barrier comm tic MPI.Wtime for in 1 iters if rank 0 MPI.Send buff, comm dest 1 MPI.Recv buff, comm source 1 else MPI.Recv buff, comm source 0 MPI.Send buff, comm dest 0 end end toc MPI.Wtime return toc tic iters end MPI.Init pingpong Float64, 1024, 10 end md\"\"\" Rank 0 sends a message and waits for it to bounce back. Each iteration is therefore one full round trip , so ` toc tic iters` is the round trip time halve it to estimate the one way latency between two ranks. Try increasing `sz` to see the cost grow as we become bandwidth bound. note The program above is limited to two ranks Let's send data around in a ring \"\"\" mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm N MPI.Comm size comm assert N 1 if rank 0 token MPI.recv comm source rank 1 print \"Rank rank received token token.\\n\" else token 1 end MPI.send token, comm dest mod rank 1, N Now rank 0 can receive the token if rank 0 token MPI.recv comm source N 1 print \"Rank rank received token token.\\n\" end token end md\"\"\" Non Blocking It is quite tricky to reason about the order of `recv` and `send` across all processes. Note how we always have to set up some `recv` then perform the matching `send`. Then flip the operations. danger \"Deadlock\" What happens if every rank tries to send before it receives? ```julia Don't run this — it deadlocks MPI.Send send buf, comm dest mod rank 1, N MPI.Recv recv buf, comm source mod rank 1, N ``` `MPI.Send` is allowed to block until the matching `Recv ` is posted. If all ranks are stuck in `Send`, nobody ever reaches `Recv ` and the program hangs forever. For small messages the MPI library often buffers eagerly and it happens to work — which makes the bug even nastier, since it only deadlocks once the message grows. `Irecv ` and `Isend` are non blocking operations, that means we can issue them and wait later for them to be completed. Because the receive is already posted before we wait, the send always has somewhere to land and the deadlock disappears. note \"Tagging\" Since we now can have many operations in flight at once, one might need to tag the operations. \"\"\" mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm N MPI.Comm size comm nonblocking receive from previous rank recv buf Array Float64 undef, 2 recv req MPI.Irecv recv buf, comm source mod rank 1, N , tag 0 nonblock send to next rank send buf Float64 rank, rank send req MPI.Isend send buf, comm dest mod rank 1, N , tag 0 block until communication is complete MPI.Waitall recv req, send req print \" rank Received recv buf\\n\" end md\"\"\" Ghost cells \"\"\" md\"\"\" | 1 | 2 11 | 12 | | | | | | left | mine | right | \"\"\" mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm N MPI.Comm size comm M 10 data zeros M 2 Computation phase interior data 2 M 1 . rank Communication phase begin Receive into ghost cells from neighbors left recv MPI.Irecv view data, 1 , comm source mod rank 1, N , right recv MPI.Irecv view data, M 2 , comm source mod rank 1, N Send to neighbors left send MPI.Isend view data, 2 , comm dest mod rank 1, N , right send MPI.Isend view data, M 1 , comm dest mod rank 1, N block until communication is complete MPI.Waitall left recv, right recv, left send, right send end print \" rank Received data\\n\" Computation phase ... end md\"\"\" Collective communication and synchronization \"\"\" md\"\"\" MPI defines a series of \"collective\" operations, operations that all processes within a communicator partake in. We have already encountered the `MPI.Barrier` operation that forces all processes to wait. \"\"\" md\"\"\" One possible but inefficient way to implement a barrier is the ring communication example from above ```julia if rank 0 token MPI.recv comm source rank 1 print \"Rank rank received token token.\\n\" else token 1 end MPI.send token, comm dest mod rank 1, N Now rank 0 can receive the token if rank 0 token MPI.recv comm source N 1 print \"Rank rank received token token.\\n\" end ``` \"\"\" md\"\"\" We set up a chain of blocking operations and the program can only continue thereafter \"\"\" md\"\"\" Broadcast Sends data from 1 to N \"\"\" mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm if rank 0 x rand 3 else x zeros 3 end MPI.Bcast x, comm root 0 print \" rank Received x\\n\" end mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm function my bcast data, comm root 0 rank MPI.Comm rank comm N MPI.Comm size comm if rank root send data to all other ranks for other in 0 N 1 if other root MPI.Send data, comm dest other end end else MPI.Recv data, comm source root end end if rank 0 x rand 3 else x zeros 3 end my bcast x, comm root 0 print \" rank Received x\\n\" end md\"\"\" MPI implementation may choose their own more efficient implementations This implementation is inefficient since it is limited by the bandwidth of the root process. Better would be something like a communication tree \"\"\" bind np bench Select 2, 4, 8, 12 , default 4 avg my bcast mean x x 1 , bcast times avg mpi bcast mean x x 2 , bcast times avg mpi bcast avg my bcast bcast times mpi np bench let function my bcast data, comm root 0 rank MPI.Comm rank comm N MPI.Comm size comm if rank root send data to all other ranks for other in 0 N 1 if other root MPI.Send data, comm dest other end end else MPI.Recv data, comm source root end end function benchmark T, sz, iters comm MPI.COMM WORLD data zeros T, sz my bcast time 0.0 mpi bcast time 0.0 warmup my bcast data, comm MPI.Bcast data, comm for in 1 iters MPI.Barrier comm my bcast time MPI.Wtime my bcast data, comm MPI.Barrier comm my bcast time MPI.Wtime MPI.Barrier comm mpi bcast time MPI.Wtime MPI.Bcast data, comm MPI.Barrier comm mpi bcast time MPI.Wtime end my bcast time iters, mpi bcast time iters end MPI.Init benchmark Float64, 400000, 10 magical return to Pluto value end md\"\"\" Scatter Scatter is very similar to broadcast. Broadcast sends the same data to all processes. Scatter sends chunks of an array to different processes. \"\"\" mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm N MPI.Comm size comm x zeros Int, 3 data rank 0 ? collect 1 N 3 nothing MPI.Scatter data, x, comm root 0 print \" rank Received x\\n\" end md\"\"\" Gather Collect data from N to 1. note Gather is the inverse of scatter. \"\"\" md\"\"\" In MPI communication is explicit Let's generate `np` np random numbers and send them all to the root rank. \"\"\" mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm x rand xs rank 0 ? zeros MPI.Comm size comm nothing if rank 0 show xs end MPI.Gather Ref x , xs, comm root 0 if rank 0 show xs end x magical return to Pluto value end md\"\"\" note `Ref x ` is a one element container, like ` x ` \"\"\" mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm x rand xs MPI.Gather x , comm root 0 if rank 0 show xs end x end mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm x rand xs MPI.gather x, comm root 0 if rank 0 show xs end x end md\"\"\" MPI.jl often has three variants of functions `Gather ` Closest to the underlying MPI implementation, expects input and output buffer. `Gather` Convinience function that allocates on the receiving side, expects and input buffer. `gather` Handles arbitrary Julia objects and allocates on the receiving side. \"\"\" md\"\"\" Reduce `Gather` moves all values from all ranks to one root rank. Instead of copying values and then manually reducing them, MPI supports an `MPI.Reduce` call. The standard ships a set of predefined reduction operators — `MPI.SUM`, `MPI.PROD`, `MPI.MAX`, `MPI.MIN`, … — which the implementation can apply very efficiently. We use `MPI.SUM` below further down we will see that MPI.jl also lets us pass an arbitrary Julia function. \"\"\" md\"\"\" Compute \\int 0^1 \\frac 4 1 x^2 dx 4 atan x 0^1 which evaluates to π. Each rank sums its share of the terms into `mypi`. We then reduce these partials two ways the `MPI.Reduce` inside the program prints the error on the root rank, while we also return `mypi` so that the next Pluto cell can sum the partials `sum pis ` on the Julia side. The two should agree. \"\"\" pis mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm TODO Support interpolation into ` mpi` macro n 50 000 000 s 0.0 for i MPI.Comm rank comm 1 MPI.Comm size comm n x i .5 n s 4 1 x^2 end mypi s n our π MPI.Reduce mypi, MPI.SUM, comm root 0 if rank 0 println \"Error our π π our π π \" end mypi end our π sum pis mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm sum of ranks MPI.Reduce rank, , comm root 0 if rank 0 println \"sum of ranks sum of ranks \" end sum of ranks end md\"\"\" note We can also use any Julia function and not just the limited set of reduction operators defined by the standard. \"\"\" md\"\"\" Allgather and Allreduce `Gather` and `Reduce` are N to 1 collectives and they have sibling varieties that perform an all to all communication. \"\"\" mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm sum of ranks MPI.Allreduce rank, , comm if rank 0 println \"sum of ranks sum of ranks \" end sum of ranks end mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm ranks MPI.Allgather rank , comm ranks end md\"\"\" Communicators \"\"\" md\"\"\" So far we have been using ```julia comm MPI.COMM WORLD ``` without much explanation. `MPI.COMM WORLD` is the default communicator and contains all available ranks Other communicators can be used to partition our computation. `MPI.COMM SELF` Just ourselves `MPI.COMM TYPE SHARED` All ranks on one physical node `MPI.COMM NULL` Nobody We can split our communicator \"\"\" mpi np let MPI.Init world rank MPI.Comm rank MPI.COMM WORLD comm MPI.Comm split MPI.COMM WORLD, iseven world rank , world rank rank MPI.Comm rank comm print \"World rank world rank, Split rank rank\\n\" end md\"\"\" note `Comm split` is probably the easiest way of splitting a communicator, but modern MPI uses MPI Groups which are a bit more flexible. \"\"\" mpi np let MPI.Init world rank MPI.Comm rank MPI.COMM WORLD world group MPI.Comm group MPI.COMM WORLD odd ranks convert. Int32, collect 1 2 MPI.Comm size MPI.COMM WORLD odd group MPI.Group incl world group, odd ranks even group MPI.Group difference world group, odd group group iseven world rank ? even group odd group comm MPI.Comm create group MPI.COMM WORLD, group, tag 0 non collective only ranks in group must participate rank MPI.Comm rank comm print \"World rank world rank, Split rank rank\\n\" end md\"\"\" We can also duplicate communicators This can be important if you want to hand off a communicator to a library maybe one implemented in C Fortran and you are unable to negotiate a common tag scheme. Otherwise we might respond to messages meant for the library and not us \"\"\" mpi np let MPI.Init world rank MPI.Comm rank MPI.COMM WORLD comm MPI.Comm dup MPI.COMM WORLD rank MPI.Comm rank comm print \"World rank world rank, Other rank rank\\n\" end md\"\"\" Other examples \"\"\" md\"\"\" Custom ops and custom data types \"\"\" mpi np let using Statistics Define a custom struct This contains the summary statistics mean, variance, length of a vector struct SummaryStat mean Float64 var Float64 n Float64 end function SummaryStat X AbstractArray m mean X v varm X,m, corrected false n length X SummaryStat m,v,n end Define a custom reduction operator this computes the pooled mean, pooled variance and total length function pool S1, S2 n S1.n S2.n m S1.mean S1.n S2.mean S2.n n v S1.n S1.var S1.mean S1.mean m S2.n S2.var S2.mean S2.mean m n SummaryStat m,v,n end MPI.Init comm MPI.COMM WORLD root 0 X randn 10,3 . 1,3,7 ' Perform a scalar reduction summ MPI.Reduce SummaryStat X , pool, comm root if MPI.Comm rank comm root show summ.var end Perform a vector reduction the reduction operator is applied elementwise col summ MPI.Reduce mapslices SummaryStat,X,dims 1 , pool, comm root if MPI.Comm rank comm root col var map summ summ.var, col summ show col var end nothing end md\"\"\" Static Vectors \"\"\" mpi np let using StaticArrays MPI.Init comm MPI.COMM WORLD x ones SVector 3, Float64 sum MPI.Allreduce x , , comm if MPI.Comm rank comm 0 show sum end nothing end "},{"url":"mod3_parallelism/interactive_tuning/","title":"Interactive performance tuning with Julia","description":"","tags":["module3","track_performance"],"text":" A Pluto.jl notebook v1.0.1 frontmatter chapter \"3\" section \"1\" order \"8.2\" title \"Interactive performance tuning with Julia\" tags \"module3\", \"track performance\" layout \"layout.jlhtml\" indepth number \"3\" frontmatter.author name \"Mosè Giordano\" url \"https giordano.github.io \" using Markdown using InteractiveUtils This Pluto notebook uses bind for interactivity. When running this notebook outside of Pluto, the following 'mock version' of bind gives bound variables a default value instead of an error . macro bind def, element format off return quote local iv try Base.loaded modules Base.PkgId Base.UUID \"6e696c72 6542 2067 7265 42206c756150\" , \"AbstractPlutoDingetjes\" .Bonds.initial value catch b missing end local el esc element global esc def Core.applicable Base.get, el ? Base.get el iv el el end format on end TableOfContents md\"\"\" Interactive performance tuning with Julia note \"Thread pinning\" Remember to pin the threads of your programs This notebook runs on a Slurm cluster and uses `ThreadPinning.jl` https github.com carstenbauer ThreadPinning.jl to pin the Julia process to the CPU threads allocated by Slurm by using the `SLURM CPU BIND LIST` environment variable. Part of this presentation will use material from the workshop Julia for HPC UCL 2024 https github pages.arc.ucl.ac.uk julia hpc 2024 run by Carsten Bauer , the course material is available at carstenbauer JuliaUCL24 https github.com carstenbauer JuliaUCL24 . \"\"\" md\"\"\" warning This notebook uses and LIKWID & LinuxPerf.jl, which requires Linux \"\"\" using ThreadPinning ThreadPinning.pinthreads Int. log2. parse. BigInt, first split ENV \"SLURM CPU BIND LIST\" , ',' , Threads.nthreads warn false ThreadPinning.pinthreads cores with terminal ThreadPinning.threadinfo using CpuId cpuinfo Sys.CPU NAME md\"\"\" Simple linear algebra \"\"\" using LIKWID using LinuxPerf using PlutoUI using BenchmarkTools begin a T pi x rand T, N y rand T, N z zeros T, N end md\"\"\"T bind T Select Float32, Float64 \"\"\" let range trunc. Int, exp10. 0 0.2 5 md\"N bind N Slider range, default 10 000, show value true \" end function axpy z, a, x, y for idx in eachindex z, x, y z idx a x idx y idx end end benchmark axpy z, a, x, y md\"\"\" We can use `LIKWID.jl` https github.com JuliaPerf LIKWID.jl to measure the number of floating point operations performed in the `axpy ` function details below 👇 . \"\"\" N FLOPs first events perf group \"RETIRED SSE AVX FLOPS ALL\" N FLOPs per iteration N FLOPs N with terminal code llvm debuginfo none axpy z, a, x, y with terminal code native debuginfo none axpy z, a, x, y perf group T Float32 ? \"FLOPS SP\" \"FLOPS DP\" metrics, events perfmon perf group axpy z, a, x, y first metrics perf group md\"\"\"We can also access Linux's `perf` https perf.wiki.kernel.org index.php Main Page performance counters via `LinuxPerf.jl` https github.com JuliaPerf LinuxPerf.jl .\"\"\" measure axpy z, a, x, y perf events \" cache references,cache misses \" pstats perf events axpy z, a, x, y md\"\"\" Parallelising the `sum` ...not so fast. \"\"\" data rand 1 000 000 Threads.nthreads sum data benchmark sum data using Base.Threads using ChunkSplitters function sum threads chunks data nchunks nthreads psums zeros eltype data , nchunks threads for c, idcs in enumerate chunks data n nchunks for i in idcs psums c data i end end return sum psums end sum data ≈ sum threads chunks data benchmark sum threads chunks data pstats perf events sum data pstats perf events sum threads chunks data md\"High cache trashing frequency false sharing https en.wikipedia.org wiki False sharing https d3i71xaburhd42.cloudfront.net cb149ebdbe097867a3b307a0f8d24c5867c0aa68 19 Figure4 1.png Good solution Create a new local accumulator inside the `for` loop, and update the per chunk accumulator at the end of the inner loop.\" function sum threads chunks local data nchunks nthreads psums zeros eltype data , nchunks threads for c, idcs in enumerate chunks data n nchunks local s zero eltype data for i in idcs s data i end psums c s end return sum psums end sum data ≈ sum threads chunks local data benchmark sum threads chunks local data pstats perf events sum threads chunks local data md\"\"\" Better solution Don't do the manual accumulation, use tasks. \"\"\" function sum map spawn data nchunks nthreads ts map chunks data, n nchunks do idcs spawn views sum data idcs end return sum fetch. ts end sum data ≈ sum map spawn data benchmark sum map spawn data pstats perf events sum map spawn data let range trunc. Int, exp10. 0 0.2 9 md\"\"\" large\\ N bind large N Slider range, default 10, show value true \"\"\" end "},{"url":"mod3_parallelism/shared-memory/","title":"Shared-memory parallelism","description":"","tags":["module3","track_parallel"],"text":" A Pluto.jl notebook v1.0.1 frontmatter chapter \"3\" section \"1\" order \"8\" title \"Shared memory parallelism\" date \"2026 05 27\" tags \"module3\", \"track parallel\" layout \"layout.jlhtml\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils begin using PlutoUI, PlutoTeachingTools using PlutoUI Slider PlutoUI.TableOfContents depth 4 end ChooseDisplayMode md\"\"\" Shared Memory parallelism This notebook uses Threads.nthreads threads \"\"\" with terminal do Sys.cpu summary end using Hwloc with terminal do topology info end md\"\"\" CPU architecture \"\"\" md\"\"\" In Introduction to Parallelism .. mod1 introduction parallelism we briefly touched on computer architecture. One of the more peculiar issues are so called Hyperthreads HT . Hyperthread is the Intel brand name for Simultaneous multithreading SMT . SMT N provides N logical CPU cores per physical CPU core. Some platforms may even make N configurable. On Intel and AMD systems N 2, but systems with higher N are available. The core idea is that memory accesses are slow and superscalar CPUs can have many instructions in flight at once. The CPU architecture exposes a limited amount of resources registers that can be used, but the actual implementation in hardware may have an abundance of resources. This can be used to execute operations that can be speculatively executed or we can execute multiple threads on the same physical core. note For our purposes the distinction can be mostly ignored, but we should not expect perfect scaling when exceeding the number of physical cores. \"\"\" using ThreadPinning with terminal do threadinfo end md\"\"\" Julia tasks In Julia concurrent and parallel programming is done through tasks . Tasks are independent and communication units of computation. They execute always concurrently, but not always parallel. info \"Further reading\" https docs.julialang.org en v1 manual asynchronous programming https proceedings.juliacon.org papers 10.21105 jcon.00054 \"\"\" import Base.Threads sync, spawn function fib x if x 1 return 1 else b spawn fib x 2 a fib x 1 return a fetch b Int end end fib 5 md\"\"\" note A core principle at play is the notion of divide and conquer. Can I split a problem into smaller sub problems? Often, in parallel computing we split problems until a grain size is reached for which we execute a base case. Today's exercise \"Parallel sorting\" is an example of divide and conquer \"\"\" md\"\"\" question The Fibonacci function above is just an example to demonstrate basic ideas of parallel programming. How would you implement it if you were interested in getting the result as fast as possible? \"\"\" md\"\"\" Channels \"\"\" md\"\"\" As mentioned Julia tasks are communicating , communication can happen with dedicated programming concepts like `Channel`, or directly through memory shared with another tasks. Channels are first in first out queues that can either be buffered e.g. contain a reservoir for a number of elements , or un buffered blocking. Julia's channels support `put `, `take ` \"\"\" let ch Channel Int Inf buffered sync begin for i in 1 10 spawn put ch, rand Int end end close ch Otherwise collect will wait for more data collect ch end md\"\"\" Race conditions `Channel` is a concurrent data structure and ensure that it safe to use with multiple tasks. When we use our own data structures we have to make sure that we make them safe if necessary, otherwise we will observe data races. \"\"\" mutable struct BrokenCounter x Int end let a BrokenCounter 0 N 10 K 100 000 sync for i in 1 N spawn for i in 1 K a.x 1 GC.safepoint end end a.x, N K, a.x N K end md\"\"\" note We launch `N` tasks each performing `K` updates to our counter. We expect `N K` updates, but we only observe a fraction of them. ```julia a.x 1 Can be written as a.x a.x 1 ``` But in the time between reading and writing `a.x`, there might have been another update. This is a classical race condition. \"\"\" md\"\"\" Atomics & Locks One way this can be fixed is to use atomics. Atomics allow to express the `read and increment` operation as one operation. info \"Further reading\" https marabos.nl atomics https redixhumayun.github.io systems 2024 01 03 atomics and concurrency.html \"\"\" mutable struct AtomicCounter atomic x Int end let a AtomicCounter 0 N 10 K 100 000 sync for i in 1 N spawn for i in 1 K atomic a.x 1 GC.safepoint end end a.x, N K, a.x N K end let a AtomicCounter 0 a.x 10 end let a AtomicCounter 0 atomic sequentially consistent a.x 1 2 end md\"\"\" We can also fix this using a lock. note `Lockable` was added in Julia 1.11. \"\"\" let a Base.Lockable BrokenCounter 0 , Base.ReentrantLock N 10 K 100 000 sync for i in 1 N spawn for i in 1 K lock a, a .x 1 GC.safepoint end end lock a begin a .x, N K, a .x N K end end md\"\"\" Atomics are limited to things that are \"small\" For fields Julia might put a lock variable next to the field, if the value is \"big\" . And locks are an easier way to provide access to things that should be exclusive access, but are more expensive than atomics. \"\"\" md\"\"\" Atomics on Arrays Not yet part of the base language. Use the package `Atomix`. \"\"\" using Atomix Atomix, atomic, atomicswap, atomicreplace let A ones Int, 3 Atomix. atomic A 1 1 fetch and increment A end md\"\"\" Parallel loops \"\"\" import Base.Threads threads, nthreads, threadid let a zeros Int, nthreads 2 threads for i in 1 length a a i threadid end a end md\"\"\" note `threadid` is a implementation detail, and should not be used as part of algorithms. Tasks migrate across threads, so it is not a stable observation. \"\"\" md\"\"\" Schedulers Julia has different schedulers for parallel for loops ` dynamic` the default . Chunks the iteration space. ` greedy` One task per thread, good for unequal workloads. Iteration space is interpreted as a channel. ` static` One task per thread, equal division of iteration space. Can not be nested. \"\"\" md\"\"\" Parallel primitives \"\"\" using OhMyThreads md\"\"\" While ` threads` is on the surface an acceptable interface, it is often cumbersome to implement reductions. There are several libraries that provide high level parallel primitives based on higher order functions functions that take other functions . `map f, A ` `reduce , A ` `mapreduce f, , A ` `OhMyThreads.jl` provides `tmap` `treduce` `tmapreduce` `tforeach` note It is a bit unfortunate that `map` & co are implicitly parallel on the GPU, but not on the CPU. Fixing this as been a long standing ToDo. \"\"\" let a zeros Int, nthreads 2 tforeach 1 length a do i a i threadid end a end md\"\"\" False sharing Based on the OhMyThreads.jl docs https juliafolds2.github.io OhMyThreads.jl stable literate falsesharing falsesharing . \"\"\" using BenchmarkTools data rand 1 000 000 nthreads md\"\"\" Baseline sequential sum \"\"\" function simple sum data acc zero eltype data for i in eachindex data acc data i end acc end benchmark sum data benchmark simple sum data question box md\"\"\" Note that our simple sum is slower than Julia's sum Why might that be the case? \"\"\" md\"\"\" Naive parallel implementation We allocate space for the intermediate results. \"\"\" function parallel sum falsesharing data nchunks nthreads psums zeros eltype data , nchunks sync for c, idcs in enumerate OhMyThreads.index chunks data n nchunks spawn begin for i in idcs psums c data i end end end return sum psums end sum data ≈ parallel sum falsesharing data benchmark parallel sum falsesharing data md\"\"\" Oof our parallel code is slower than our serial code The core issue is that we directly update the values in memory and this causes the CPU to bounce the cache lines from core to core \"\"\" md\"\"\" Padding to cache line size One way of solving this is to over allocate memory and pad each sum such that it is occurring on different cache lines. `std hardware destructive interference size` https en.cppreference.com w cpp thread hardware destructive interference size.html \"\"\" const CACHE LINE SIZE 64 function parallel sum padded data nchunks nthreads pad each entry stride CACHE LINE SIZE ÷ sizeof eltype data psums zeros eltype data , nchunks stride sync for c, idcs in enumerate OhMyThreads.index chunks data n nchunks spawn begin c idx c 1 stride 1 for i in idcs psums c idx data i end end end return sum psums end benchmark parallel sum padded data md\"\"\" note To make our implementation faster, we should first think about how we could make our `simple sum` implementation faster. \"\"\" md\"\"\" Task local parallel summation Another way of solving this is to do a local reduction first \"\"\" function parallel sum tasklocal data nchunks nthreads psums zeros eltype data , nchunks sync for c, idcs in enumerate OhMyThreads.index chunks data n nchunks spawn begin local s zero eltype data for i in idcs s data i end psums c s end end return sum psums end benchmark parallel sum tasklocal data md\"\"\" Reuse efficient local implementation \"\"\" function parallel sum map data nchunks nthreads psums zeros eltype data , nchunks sync for c, idcs in enumerate OhMyThreads.index chunks data n nchunks spawn begin psums c sum view data, idcs end end return sum psums end benchmark parallel sum map data md\"\"\" Of course we can just use `OhMyThreads.treduce` \"\"\" benchmark treduce , data ntasks nthreads "},{"url":"mod4_automatic_differentiation/ad_in_science/","title":"Examples of Automatic Differentiation in Scientific Computing","description":"","tags":["module4","track_ad"],"text":" A Pluto.jl notebook v1.0.1 frontmatter chapter \"4\" section \"2\" order \"12\" title \"Examples of Automatic Differentiation in Scientific Computing\" date \"2026 07 01\" tags \"module4\", \"track ad\" layout \"layout.jlhtml\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils This Pluto notebook uses bind for interactivity. When running this notebook outside of Pluto, the following 'mock version' of bind gives bound variables a default value instead of an error . macro bind def, element format off return quote local iv try Base.loaded modules Base.PkgId Base.UUID \"6e696c72 6542 2067 7265 42206c756150\" , \"AbstractPlutoDingetjes\" .Bonds.initial value catch b missing end local el esc element global esc def Core.applicable Base.get, el ? Base.get el iv el el end format on end begin using PlutoUI, PlutoTeachingTools using PlutoUI Slider PlutoUI.TableOfContents depth 4 end begin using CairoMakie set theme theme latexfonts fontsize 16, Lines linewidth 2, , markersize 16 end begin using Enzyme using Enzyme Reverse import ForwardDiff end using Symbolics using ShortCodes using SparseArrays using LinearAlgebra using Krylov using SparseConnectivityTracer begin using SparseMatrixColorings import Images end ChooseDisplayMode md\"\"\" Motivation In science and engineering we often need to solve systems of equations . If the equations are linear then linear algebra tells us a general method to solve them these are now routinely applied to solve systems of millions of linear equations. If the equations are nonlinear then things are less obvious. The main solution methods we know work by... reducing the nonlinear equations to a sequence of linear equations They do this by approximating the function by a linear function and solving that to get a better solution, then repeating this operation as many times as necessary to get a sequence of increasingly better solutions. This is an example of an iterative algorithm . A well known and elegant method, which can be used in many different contexts, is the Newton method . It does, however, have the disadvantage that it requires derivatives of the function. This can be overcome using automatic differentiation techniques. \"\"\" md\"\"\" The Newton method in 1D Based on https featured.plutojl.org computational thinking newton.html \"\"\" md\"\"\" We would like to solve equations like f x g x . We rewrite that by moving all the terms to one side of the equation so that we can write h x 0 , with h x f x g x . A point x^ such that h x^ 0 is called a root or zero of h . The Newton method finds zeros, and hence solves the original equation. \"\"\" md\"\"\" The idea of the Newton method is to follow the direction in which the function is pointing We do this by building a tangent line at the current position and following that instead, until it hits the x axis. Let's look at that visually first \"\"\" md\"\"\" n bind n2 Slider 0 10, show value true, default 1 \"\"\" md\"\"\" x₀ bind x02 Slider 10 10, show value true, default 6 \"\"\" md\"\"\" n bind n Slider 0 10, show value true, default 1 \"\"\" md\"\"\" x₀ bind x0 Slider 10 10, show value true, default 6 \"\"\" straight x0, y0, x, m y0 m x x0 function standard Newton f, n, x range, x0, ymin 10, ymax 10 f′ x ForwardDiff.derivative f, x fig Figure size 400, 300 ax Axis fig 1,1 ylims ax, ymin, ymax lines ax, x range, f hlines ax, 0.0, color magenta, linestyle dash lw 3 scatter ax, x0, 0, color green annotation ax, x0, 5, text L\"x 0\" for i in 1 n lines ax, x0, x0 , 0, f x0 , color gray, alpha 0.5 scatter ax, x0, f x0 , color red m f′ x0 lines ax, x range, straight x0, f x0 , x, m for x in x range , color blue, alpha 0.5, linestyle dash lw 2 x1 x0 f x0 m scatter ax, x1, 0, color green , ann x1, 5, L\"x % i\", 10 annotation ax, x1, 5, text L\"x % i\" x0 x1 end fig end let f x x^2 2 standard Newton f, n2, 1 0.01 10, x02, 10, 70 end let f x 0.2x^3 4x 1 standard Newton f, n, 10 0.01 10, x0, 10, 70 end md\"\"\" Mathematics of the Newton method \"\"\" md\"\"\" We can convert the idea of \"following the tangent line\" into equations as follows. You can also do so by just looking at the geometry in 1D, but that does not help in 2D. \"\"\" md\"\"\" Suppose we have a guess x 0 for the root and we want to find a hopefully better guess x 1 . Let's set x 1 x 0 \\delta , where x 1 and \\delta are still unknown. We want x 1 to be a root, so \"\"\" md\"\"\" f x 1 f x 0 \\delta \\simeq 0 \"\"\" md\"\"\" If we are already \"quite close\" to the root then \\delta should be small, so we can approximate f using the tangent line f x 0 \\delta \\, f' x 0 \\simeq 0 and hence \\delta \\simeq \\frac f x 0 f' x 0 so that x 1 x 0 \\frac f x 0 f' x 0 Now we can repeat so that x 2 x 1 \\frac f x 1 f' x 1 and in general x n 1 x n \\frac f x n f' x n . This is the Newton method in 1D. \"\"\" function newton1D f, x0 f′ x ForwardDiff.derivative f, x x x0 for i in 1 10 x f x f′ x end return x end newton1D x x^2 2, 37.0 sqrt 2 md\"\"\" Extending Newton to 2D T \\mathbb R ^2 \\to \\mathbb R ^2 \"\"\" md\"\"\" We want to find the inverse T^ 1 y , i.e. to solve the equation T x y for x . We use the same idea as in 1D, but now in 2D \"\"\" md\"\"\" T x 0 \\delta \\simeq 0 T x 0 J \\cdot \\delta \\simeq 0, where J DT x 0 is the Jacobian matrix of T at x 0 , i.e. the best linear approximation of T near to x 0 . \"\"\" md\"\"\" Hence \\delta is the solution of the system of linear equations \"\"\" md\"\"\" J \\cdot \\delta T x 0 Then we again construct the new approximation x 1 as x 1 x 0 \\delta . \"\"\" function newton2D step T, x J ForwardDiff.jacobian T, x δ J \\ T x J^ 1 T x return x δ end \"Find ``x`` such that ``T x 0``\" function newton2D T, x0, n 10 x x0 for i in 1 n x newton2D step T, x end return x end md\"\"\" Remember that Newton is designed to look for roots , i.e. places where T x 0 . We want T x y , so we need another layer \"\"\" begin \"Looks for ``x`` such that ``f x y``, i.e. ``f x y 0``\" function inverse f, y, x0 0, 0 return newton2D x f x y, x0 end inverse f y inverse f, y end T α x, y , x α y^2, y α x^2 md\"\"\" α bind α Slider 0.0 0.01 1.0, show value true \"\"\" T α 0.3, 0.4 T α ∘ inverse T α 0.3, 0.4 Essentially I v md\"\"\" Jacobian f \\mathbb R ^n\\rightarrow\\mathbb R ^m ```math J \\begin bmatrix \\frac \\partial f 1 \\partial x 1 & \\cdots & \\frac \\partial f 1 \\partial x n \\\\ 1ex % 1ex more space between rows of matrix \\vdots & \\ddots & \\vdots \\\\ 1ex \\frac \\partial f m \\partial x 1 & \\cdots & \\frac \\partial f m \\partial x n \\end bmatrix ``` \"\"\" md\"\"\" Let's take a function `F` from \\mathbb R ^2\\rightarrow\\mathbb R ^3 as a concrete example. \"\"\" function F X X 1 ^4 3 exp X 2 2 log X 1 X 2 ^2 end md\"\"\" Manual derivation Symbolics \"\"\" begin variables X 1 2 ex F F X ex J Symbolics.jacobian ex F, X scalarize Val false end md\"\"\" note Above we have the general form of the Jacobian of `F`, but more often than not we care about the Jacobian of `F` at a point `x`. \"\"\" substitute. ex J, X 1.0, 1.0 function F2 X X 1 ^4 3 X 2 0 ? X 2 exp X 2 2 log X 1 X 2 ^2 end md\"\"\" note When we have functions with control flow that depend on an input, the symbolic approach fails. \"\"\" begin ex F2 F2 X ex J2 Symbolics.jacobian ex F2, X scalarize false end md\"\"\" Using ForwardDiff \"\"\" ForwardDiff.jacobian F, 1.0,1.0 ForwardDiff.jacobian F2, 1.0, 1.0 md\"\"\" note Automatic differentiation always evaluates things at a point, we can ask for the Jacobian of a function with input dependent control flow. \"\"\" ForwardDiff.jacobian F, 1.0, 1.0 ForwardDiff.jacobian F2, 1.0, 1.0 md\"\"\" Using Enzyme \"\"\" Enzyme.jacobian Forward, F, 1.0, 1.0 | only Enzyme.jacobian Reverse, F, 1.0, 1.0 | only md\"\"\" note Both Jacobians are equivalent, but the object computed by Reverse mode is the transpose. \"\"\" let J Enzyme.jacobian Reverse, F, 1.0, 1.0 | only v 1.0, 0.0 J v end md\"\"\" Mutation \"\"\" begin function G Y, X Y 1 X 1 ^4 3 Y 2 exp X 2 2 Y 3 log X 1 X 2 ^2 end function G X Y similar X, 3 G Y, X Y end end ForwardDiff.jacobian G, 1.0, 1.0 ForwardDiff.jacobian zeros 3, 2 , G , zeros 3 , 1.0, 1.0 J Enzyme.jacobian Reverse, G, 1.0, 1.0 | only md\"\"\" Jacobian vector product \"\"\" md\"\"\" The Jacobian vector product or directional derivative is a powerful primitive. We implicitly perform a Jv operation at the point u . With Enzyme this is a direct application of `autodiff` in Forward mode. \"\"\" function JVP f F, u, v, ϵ NaN where F return only Enzyme.autodiff Forward, f, Duplicated, Duplicated u, v end md\"\"\" Approximation In many texts you will find the directional derivatives approximated using a finite difference operation. \"\"\" md\"\"\" First order finite difference \\frac F u \\epsilon \\cdot v F u \\epsilon \"\"\" First order Taylor series function JVP Finite Diff 1st F, u, v, ϵ sqrt eps F u ϵ . v F u . ϵ end md\"\"\" Second order finite difference \\frac F u \\epsilon \\cdot v F u \\epsilon \\cdot v 2\\epsilon \"\"\" Second order Taylor series Central finite difference function JVP Finite Diff 2nd F, u, v, ϵ cbrt eps 2 F u ϵ . v F u ϵ . v . 2ϵ end md\"\"\" Choosing ϵ warning A core challenge is to choose the right ϵ for the problem at hand. Common For 1st order ϵ \\sqrt \\epsilon mach For 2nd order ϵ \\sqrt 3 \\epsilon mach 2 Input dependent For 1st order ϵ \\frac \\sqrt 1 ||u|| \\epsilon mach ||v|| after DOI \"10.1016 j.jcp.2003.08.010\" For 2nd order ϵ \\frac \\sqrt 1 ||u|| ||v|| \\sqrt 3 \\epsilon mach after DOI \"10.1016 j.cam.2011.09.003\" \"\"\" JVP Finite Diff 1st F, 1.0 1.0 , 1.0 0.0 JVP Finite Diff 2nd F, 1.0 1.0 , 1.0 0.0 md\"\"\" Example Truncated Weierstrass \"\"\" md\"\"\" Truncated Weierstrass f x \\sum k 0 ^ N a^k \\cos b^k \\pi x f' x \\sum k 0 ^ N a^k b^k \\pi \\sin b^k \\pi x With a \\in 0,1 and 1 ab \"\"\" f x, a 0.3, b 3.0 N 10 sum a^k cos b^k π x for k in 0 N f′ x, a 0.3, b 3.0 N 10 sum a^k b^k π sin b^k π x for k in 0 N let xs LinRange 1.0, 1.0, 10000 fig Figure size 800, 800 lines fig 1, 1 , xs, x f x, N 3 , axis title \"Function N 3\", ylabel L\" f x \", xlabel L\" x \" lines fig 2, 1 , xs, x f x, N 10 , axis title \"Function N 10\", ylabel L\" f x \", xlabel L\" x \" lines fig 1, 2 , xs, x f′ x, N 3 , axis title \"Derivative N 3\", ylabel L\" f' x \", xlabel L\" x \" lines fig 2, 2 , xs, x f′ x, N 10 , axis title \"Derivative N 10\", ylabel L\" f' x \", xlabel L\" x \" fig end function rel error f, f′, x, method, ϵ y′ f′ x y method f, x, one x , ϵ max eps 4, abs y′ y y′ Avoid error 0 to cause issues in log plot end let ϵs 10.0^ x for x in 30 0.1 0 ϵs BigFloat. 10.0^ x for x in 30 0.1 0 x 0.2 fig Figure size 800, 800 ax Axis fig 1,1 , xscale log10, yscale log10, title \"Relative error N 3\", ylabel \"Relative error\", xlabel \"Difference step size\" ylims ax, 1e 20, 10e1 lines ax, ϵs, ϵ rel error x f x, N 3 , x f′ x, N 3 , x, JVP Finite Diff 1st, ϵ , label \"FFD 1st order\" lines ax, ϵs, ϵ rel error x f x, N 3 , x f′ x, N 3 , x, JVP Finite Diff 2nd, ϵ , label \"CFD 2nd order\" lines ax, ϵs, ϵ rel error x f x, N 3 , x f′ x, N 3 , x, JVP, ϵ , label \"AD\" vlines ax, sqrt eps , label L\" \\sqrt \\epsilon mach \", color \"red\",ymin 1 vlines ax, cbrt eps 2 , label L\" \\sqrt 3 \\epsilon mach 2 \", color \"orange\",ymin 1 axislegend ax, position lt ax Axis fig 2,1 , xscale log10, yscale log10, title \"Relative error N 10\", ylabel \"Relative error\", xlabel \"Difference step size\" ylims ax, 1e 20, 10e1 lines ax, ϵs, ϵ rel error x f x, N 10 , x f′ x, N 10 , x, JVP Finite Diff 1st, ϵ , label \"FFD 1st order\" lines ax, ϵs, ϵ rel error x f x, N 10 , x f′ x, N 10 , x, JVP Finite Diff 2nd, ϵ , label \"CFD 2nd order\" lines ax, ϵs, ϵ rel error x f x, N 10 , x f′ x, N 10 , x, JVP, ϵ , label \"AD\" vlines ax, sqrt eps , label L\" \\sqrt \\epsilon mach \", color \"red\", ymin 1 vlines ax, cbrt eps 2 , label L\" \\sqrt 3 \\epsilon mach 2 \", color \"orange\", ymin 1 axislegend ax, position lt fig end md\"\"\" Extracting a Jacobian \"\"\" ex F question box md\"\"\" Are the zeros here real structural zeros, or are they computational zero? \"\"\" ex J md\"\"\" Inadmissibility \"\"\" md\"\"\" note Why is there a NaN? \"\"\" ex F md\"\"\" ``` Base.log x Dual Dual log x.value , x.deriv x.value Base. x Dual, y Dual Dual x.value y.value, x.deriv y.deriv ``` \"\"\" 0.0 0.0 derivative of log x 0.0 0.0 0.0 2.0 md\"\"\" note Finite differences can struggle with \"inadmissible\" values. Recall the definition of `F`. We calculate log X 1 If X 1 0 then the 2nd order method will go a little bit to the left and right of zero. `log 0 eps ` is not defined. \"\"\" ex F md\"\"\" ``` log 0 eps ``` \"\"\" md\"\"\" Using Enzyme generalized w J v ``` autodiff Forward, G , Duplicated out, w , Duplicated in, v ``` w v J ``` autodiff Reverse, G , Duplicated out, copy v , Duplicated in, w ``` note Recall from last week that Enzyme zeros the shadow inputs. Necessitating the `copy v ` \"\"\" md\"\"\" warning That's clearly wrong \"\"\" let out zeros 3 w zeros 2 autodiff Reverse, G , Duplicated out, 0.0,0.0,0.0 , Duplicated 1.0, 1.0 , w w end md\"\"\" Uhm... All the seeds are zero... So why do we have a gradient ? \"\"\" typeof G zeros 3 , 1.0, 1.0 md\"\"\" Due to historic reasons Enzyme infers functions with floating point return as \"active return\" and provides an implicit seed of 1.0. \"\"\" let out zeros 3 w zeros 2 autodiff Reverse, G , Const, Duplicated out, 0.0,0.0,0.0 , Duplicated 1.0, 1.0 , w w end md\"\"\" Sanity restored... \"\"\" md\"\"\" Jacobian Operator Given a problem `F u 0` we say that `F u ` calculates the residual . We want to have an operator `J F, u ` that represents the Jacobian of `F` at `u`. \"\"\" struct JacobianOperator F, A f F F out, in out A in A function JacobianOperator f F, out, in where F return new F, typeof in f, out, in end end begin Base.size J JacobianOperator length J.out , length J.in Base.eltype J JacobianOperator eltype J.in end function assemble jacobian F, u, JVP N length u v zero u out JVP F, u, v M length out J SparseMatrixCSC eltype u , Int undef, M, N for j in 1 N v . 0 v j 1 out JVP F, u, v for i in 1 M if out i 0 J i, j out i end end end J end assemble jacobian F, 1.0 1.0 , JVP assemble jacobian F, 1.0 1.0 , JVP Finite Diff 1st assemble jacobian F, 1.0 1.0 , JVP Finite Diff 2nd assemble jacobian F, 1.0 0.0 , JVP assemble jacobian F, 1.0 0.0 , JVP Finite Diff 1st assemble jacobian F, 1.0 0.0 , JVP Finite Diff 2nd assemble jacobian F, 0.0 1.0 , JVP assemble jacobian F, 0.0 1.0 , JVP Finite Diff 1st assemble jacobian F, 0.0 1.0 , JVP Finite Diff 2nd begin LinearAlgebra.adjoint J JacobianOperator Adjoint J LinearAlgebra.transpose J JacobianOperator Transpose J end let J Enzyme.jacobian Reverse, F, 1.0, 1.0 | only v 1.0, 0.0, 0.0 transpose J v vJ end begin v 1.0, 0.0, 0.0 transpose J v end let out zeros 3 w zeros 2 autodiff Reverse, G , Duplicated out, copy v , Duplicated 1.0, 1.0 , w w end let out zeros 3 w zeros 2 autodiff Reverse, G , Const, Duplicated out, copy v , Duplicated 1.0, 1.0 , w w end function maybe duplicated f, df if Enzyme.Compiler.guaranteed const typeof f return DuplicatedNoNeed f, df else return Const f end end function LinearAlgebra.mul out, J JacobianOperator, v autodiff Forward, maybe duplicated J.f, Enzyme.make zero J.f , Const, Duplicated J.out, out , Duplicated J.in, v return nothing end function LinearAlgebra.mul out, J′ Union Adjoint Any, JacobianOperator , Transpose Any, JacobianOperator , v J parent J′ If out is non zero we get spurious gradients out . 0 autodiff Reverse, maybe duplicated J.f, Enzyme.make zero J.f , Const, Enzyme zeros input derivatives and that confuses the solvers. Duplicated J.out, copy v , Duplicated J.in, out return nothing end function Base.collect JOp JacobianOperator N, M size JOp v zeros eltype JOp , M out zeros eltype JOp , N J SparseMatrixCSC eltype v , Int undef, size JOp ... for j in 1 M out . 0.0 v . 0.0 v j 1.0 mul out, JOp, v for i in 1 N if out i 0 J i, j out i end end end return J end JOp JacobianOperator G , zeros 3 , 1.0, 1.0 let w zeros 3 mul w, JOp, 1.0, 0.0 w end let w zeros 2 mul w, transpose JOp , 1.0, 0.0, 0.0 w end md\"\"\" Newton Krylov F u 0 J F, u \\times x F u Problems of the form F u 0 can be solved with a Newton method, this is particularly helpful for finding an iterative solution of nonlinear equations. We can use a Krylov solver \"gmres\" to solve the linear system Ax b without \"materializing\" A . This is called a matrix free method. \"\"\" begin function G2 y, x y 1 x 1 ^2 x 2 ^2 2 y 2 exp x 1 1 x 2 ^2 2 end function G2 x y similar x G2 y, x return y end end let xs LinRange 3, 4, 1000 ys LinRange 5, 5, 1000 levels 0.1, 0.25, 0.5 2 10..., 10.0 10 200..., 200 100 4000... fig, ax contour xs, ys, x, y norm G2 x, y levels fig end function newton krylov F , u tol rel 1.0e 6, tol abs 1.0e 12, max niter 50, callback x nothing res similar u F res, u n res norm res callback u tol tol rel n res tol abs iter 0 while n res tol && iter max niter Ax b J JacobianOperator F , res, u x, stats gmres J, res Take a step in the newton direction u . x F res, u n res norm res callback u if isinf n res || isnan n res error \"Solver blew up\" end iter 1 end return u, solved n res tol, iter end trace 1 let x₀ 2.0, 0.5 xs Vector Tuple Float64, Float64 undef, 0 hist x push xs, x 1 , x 2 nothing newton krylov G2 , x₀, callback hist xs end trace 2 let x₀ 2.5, 3.0 xs Vector Tuple Float64, Float64 undef, 0 hist x push xs, x 1 , x 2 nothing newton krylov G2 , x₀, callback hist xs end G2 trace 2 end ... let xs LinRange 3, 4, 1000 ys LinRange 5, 5, 1000 levels 0.1, 0.25, 0.5 2 10..., 10.0 10 200..., 200 100 4000... fig, ax contour xs, ys, x, y norm G2 x, y levels lines ax, trace 1 lines ax, trace 2 fig end md\"\"\" Coloring \"\"\" md\"\"\" Matrix free methods are powerful especially for systems that are 1. Large 2. Unknown structure of the Jacobian But special care is needed to make sure that the number of Jacobian Vector product evaluations is low. In particular for Newton Krylov one has to use inexact Newton Krylov with an Eisenstat Walker condition. Matrix coloring is an alternative to reduce the cost of calculating the Jacobian at each iteration. \"\"\" md\"\"\" note Adrian Hill and Guillaume Dalle have done some great work on this in the Julia AD landscape. 1. https iclr blogposts.github.io 2025 blog sparse autodiff 2. https arxiv.org abs 2501.17737 warning Remember the issues we had earlier with \"structural sparsity\" and \"input dependent computation\". These issues are particularly pernicious for automatic sparsity detection. \"\"\" jacobian sparsity F, 1.0, 1.0 , TracerSparsityDetector jacobian sparsity F, 1.0, 0.0 , TracerSparsityDetector jacobian sparsity F, 1.0, 0.0 , TracerLocalSparsityDetector jacobian sparsity x x 1 x 2 , 1.0, 0.0 , TracerLocalSparsityDetector jacobian sparsity x x 1 x 2 , 0.0, 1.0 , TracerLocalSparsityDetector jacobian sparsity x x 1 x 2 , 1.0, 1.0 , TracerLocalSparsityDetector jacobian sparsity x x 1 x 2 , 0.0, 0.0 , TracerSparsityDetector jacobian sparsity F2, 1.0, 1.0 , TracerSparsityDetector jacobian sparsity F2, 1.0, 1.0 , TracerLocalSparsityDetector md\"\"\" Recall the diffuse example from the beginning of term. \"\"\" function heat 1D du, u, a, Δx , t N length u Enforce the boundary condition u 1 0 u end 0 du 1 0 du end 0 Only compute within for i in 2 N 1 du i a u i 1 2u i u i 1 Δx^2 end return end function heat 1D u du similar u heat 1D du, u, 0.01, 0.01 , 0.01 return du end md\"\"\" What is its Jacobian? \"\"\" ForwardDiff.jacobian heat 1D, ones 12 assemble jacobian heat 1D, ones 12 , JVP J heat 1D sparsity jacobian sparsity heat 1D, ones 12 , TracerSparsityDetector result coloring J heat 1D sparsity, ColoringProblem , GreedyColoringAlgorithm SparseMatrixColorings.show colors result 1 ncolors result column colors result function assemble jacobian with colors F, u, JVP sparsity jacobian sparsity F, u, TracerSparsityDetector result coloring sparsity, ColoringProblem , GreedyColoringAlgorithm N length u v zero u out JVP F, u, v M length out J Matrix eltype u undef, M, ncolors result for c in 1 ncolors result v . 0 for j in 1 N if c column colors result j v j 1 end end J ,c JVP F, u, v end decompress J, result end assemble jacobian with colors heat 1D, ones 12 , JVP md\"\"\" note Instead of doing `length u ` JVPs, we are only doing `ncolors`. As long as the structure of our Jacobian is stable we can reuse the color information to extract it efficiently. \"\"\" "},{"url":"mod4_automatic_differentiation/lagrangian_ode/","title":"Lagrangian Mechanics via AD","description":"","tags":["module4","track_ad"],"text":" A Pluto.jl notebook v1.0.1 frontmatter chapter \"4\" section \"2\" order \"12.1\" title \"Lagrangian Mechanics via AD\" tags \"module4\", \"track ad\" layout \"layout.jlhtml\" indepth number \"4\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils begin using PlutoUI using CairoMakie, Enzyme, OrdinaryDiffEqTsit5, RecursiveArrayTools using LinearOperators, Krylov import Enzyme Reverse end md\"\"\" Acknowledgement Based on material from Simone Chiocchetti \"\"\" md\"\"\" The Euler Lagrange equations Given L kinetic\\ energy potential\\ energy of the system, the Euler Lagrange equations of motion are, by definition of \\dot q \\frac d dt q \\dot q \\frac dL dq \\frac d dt \\frac dL d\\dot q 0 the equation of motion can be cast by chain rule as \\frac dL dq \\frac d d\\dot q \\frac dL d\\dot q \\frac d\\dot q dt 0 \\frac dL dq \\frac d^2L d\\dot q ^2 \\frac d\\dot q dt 0 \\frac dL dq H \\frac d\\dot q dt 0 which gives \\frac d\\dot q dt H^ 1 \\frac dL dq \"\"\" md\"\"\" Motivation Of course this will cost more than actually using a precomputed right hand side which does not require inverting the Hessian for every RHS evaluation , but sometimes some systems are lightweight enough, yet complicated enough to differentiate by hand, that just implementing the Lagrangian and forgetting about everything else is a nice bonus. Generally, writing down the Lagrangian of many physical mechanical systems is trivial, while writing down the equations of motion is a huge amount of work to be done by hand or with SymPy . \"\"\" md\"\"\" First example Elastic rope first example problem an elastic rope made from N point masses attached by springs \"\"\" function lagrangian elastic rope q, qdot, p N length q m, k, g p kinetic energy is just 0.5 mass velocity^2 Eₖ 0.5 sum m . qdot.^2 potential energy is the elastic energy of the springs plus the gravitational potential Eₚ k 1 q 1 ^2 for i in 2 N Eₚ k i q i q i 1 ^2 end Eₚ 0.5 Eₚ g sum m . q return Eₖ Eₚ lagrangian kinetic energy potential energy end function parameters elastic rope N 3 a helper to setup the parameters of the problem N min N, 5 time domain t0 0 tend 50 initial condition q0 1.0, 2.0, 3.0, 4.0, 5.0 qdot0 0.2, 0.3, 0.7, 0.4, 0.5 q0, qdot0 q0 1 N , qdot0 1 N parameters m 1.1, 1.2, 1.3, 1.4, 1.5 k 1.5, 1.3, 1.7, 1.9, 2.1 g 1. m, k m 1 N , k 1 N params m, k, g return params, q0, qdot0, t0, tend end md\"\"\" Second Example Celestial Mechanics \"\"\" function lagrangian celestial mechanics q, qdot, params the lagrangian for the celestial mechanics system, this is the function that the user will implement note that it just computes the scalar value of the lagrangian of the system m, g params N length q ÷ 2 x view q, 1 N y view q, N 1 2N u view qdot, 1 N v view qdot, N 1 2N x, y q u, v qdot Eₖ 0.5 sum m . u.^2 . v.^2 kinetic energy is mass times velocity^2 2 potential energy is the sum of the gravitational potentials of all planets wrt all other planets the planet itself can be summed too, just use r r 2 eps as a division operator Eₚ 0.0 for i in 1 N rq x . x i .^2 . y . y i .^2 Eₚ 0.5 g m i sum m . sqrt. rq . rq . 1.0e 28 end return Eₖ Eₚ lagrangian kinetic energy potential energy end function parameters celestial mechanics N 4 a helper to setup the parameters of the problem N min N, 4 time domain t0 0.0 tend 1.0e1 parameters m 1000.0, 100.0, 10.0, 1.0 m m 1 N g 1.0 initial condition x0 0.0, 0.8, 3.7, 5.9 y0 0.0, 0.1, 0.2, 0.3 r0q x0.^2 . y0.^2 v0 sqrt. m 1 . g . sqrt. r0q . r0q . 1.0e 20 u0 0.0, 0.4, 0.5, 0.6 . v0 x0, y0, u0, v0 x0 1 N , y0 1 N , u0 1 N , v0 1 N v0 1 sum m . v0 m 1 u0 1 sum m . u0 m 1 q0 NamedArrayPartition x x0, y y0 qdot0 NamedArrayPartition u u0, v v0 q0 vcat x0, y0 qdot0 vcat u0, v0 params m, g return params, q0, qdot0, t0, tend end , q0, parameters celestial mechanics md\"\"\" Machinery \"\"\" md\"\"\" We remember that we want to solve \\frac d\\dot q dt H^ 1 \\frac dL dq First we need \\frac dL dq which we can get through automatic differentiation. Particularly efficent is in this case reverse mode. \"\"\" Note on 1.11 there is currently an Enzyme bug with broadcasting https github.com EnzymeAD Enzyme.jl issues 2397 static if VERSION v\"1.11\" function dLdq L, q, qdot, p dq Enzyme.make zero q autodiff Reverse, Const L , Active, Duplicated q, dq , Const qdot , Const p dq end else function dLdq L, q, qdot, p dq Enzyme.make zero q autodiff Reverse, Const L , Active, Duplicated q, dq , Duplicated qdot, Enzyme.make zero qdot , Duplicated p, Enzyme.make zero p dq end end md\"\"\" Secondly we need to solve H^ 1 \\frac dL dq . H is the Hessian and as such we want to do `H \\ b` where b \\frac dL dq . `manual dldqdotqdot` is using finite difference to take the 2nd derivative of L w.r.t \\dot q `dLdqdotdqdot explicit` uses automatic differentiation to perform forward over reverse mode `dLdqdotdqdot matrix free` constructs a matrix free variant that can be used by `Krylov.jl` to solve `H\\b`. \"\"\" function manual dLdqdotdqdot L, q, qdot, params, epsilon1 1.0e 4, epsilon2 1.0e 4 the lagrangian is a function of the lagrangian coordinates q and of the derivatives of the lagrangian coordinates qdot treated as independent variables this function computes the hessian of the lagrangian with respect to qdot brutally using finite differences N length qdot H zeros N, N f zeros 2, 2 for i in 1 N for j in 1 i for ii in 1 2 for jj in 1 2 qdotk copy qdot qdotk i qdotk i 2 ii 1 epsilon1 qdotk j qdotk j 2 jj 1 epsilon2 f ii,jj L q, qdotk, params end end H i,j f 1,1 f 1,2 f 2,1 f 2,2 4.0 epsilon1 epsilon2 end end for i in 1 N for j in i 1 N H i,j H j,i end end return H end function dLdqdotdqdot explicit L, q, qdot, p N length qdot v Enzyme.make zero qdot w similar v H zeros eltype qdot , N, N for i in 1 N v . 0 v i 1 Enzyme.hvp w, Const qdot L q, qdot, p , qdot, v H , i . w end H end function dLdqdotdqdot matrix free L, q, qdot, p N length qdot LinearOperator eltype qdot , N, N, false, false, w, v Enzyme.hvp w, Const qdot L q, qdot, p , qdot, v , nothing, nothing end function source from lagrangian L function source S, Q, p, t q, qdot Q N length q S.q . qdot S 1 N . qdot b dLdq L, q, qdot, p H dLdqdotdqdot matrix free L, q, qdot, p H dLdqdotdqdot explicit L, q, qdot, p x, stats gmres H, b assert stats.solved S.qdot . x S N 1 end . H \\ b return end end f elastic rope source from lagrangian lagrangian elastic rope sol elastic rope let p, q0, qdot0, t0, tend parameters elastic rope u0 NamedArrayPartition q q0, qdot qdot0 prob ODEProblem f elastic rope, u0, t0, tend , p solve prob, Tsit5 end let sol sol elastic rope fig Figure ts sol.t N length first sol.u .q for i in 1 N ax Axis fig 1,i , xlabel \"t\", ylabel L\"q\" ax2 Axis fig 2,i , xlabel \"t\", ylabel L\"\\dot q \" ax3 Axis fig 3,i , xlabel \"q\", ylabel L\"\\dot q \" q map j sol.u j .q i , 1 length ts qdot map j sol.u j .qdot i , 1 length ts lines ax, ts, q lines ax2, ts, qdot lines ax3, q, qdot end fig end f celestial mechanics source from lagrangian lagrangian celestial mechanics sol celestial mechanics let p, q0, qdot0, t0, tend parameters celestial mechanics u0 NamedArrayPartition q q0, qdot qdot0 prob ODEProblem f celestial mechanics, u0, t0, tend , p solve prob, Tsit5 end let sol sol celestial mechanics fig Figure ts sol.t N length first sol.u .q ÷ 2 ax Axis fig 1,1 , xlabel L\"x\", ylabel L\"y\" for i in 1 N x map j sol.u j .q i , 1 length ts y map j sol.u j .q N i , 1 length ts lines ax, x, y end fig end "},{"url":"mod4_automatic_differentiation/machine_learning/","title":"Automatic Differentiation and Machine Learning","description":"","tags":["module4","track_ad"],"text":" A Pluto.jl notebook v1.0.1 frontmatter chapter \"4\" section \"1\" order \"11\" title \"Automatic Differentiation and Machine Learning\" date \"2025 06 18\" tags \"module4\", \"track ad\" layout \"layout.jlhtml\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils begin using PlutoUI, PlutoTeachingTools using PlutoUI Slider PlutoUI.TableOfContents depth 4 end begin using CairoMakie set theme theme latexfonts fontsize 16, Lines linewidth 2, , markersize 16 end using BenchmarkTools, LinearAlgebra using Lux, Random, Optimisers begin ENV \"DATADEPS ALWAYS ACCEPT\" true using MLDatasets MNIST, convert2image using ImageShow end using OneHotArrays, Statistics, MLUtils, Zygote using Printf ChooseDisplayMode md\"\"\" Automatic Differentiation and Machine Learning \"\"\" md\"\"\" note Recall our example from lecture 4. \"\"\" begin import ForwardDiff import Enzyme end function m x, a, b, c, d return sin a x b cos c x d end xs 0.0 0.01 2π ys m. xs, 0.3, 1.2, 0.5, 0.7 function mse ŷ, y sum ŷ . y .^2 length y end function dm loss, ys, xs, c ForwardDiff.derivative a loss ys, m. xs, a, c 2 , c 3 , c 4 , c 1 ForwardDiff.derivative a loss ys, m. xs, c 1 , a, c 3 , c 4 , c 2 ForwardDiff.derivative a loss ys, m. xs, c 1 , c 2 , a, c 4 , c 3 ForwardDiff.derivative a loss ys, m. xs, c 1 , c 2 , c 3 , a , c 4 end md\"\"\" note We have to perform one ForwardDiff.derivative call per parameter \"\"\" coeffs guess rand 2.0 0.1 2.0, 4 m. xs, coeffs guess... dm mse, ys, xs, coeffs guess... begin learning rate 0.01 steps 1000 plot every 100 end let fig Figure ax Axis fig 1,1 coeffs rand 4 errs Float64 for i in 1 steps ys m. xs, coeffs... err mse ys, ys push errs, err if mod1 i, plot every 1 lines xs, ys, label \"Epoch i\" end dcoeffs dm mse, ys, xs, coeffs coeffs . learning rate . dcoeffs end ys m. xs, coeffs... err mse ys, ys lines xs, ys, label \"Epoch steps 1 \" lines xs, ys, label \"Goal\" axislegend ax ax2 Axis fig 2,1 lines ax2, errs fig end md\"\"\" Instead of using Forward Mode automatic differentiation we can perform reverse mode AD. \"\"\" function dm rev loss, ys, xs, c dc Enzyme.make zero c function f ys, xs, c loss ys, map x m x, c... , xs end Enzyme.autodiff Enzyme.Reverse, f, Enzyme.Active, Enzyme.Const ys , Enzyme.Duplicated xs, Enzyme.make zero xs , Enzyme.Duplicated c, dc dc end dm rev mse, ys, xs, coeffs guess... dm mse, ys, xs, coeffs guess... md\"\"\" note In one call we can obtain the derivatives of all coefficients This is a great help when the number of parameters grow very large \"\"\" md\"\"\" Reverse Mode \"\"\" md\"\"\" Forward vs. reverse mode AD What we have discussed previously is the basic idea of forward mode AD. It is called this way since the information of the derivatives propagates in the same way as the usual computation. Let's now consider a typical optimization problem, e.g., \\min x \\| A x b \\| 2^2, where A and b are given. To use something like gradient descent, we need to compute the derivative with respect to x , i.e., \\nabla x \\| A x b \\| 2^2 2 A x b ^T A. Let's check this with a simple example. \"\"\" let A 1.0 2.0 3.0 4.0 b 5.0, 6.0 f x1, x2 begin y A x1, x2 b return y 1 ^2 y 2 ^2 end x randn 2 result ad ForwardDiff.gradient X f X... , x result 2 A x b ' A abs2 result ad 1 result 1 abs2 result ad 2 result 2 end md\"\"\" Next, let's consider a non square matrix A and set b 0 for simplicity. Then, we have \\nabla x \\| A x \\| 2^2 2 A x ^T A 2 x^T A^T A. If you want to calculate this gradient by hand, you have to choose whether you want to first calculate A^T A and then x^T A^T A first calculate x^T A^T and then x^T A^T A Which order would you choose? \"\"\" order 1 A, x x' A' A order 2 A, x x' A' A md\" 4 \\times 4 matrix\" A1 randn 4, 4 x1 randn size A1, 2 benchmark order 1 A1, x1 benchmark order 2 A1, x1 md\" 2 \\times 8 matrix\" A3 randn 2, 8 x3 randn size A3, 2 benchmark order 1 A3, x3 benchmark order 2 A3, x3 md\"\"\" General introduction The difference between these orders is the basic idea of forward vs. reverse mode AD. To give you a rough idea, consider the chain rule applied to the function x \\mapsto f\\Bigl g\\bigl h x \\bigr \\Bigr , i.e., \\nabla x f\\Bigl g\\bigl h x \\bigr \\Bigr f'\\Bigl g\\bigl h x \\bigr \\Bigr \\cdot g'\\bigl h x \\bigr \\cdot h' x . Forward mode AD is like computing the derivatives from right to left like the usual data flow when computing f \\circ g \\circ h x . In contrast, reverse mode AD is like computing the derivatives from left to right, i.e., in reverse order. To be able to do so, you need to store the intermediate values h x and g\\bigl h x \\bigr . \"\"\" function forward f, f′, g, g′, h, h′, x h x h x h′ x h′ x gh x g h x gh′ x g′ h x h′ x fgh x f gh x fgh′ x f′ gh x gh′ x return fgh x, fgh′ x end function reverse f, f′, g, g′, h, h′, x h x h x gh x g h x fgh x f gh x f′ ghx f′ gh x fg′ hx f′ ghx g′ h x fgh′ x fg′ hx h′ x return fgh x, fgh′ x end const A randn 10, 10^2 const b randn size A, 1 h x AbstractVector A x h′ x AbstractVector A g Ax AbstractVector Ax b g′ Ax AbstractVector I f Ax b AbstractVector sum abs2, Ax b f′ Ax b AbstractVector 2 Ax b' let x randn size A, 2 fwd forward f, f′, g, g′, h, h′, x rev reverse f, f′, g, g′, h, h′, x fwd 1 rev 1 , norm fwd 2 rev 2 end let x randn size A, 2 benchmark forward f, f′, g, g′, h, h′, x end let x randn size A, 2 benchmark reverse f, f′, g, g′, h, h′, x end md\"\"\" Thinking about Reverse Mode \"\"\" function simple mlp input, weight, bias tmp1 input weight tmp2 tmp1 bias return tmp2 end function dsimple mlp input, weight, bias tmp1 input weight tmp2 tmp1 bias return tmp2 start by creating and zero'ing shadow variables for each differentiable input dinput 0.0 dweight 0.0 dbias 0.0 start by creating and zero'ing shadow variables for each differentiable variable dtmp1 0.0 dtmp2 0.0 derivative of return tmp2, set dtmp2 1.0 dtmp2 1.0 derivative of tmp2 tmp1 bias dtmp1 dtmp2 dbias dtmp2 dtmp2 0.0 After a read of a shadow we always must zero derivative of tmp1 input weight dinput dtmp1 weight dweight dtmp1 input dtmp1 0.0 After a read of a shadow we always must zero we're done, return the gradient derivative of all input variable return dinput, dweight, dbias end dsimple mlp 2.0, 0.5, 0.3 import Enzyme Reverse, Active, Duplicated, ReverseWithPrimal Enzyme.autodiff ReverseWithPrimal, simple mlp, Active 2.0 , Active 0.5 , Active 0.3 function with memory x x x 1 tmp1 x ^ 2 return tmp1 end function dwith memory x x x 1 tmp1 x ^ 2 return tmp1 dx Enzyme.make zero x dx 0.0 dtmp1 0.0 dtmp1 1.0 dx dtmp1 2 x dtmp1 0.0 dx 1 dx note the `accumulate` operation return dx, end dwith memory 2.0 let dx 0.0 Enzyme.autodiff Reverse, with memory, Active, Duplicated 2.0 , dx dx end function with memory2 x, y tmp1 x 1 ^ 2 y 1 tmp1 return nothing end md\"\"\" This time let the user provide dx, dy \"\"\" function dwith memory2 x, dx , y, dy tmp1 x 1 ^ 2 y 1 tmp1 dtmp1 0.0 dtmp1 dy 1 dy 1 0.0 dx 1 dtmp1 2 x 1 Note Needs to prove the x is read only and not aliased with y return nothing end let x 2.0 y 0.0 with memory2 x, y x, y end let x 2.0 y x with memory2 x, y x, y end let dx 0.0 dy 1.0 dwith memory2 2.0 , dx , 0.0 , dy dx, dy end let x 2.0 y x dx 0.0 dy 1.0 dwith memory2 x, dx , y, dy dx, dy end let dx 0.0 dy 1.0 Enzyme.autodiff Reverse, with memory2, Duplicated 2.0 , dx , Duplicated 0.0 , dy dx, dy end function with control x cond x 0 if cond tmp1 x^2 else tmp1 x end return tmp1 end function dwith control x cond x 0 if cond tmp1 x^2 else tmp1 x end dx 0.0 dtmp1 0.0 dtmp1 1.0 if cond d x^2 dx 2x dx dtmp1 2 x else d x dx 1 dx dtmp1 1 end return dx, end dwith control 2.0 Enzyme.autodiff Enzyme.Reverse, with control, Enzyme.Active 2.0 dwith control 2.0 Enzyme.autodiff Enzyme.Reverse, with control, Enzyme.Active 2.0 md\"\"\" Machine Learning \"\"\" model Chain Dense 128, 256, tanh , Chain Dense 256, 64, tanh , Dense 64, 10 md\"\"\" Dense A fully connected, feed forward layer computing ``y σ. W x b ``. Transfer function activation ``σ`` A non linearity applied element wise, e.g. `tanh`, `relu`, or `sigmoid`. \"\"\" begin rng Random.default rng Random.seed rng, 0 end ps, st Lux.setup rng, model md\"\"\" note Conscious handling of randomness. \"\"\" let Dummy Input x rand rng, Float32, 128, 2 Run the model y, Lux.apply model, x, ps, st end let x rand rng, Float32, 128, 2 train state Lux.Training.TrainState model, ps, st, Adam 0.0001f0 gs, loss, stats, train state Lux.Training.compute gradients AutoEnzyme , MSELoss , x, rand rng, Float32, 10, 2 , train state Training.apply gradients train state, gs train state, gs end let x rand rng, Float32, 128, 2 train state Lux.Training.TrainState model, ps, st, Adam 0.0001f0 gs, loss, stats, train state Training.single train step AutoEnzyme , MSELoss , x, rand rng, Float32, 10, 2 , train state end md\"\"\" MNIST \"\"\" dataset MNIST convert2image dataset, 10 convert2image dataset, 12 dataset.targets 10 function loadmnist batchsize, train split Load MNIST N 1500 nothing dataset MNIST split train if N nothing imgs dataset.features , , 1 N labels raw dataset.targets 1 N else imgs dataset.features labels raw dataset.targets end Process images into H, W, C, BS batches x data Float32. reshape imgs, size imgs, 1 , size imgs, 2 , 1, size imgs, 3 y data onehotbatch labels raw, 0 9 x train, y train , x test, y test splitobs x data, y data at train split return Use DataLoader to automatically minibatch and shuffle the data DataLoader collect. x train, y train batchsize, shuffle true, partial false , Don't shuffle the test data DataLoader collect. x test, y test batchsize, shuffle false, partial false , end lux model Chain Conv 5, 5 , 1 6, relu , MaxPool 2, 2 , Conv 5, 5 , 6 16, relu , MaxPool 2, 2 , FlattenLayer 3 , Chain Dense 256 128, relu , Dense 128 84, relu , Dense 84 10 , const lossfn CrossEntropyLoss logits Val true function accuracy model, ps, st, dataloader total correct, total 0, 0 st Lux.testmode st for x, y in dataloader target class onecold Array y predicted class onecold Array first model x, ps, st total correct sum target class . predicted class total length target class end return total correct total end function train model rng Random.default rng , kwargs... train dataloader, test dataloader loadmnist 128, 0.9 ps, st Lux.setup rng, model vjp AutoEnzyme Oof Enzyme takes forever to come up with the gradient 105s vjp AutoZygote train state Training.TrainState model, ps, st, Adam 3.0f 4 Lets train the model nepochs 10 tr acc, te acc 0.0, 0.0 for epoch in 1 nepochs stime time for x, y in train dataloader , , , train state Training.single train step vjp, lossfn, x, y , train state end ttime time stime tr acc accuracy model, train state.parameters, train state.states, train dataloader 100 te acc accuracy model, train state.parameters, train state.states, test dataloader 100 printf \" %2d %2d \\t Time %.2fs \\t Training Accuracy %.2f%% \\t Test Accuracy \\ %.2f%%\\n\" epoch nepochs ttime tr acc te acc end return tr acc, te acc end tr acc, te acc train lux model "}]