[{"url":"cheatsheets/","title":"Resources","tags":["welcome"],"text":"ResourcesModern Julia WorkflowsJuliaLang DiscourseJuliaLang DocumentationJulia CommunityCheatsheetsFastrack to Julia cheatsheet.MATLAB-Julia-Python comparative cheatsheet by QuantEcon groupPlots.jl cheatsheet"},{"url":".","title":"index","tags":["homepage"],"text":""},{"url":"installation/","title":"Software installation","tags":["welcome"],"text":"First-time setup: Install Julia & PlutoText and pictures version:Step 1: Install Julia 1.8.2Go to https://julialang.org/install/ and download juliaup, juliaup will install the current stable release, Julia 1.8.2, using the correct version for your operating system (Linux x86, Mac, Windows, etc).Step 2: Run JuliaAfter installing, make sure that you can run Julia. On some systems, this means searching for the “Julia 1.8.2” program installed on your computer; in others, it means running the command julia in a terminal. Make sure that you can execute 1 + 1:Make sure that you are able to launch Julia and calculate 1+1 before proceeding!Step 3: Install PlutoNext we will install the Pluto, the notebook environment that we will be using during the course. Pluto is a Julia programming environment designed for interactivity and quick experiments.Open the Julia REPL. This is the command-line interface to Julia, similar to the previous screenshot.Here you type Julia commands, and when you press ENTER, it runs, and you see the result.To install Pluto, we want to run a package manager command. To switch from Julia mode to Pkg mode, type ] (closing square bracket) at the julia> prompt:\njulia> ]\n\n(@v1.8) pkg>\nThe line turns blue and the prompt changes to pkg>, telling you that you are now in package manager mode. This mode allows you to do operations on packages (also called libraries).To install Pluto, run the following (case sensitive) command to add (install) the package to your system by downloading it from the internet.\nYou should only need to do this once for each installation of Julia:\n(@v1.8) pkg> add Pluto\nThis might take a couple of minutes, so you can go get yourself a cup of tea!You can now close the terminal.Step 4: Use a modern browser: Mozilla Firefox or Google ChromeWe need a modern browser to view Pluto notebooks with. Firefox and Chrome work best.Second time: Running Pluto & opening a notebookRepeat the following steps whenever you want to work on a project or homework assignment.Step 1: Start PlutoStart the Julia REPL, like you did during the setup. In the REPL, type:julia> using Pluto\n\njulia> Pluto.run()\nThe terminal tells us to go to http://localhost:1234/ (or a similar URL). Let’s open Firefox or Chrome and type that into the address bar.If you’re curious about what a Pluto notebook looks like, have a look at the Featured Notebooks. These notebooks are useful for learning some basics of Julia programming.If you want to hear the story behind Pluto, have a look a the JuliaCon presentation.If nothing happens in the browser the first time, close Julia and try again. And please let us know!Step 2a: Opening a notebook from the webThis is the main menu - here you can create new notebooks, or open existing ones. Our homework assignments will always be based on a template notebook, available in this GitHub repository. To start from a template notebook on the web, you can paste the URL into the blue box and press ENTER.For example, homework 0 is available here. Go to this page, and on the top right, click on the button that says “Edit or run this notebook”. From these instructions, copy the notebook link, and paste it into the box. Press ENTER, and select OK in the confirmation box.The first thing we will want to do is to save the notebook somewhere on our own computer; see below.Step 2b: Opening an existing notebook fileWhen you launch Pluto for the second time, your recent notebooks will appear in the main menu. You can click on them to continue where you left off.If you want to run a local notebook file that you have not opened before, then you need to enter its full path into the blue box in the main menu. More on finding full paths in step 3.Step 3: Saving a notebookWe first need a folder to save our homework in. Open your file explorer and create one.Next, we need to know the absolute path of that folder. Here’s how you do that in Windows, MacOS and Ubuntu.For example, you might have:C:\\Users\\fons\\Documents\\18S191_assignments\\ on Windows/Users/fons/Documents/18S191_assignments/ on MacOS/home/fons/Documents/18S191_assignments/ on UbuntuNow that we know the absolute path, go back to your Pluto notebook, and at the top of the page, click on “Save notebook…”.This is where you type the new path+filename for your notebook:Click Choose.Step 4: Sharing a notebookAfter working on your notebook (your code is autosaved when you run it), you will find your notebook file in the folder we created in step 3. This the file that you can share with others, or submit as your homework assignment to Canvas.\nconst run = f => f();\nrun(async () => {\nconst versions = await (await fetch(`https://julialang-s3.julialang.org/bin/versions.json`)).json()\nconst sortby = v => v.split(\"-\")[0].split(\".\").map(parseFloat).reduce((a,b) => a*10000 + b)\nconst version_names = Object.keys(versions).sort((a,b) => sortby(a) - sortby(b)).reverse()\nconst stable = version_names.find(v => versions[v].stable)\nconsole.log({stable})\nconst pkg_stable = /\\d+\\.\\d+/.exec(stable)[0]\ndocument.querySelectorAll(\"auto-julia-version\").forEach(el => {\n    console.log(el)\n    el.innerText = el.getAttribute(\"short\") == null ? stable : pkg_stable\n})\n});"},{"url":"logistics/","title":"Class logistics","tags":["welcome"],"text":"main a img {\n    width: 5rem;\n    margin: 1rem;\n}\nCourse logisticsProjectsChoose project by 2025-05-14Discuss idea first with me.Half-page project proposal.Project presentations: 2025-07-16Project should be relevant both to your work and topics discussed in class.Demonstrate best practices for research software development.\nTestingReproducibilityDocumentationAccessibilityDevelop “in the open” on Github.GradesIf you need a grade for this class, in addition to the project presentation, you will need to submit a short project report (2 pages) and your repository with instruction on how to reproduce your results."},{"url":"search/","title":"Search results","tags":[],"text":"window.init_search();SearchResults\nLoading..."},{"url":"assets/scripts/get_highlights/","title":"get_highlights","tags":[],"text":"if isempty get metadata \"homepage\" , \"highlights\", nothing else highlights htl \"\"\" section div class \"content\" h2 x \"name\" h2 p x \"text\" p div div class \"preview\" img src \" x \"img\" \" div section \"\"\" for x in metadata \"homepage\" \"highlights\" htl \"\"\" div class \"subjectscontainer wide\" h1 Highlights h1 div class \"contain\" highlights div div \"\"\" end"},{"url":"assets/scripts/get_schedule/","title":"get_schedule","tags":[],"text":"let sections metadata \"sidebar\" sections htl \"\"\" let input other page.input output other page.output name get output.frontmatter, \"title\", basename input.relative path desc get output.frontmatter, \"description\", nothing tags get output.frontmatter, \"tags\", String date get output.frontmatter, \"date\", nothing class \"no decoration\", \"tag replace x, \" \" \" \" \" for x in tags ..., if date nothing htl \"\"\" a title desc class class href root url \" \" other page.url h3 name h3 date a \"\"\" else nothing end end for other page in collections section id .pages \"\"\" for section id, section name in sections isempty sections ? nothing htl \"\"\" div class \"wide subjectscontainer\" h1 Schedule h1 div class \"subjects\" sections div div \"\"\" end"},{"url":"assets/scripts/get_subjects/","title":"get_subjects","tags":[],"text":"let sections metadata \"sidebar\" sections htl \"\"\" let input other page.input output other page.output name get output.frontmatter, \"title\", basename input.relative path desc get output.frontmatter, \"description\", nothing tags get output.frontmatter, \"tags\", String image get output.frontmatter, \"image\", nothing class \"no decoration\", \"tag replace x, \" \" \" \" \" for x in tags ..., image nothing || isempty image ? nothing htl \"\"\" a title desc class class href root url \" \" other page.url h3 name h3 img src image a \"\"\" end for other page in collections section id .pages \"\"\" for section id, section name in sections isempty sections ? nothing htl \"\"\" div class \"wide subjectscontainer\" h1 Subjects h1 div class \"subjects\" sections div div \"\"\" end"},{"url":"exercises/exercise_10_shmem_parallel/","title":"Shared memory parallelism","tags":["module3","track_parallel","exercises"],"text":" A Pluto.jl notebook v0.20.13 frontmatter order \"8.3\" exercise number \"10\" title \"Shared memory parallelism\" tags \"module3\", \"track parallel\", \"exercises\" license \"MIT\" layout \"layout.jlhtml\" description \"sample exercise\" frontmatter.author name \"Valentin Churavy\" image \"https avatars.githubusercontent.com u 145258?v 4\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils using PlutoTeachingTools md\"\"\" Exercise Parallel PI We randomly throw \"darts\" at a circle with radius 1, contained in a square from 1 to 1 and count how many darts land within the circle. For ease we only sample the portion of the circle that is in the postive quadrant. \"\"\" question box md\"\"\" How would you parallelize the following code? Are there advantages of a \"functional\" style using mapreduce vs explicit for loops? \"\"\" using OhMyThreads function monte carlo pi N M mapreduce , 1 N do i rand ^2 rand ^2 1.0 end pi 4 M N return pi end hint md\"\"\" OhMyThreads.jl has a parallel `tmapreduce`. \"\"\" monte carlo pi 10 000 md\"\"\" Exercise Parallel Histogram Use `Base. threads` to parallelize. \"\"\" using Atomix atomic, atomicswap, atomicreplace question box md\"\"\" If you naively parallelize, is there a data race? If yes how would you fix it? Is there a risk of false sharing? Would implementing this algorithm with `tmapreduce` help? \"\"\" import StatsBase StatsBase, Histogram begin function histogram data, nbins, lb, ub buckets zeros Int, nbins width ub lb nbins 1 for x in data bucket round Int, x lb width 1 buckets bucket 1 end return Histogram lb width width ub, buckets, right end histogram data, nbins histogram data, nbins, extrema data end hint md\"\"\" ``` threads for x in data ``` \"\"\" using UnicodePlots data rand 1 64, 1024 UnicodePlots.vertical histogram StatsBase.fit Histogram, data, nbins 64, closed right , height 10 let data rand 1 64, 1024 hist histogram data, 64, 1, 64 UnicodePlots.vertical histogram hist, height 10 end md\"\"\" Exercise Parallel sorting note Implement a parallel merge sort using the skeleton below \"\"\" function merge out, left, right ll, lr length left , length right assert ll lr length out i, il, ir 1, 1, 1 inbounds while il ll && ir lr l, r left il , right ir if isless r, l out i r ir 1 else out i l il 1 end i 1 end inbounds while il ll out i left il il 1 i 1 end inbounds while ir lr out i right ir ir 1 i 1 end return out end perform a merge sort on `v` function psort v AbstractVector hi length v if hi 100 000 below some cutoff, run in serial return sort v, alg MergeSort end split the range and sort the halves in parallel recursively TODO Parallelize mid 1 hi 1 left psort view v, 1 mid right psort view v, mid 1 hi perform the merge on the result out similar v merge out, left, right return out end hint md\"\"\" Recall the parallel `fib` \"\"\" v rand 16 000 000 v sorted ours psort v v sorted theirs sort v v sorted ours v sorted theirs "},{"url":"exercises/exercise_11_gpu/","title":"Introduction to KernelAbstractions","tags":["module3","track_parallel","exercises"],"text":"Getting started on Google ColabRecently Google Colab started supporting Julia directly. One needs to switch the change the “runtime type” to “Julia”,\nand Google Colab also provides some GPU support for free.To get started with today’s exercise:"},{"url":"exercises/exercise_12_mpi/","title":"Running MPI.jl locally","tags":["module3","track_parallel","exercises"],"text":"ConfigurationThere are multiple MPI implementationsOpenMPIMPICHIntel MPIMicrosoft MPIIBM Spectrum MPIMVAPICHCray MPICHFujitsu MPIHPE MPT/HMPTjulia> using MPI\n\njulia> MPI.versioninfo()\n\nMPIPreferences:\n  binary:  MPICH_jll\n  abi:     MPICH\n\nPackage versions\n  MPI.jl:             0.20.22\n  MPIPreferences.jl:  0.1.11\n  MPICH_jll:          4.3.0+1\n\nLibrary information:\n  libmpi:  /home/vchuravy/.julia/artifacts/05d8c79b270470018e9de8dd24ddb6d7954aff9d/lib/libmpi.so\n  libmpi dlpath:  /home/vchuravy/.julia/artifacts/05d8c79b270470018e9de8dd24ddb6d7954aff9d/lib/libmpi.so\n  MPI version:  4.1.0\n  Library version:\n    MPICH Version:      4.3.0\n    MPICH Release date: Mon Feb  3 09:09:47 AM CST 2025\n    MPICH ABI:          17:0:5\n    MPICH Device:       ch3:nemesis\n    MPICH configure:    --build=x86_64-linux-musl --disable-dependency-tracking --disable-doc --enable-fast=ndebug,O3 --enable-static=no --host=x86_64-linux-gnu --prefix=/workspace/destdir --with-device=ch3 --with-hwloc=/workspace/destdir\n    MPICH CC:           cc     -DNDEBUG -DNVALGRIND -O3\n    MPICH CXX:          c++   -DNDEBUG -DNVALGRIND -O3\n    MPICH F77:          gfortran   -O3\n    MPICH FC:           gfortran   -O3\n    MPICH features:\nOn Unix, MPI.jl will install and use MPICH through the MPICH_jll package.MPIPreferencesTo switch which MPI implementation MPI.jl uses you can use the package MPIPreferences.jl.For more information see: https://juliaparallel.org/MPI.jl/stable/configuration/When executing on a cluster you will likely need to configure MPI.jl to use the system provided MPI.Installing mpiexecjljulia> using MPI\njulia> MPI.install_mpiexecjl()\nBy default, it will install to ~/.julia/bin, but you can also choose to install it somewhere elseAs an example to install it in the current working directory.julia> using MPI\njulia> MPI.install_mpiexecjl(destdir=\".\")\nAfter installing it, you can use it to start Julia.mpiexecjl --project=/path/to/project -n 4 julia script.jl\n# or\n./mpiexecjl --project=/path/to/project -n 4 julia script.jl\nExercisesMPI.jl has a series of examples:Hello wordBroadcastReduceSend/receiveDiffusionIn exercise 2 we looked at a diffusion kernel.\nInstead of implementing this on the GPU you can also implement it with MPI.NoteThe “hard” part is the handling of the boundary-conditions and ghost cells. So focus on that in the beginning.\nHow are you going to split the computational domain? Who needs to talk to whom?"},{"url":"exercises/exercise_13_reverse/","title":"Reverse mode AD","tags":["module4","track_ad","exercises"],"text":" A Pluto.jl notebook v0.20.13 frontmatter order \"11.1\" exercise number \"13\" chapter \"4\" section \"1\" title \"Reverse mode AD\" tags \"module4\", \"track ad\", \"exercises\" license \"MIT\" layout \"layout.jlhtml\" description \"sample exercise\" frontmatter.author name \"Valentin Churavy\" image \"https avatars.githubusercontent.com u 145258?v 4\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils using PlutoTeachingTools using Enzyme function f1 x, y x exp y end autodiff Reverse, f1, Active 2.0 , Active 1.0 md\"\"\" Exercise Manually write the reverse mode implementation for `f1`. \"\"\" function df1 x, y ... return dx, dy end md\"\"\" Enzyme allows for a selection of arguments to be \"active\". How would your implementation change if you knew that you didn't need the gradient of an argument? \"\"\" | question box autodiff Reverse, f1, Active 2.0 , Const 1.0 autodiff Reverse, f1, Const 2.0 , Active 1.0 function f2 x Array Float64 , y Array Float64 y 1 x 1 x 1 x 2 x 1 return nothing end let x 2.0, 2.0 dx 0.0, 0.0 y 0.0 dy 1.0 Enzyme.autodiff Reverse, f2, Duplicated x, dx , Duplicated y, dy dx, dy end md\"\"\" Exercise Manually write the reverse mode implementation for `f2`. note It might be very helpful to \"desugar\" the implementation as much as possible. \"\"\" md\"\"\" Note that Enzyme zeros out the `dy` memory. What would happen if it didn't? \"\"\" | question box f3 x exp sin x ^2 "},{"url":"exercises/exercise_14_ml/","title":"Reverse mode AD","tags":["module4","track_ad","exercises"],"text":" A Pluto.jl notebook v0.20.13 frontmatter order \"11.2\" exercise number \"14\" chapter \"4\" section \"1\" title \"Reverse mode AD\" tags \"module4\", \"track ad\", \"exercises\" license \"MIT\" layout \"layout.jlhtml\" description \"sample exercise\" frontmatter.author name \"Valentin Churavy\" image \"https avatars.githubusercontent.com u 145258?v 4\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils using UnicodePlots md\"\"\" x j b j \\sum i w ij x i \"\"\" function perceptron inputs, weights, bias, f f sum w i input i for i in 1 length inputs bias end lineplot 3,3, identity relu x max 0, x lineplot 3,3, relu sigmoid x 1 1 exp x lineplot 3,3, sigmoid tanh x exp x exp x exp x exp x lineplot 3,3, tanh md\"\"\" While one can define a perceptron like above, it is better to think of it in terms of matrix multiplication. \"\"\" Ni 10 Nn 5 x rand Ni b rand Nn W rand Nn, Ni tanh. W x . b md\"\"\" Exercise Write your own Neuronal Network model with a couple of layers maybe 2 Remember https vchuravy.dev rse course exercises exercise 7 gradient descent as a starting point. \"\"\" "},{"url":"exercises/exercise_1_multithreading/","title":"Shared-memory parallelism","tags":["module1","track_parallel","exercises"],"text":" A Pluto.jl notebook v0.20.13 frontmatter order \"2.1\" exercise number \"1\" title \"Shared memory parallelism\" tags \"module1\", \"track parallel\", \"exercises\" layout \"layout.jlhtml\" description \"sample exercise\" using Markdown using InteractiveUtils using PlutoTeachingTools, PlutoUI ChooseDisplayMode PlutoUI.TableOfContents depth 4 md\"\"\" Exercise Shared memory parallelism \"\"\" md\"\"\" This notebook uses Threads.nthreads threads \"\"\" md\"\"\" Parallel fibonacci Remember ```julia t Threads. spawn begin ` spawn` returns right away 3 3 end fetch t `fetch` waits for the task to finish ``` \"\"\" function fib n if n 1 return n end return fib n 1 fib n 2 end fib 12 TODO Implement pfib let if isdefined pfib func not defined pfib elseif pfib 12 fib 12 keep working md\"Your solution and the reference solution disagree \" else correct end end answer box hint md\"\"\" ```julia function pfib n if n 1 return n end t Threads. spawn pfib n 2 return pfib n 1 fetch t Int end ``` \"\"\" md\"\"\" Multi threaded map \"\"\" using LinearAlgebra, Random using BenchmarkTools function tmap fn, itr for each i ∈ itr, spawn a task to compute fn i tasks map i Threads. spawn fn i , itr fetch and return all the results return fetch. tasks end Ms rand 100,100 for i in 1 8 Threads.nthreads begin BLAS.set num threads Sys.CPU THREADS Fix number of BLAS threads BLAS.set num threads 1 blas edge nothing end begin blas edge serial map svdals b benchmark map svdvals, Ms samples 10 evals 3 end begin blas edge threaded map svdals b benchmark tmap svdvals, Ms samples 10 evals 3 end minimum serial map svdals b.times minimum threaded map svdals b.times Threads.nthreads 100 parallel efficiency md\"\"\" note Vary the number of threads the BLAS library uses. See the cell above with `BLAS.set num threads ` \"\"\" "},{"url":"exercises/exercise_2_accelerated/","title":"Introduction to accelerated computing","tags":["module1","track_parallel","exercises"],"text":" A Pluto.jl notebook v0.20.13 frontmatter order \"2.2\" exercise number \"2\" title \"Introduction to accelerated computing\" tags \"module1\", \"track parallel\", \"exercises\" layout \"layout.jlhtml\" description \"Introduction to accelerated computing\" using Markdown using InteractiveUtils This Pluto notebook uses bind for interactivity. When running this notebook outside of Pluto, the following 'mock version' of bind gives bound variables a default value instead of an error . macro bind def, element format off return quote local iv try Base.loaded modules Base.PkgId Base.UUID \"6e696c72 6542 2067 7265 42206c756150\" , \"AbstractPlutoDingetjes\" .Bonds.initial value catch b missing end local el esc element global esc def Core.applicable Base.get, el ? Base.get el iv el el end format on end using PlutoTeachingTools, PlutoUI PlutoUI.TableOfContents depth 4 ChooseDisplayMode md\"\"\" Exercise Introduction to accelerated computing AXPY with KernelAbstractions KernelAbstractions is a light weight domain specific language that enables you to write portable kernels. It allows us to target many different backend packages CUDA.jl Metal.jl OneAPI.jl AMDGPU.jl And it provides a fallback implementation for CPU \"\"\" using LinearAlgebra, Random using BenchmarkTools using KernelAbstractions begin import CUDA import Metal Work around https github.com JuliaGPU oneAPI.jl issues 445 . ENV \"SYCL PI LEVEL ZERO BATCH SIZE\" \"1\" import oneAPI sigh Currently crashes inside Pluto import AMDGPU end md\"\"\" Choose your own backend \"\"\" let available backends KernelAbstractions.Backend CPU CI get ENV, \"GITHUB ACTIONS\", \"false\" \"true\" Always shows all backend when rendering the notebook on GitHub Actions, notebooks will be static anyway. if CUDA.functional || CI push available backends, CUDA.CUDABackend end if oneAPI.functional || CI push available backends, oneAPI.oneAPIBackend end if Metal.functional || CI push available backends, Metal.MetalBackend end if AMDGPU.functional || CI push available backends, AMDGPU.ROCBackend end bind backend Select available backends end md\"\"\" The AXPY kernel is defined as Y Y a X note Vary the `M` below. \"\"\" M 1024 function axpy serial Y, a, X for i in eachindex Y, X TODO Implement end return nothing end let Y randn Float32, M X randn Float32, M a randn Float32 reference Y . a . X axpy serial Y, a, X if reference Y correct else keep working md\"Your solution and the reference solution disagree \" end end answer box hint md\"\"\" ```julia function axpy serial Y, a, X for i in eachindex Y, X Y i a X i end return nothing end ``` \"\"\" kernel function axpy kernel Y, a, X implement me end Creating a wrapper kernel for launching with error checks function axpy ka Y, a, X if length Y length X error \"Arrays disagree in length\" return nothing end backend KernelAbstractions.get backend Y kernel axpy kernel backend kernel Y, a, X, ndrange size Y KernelAbstractions.synchronize backend return end let Y randn allocate backend, Float32, M X randn allocate backend, Float32, M a randn Float32 reference Y . a . X axpy ka Y, a, X if reference Y correct else keep working md\"Your solution and the reference solution disagree \" end end answer box hint md\"\"\" ```julia function axpy kernel Y, a, X i index Global Y i a X i end ``` \"\"\" question box md\"Benchmark the KernelAbstraction implementation against a high level array implementation at different problem sizes\" md\"\"\" Diffusion kernel \"\"\" begin using OffsetArrays using CairoMakie end begin a 0.01 dx 0.01 x grid spacing dy 0.01 y grid spacing end dt dx^2 dy^2 2.0 a dx^2 dy^2 Largest stable time step N 64 md\"\"\" Implement a kernel that solves the 2D Diffusion Equation. \\frac \\partial U \\partial t a \\frac \\partial^2 U \\partial x^2 \\frac \\partial^2 U \\partial y^2 Using a finite difference approximation dU a dt \\frac U i 1, j 2U i,j U i 1,j dx^2 \\frac U i, j 1 2U i,j U i,j 1 dy^2 note The code below implements the harness using OffsetArrays and wrap around boundary conditions to make your life easier. \"\"\" kernel function diffuse dU, Const U , a, dt, dx, dy implement me end function diffuse U, a, dt, dx, dy dU zero U diffuse get backend U dU, U, a, dt, dx, dy ndrange N,N U . dU update boundary condition wrap around U 0, . U N, U N 1, . U 1, U , 0 . U , N U , N 1 . U , 0 U end answer box hint md\"\"\" ```julia kernel function diffuse dU, Const U , a, dt, dx, dy i, j index Global, NTuple out i, j a dt U i 1, j 2 U i, j U i 1, j dx^2 U i, j 1 2 U i, j U i, j 1 dy^2 end ``` \"\"\" let xs 0 N 1 ys 0 N 1 domain OffsetArray KernelAbstractions.zeros backend, Float32, N 2, N 2 , xs, ys TODO Split out into initalize function parent domain 16 32, 16 32 . 5 fig, ax, hm heatmap xs, ys, Array parent domain Makie.Record fig, 1 250 do i diffuse domain, a, dt, dx, dy update data autolimits ax update limits end end "},{"url":"exercises/exercise_3_type_stability/","title":"Type-stability","tags":["module1","track_parallel","exercises"],"text":" A Pluto.jl notebook v0.20.13 frontmatter order \"3.2\" exercise number \"3\" title \"Type stability\" tags \"module1\", \"track parallel\", \"exercises\" layout \"layout.jlhtml\" description \"sample exercise\" using Markdown using InteractiveUtils using BenchmarkTools md\"\"\" Type stability One way to optimize code in Julia is to ensure type stability. If the type s of some variables in a function are subject to change or ambiguity, the compiler cannot reason as well about those variables, and performance will take a hit. Conversely, we allow the compiler to optimize and generate more efficient machine code when we declare variables so that their types will be fixed throughout the function body. For example, let's say we had functions called `baz` and `bar` with the following definitions \"\"\" function baz s rand if s 2 3 return .666667 elseif s 1 3 return 1 3 else return 0 end end function bar s rand if s 2 3 return .666667 elseif s 1 3 return .3333333 else return 0.0 end end benchmark baz benchmark bar md\"\"\" I see that `bar` is almost three times as fast as `baz ` The reason is that `bar` is type stable while `baz` is not the compiler can tell that `bar` will always return a `Float64`, whereas `baz` could return a `Float64`, an `Int`, or a `Rational`. When the compiler can tell what the types of outputs from a function, or variables declared within a function are without running the code, it can do much better. \"\"\" Base.return types baz Base.return types bar md\"\"\" Exercise 1 The following definition for `my sum` is not type stable. ```julia function my sum A output 0 for x in A output x end return output end ``` Copy and execute the above code into a new cell. Benchmark it using A rand 10^3 . Then write a new function called `my sum2` with the same function body as `my sum`. Update `my sum2` to make it type stable, and benchmark it for a randomly populated array with 10^3 entries. How much does type stability impact performance? If you'd like, try this same exercise for multiple sizes of `A` to see if this changes your answer \"\"\" md\"\"\" Exercise 2 Make the following code type stable. You'll know your efforts are paying off when you see a performance boost ​ ```julia Calculate the square root of `x` with Newton's method. function my sqrt x output 1 for i in 1 1000 output .5 output x output end output end ``` \"\"\" "},{"url":"exercises/exercise_5_optimization/","title":"Optimization of parameters","tags":["module1","track_ad","exercises"],"text":" A Pluto.jl notebook v0.20.13 frontmatter order \"4.1\" exercise number \"6\" title \"Optimization of parameters\" tags \"module1\", \"track ad\", \"exercises\" layout \"layout.jlhtml\" description \"sample exercise\" using Markdown using InteractiveUtils begin using PlutoUI, PlutoTeachingTools using PlutoUI Slider PlutoUI.TableOfContents depth 4 end ChooseDisplayMode begin using CairoMakie set theme theme latexfonts fontsize 16, Lines linewidth 2, , markersize 16 end md\"\"\" Optimization \"\"\" using ForwardDiff md\"\"\" Example Rosenbrock function f x,y 1 x ^2 100 y x^2 ```julia rosenbrock x, y 1 x ^ 2 100 y x ^ 2 ^ 2 ``` \"\"\" rosenbrock x, y 1 x ^ 2 100 y x ^ 2 ^ 2 begin xs 6 0.04 6 ys 2 0.04 5 zs log10 rosenbrock x, y for x in xs, y in ys end surface xs, ys, zs, axis type Axis3, contourf xs, ys, zs using Optim let f x rosenbrock x... sol optimize f, 0.0, 0.0 show Optim.minimizer sol sol end ∂f ∂x f, x, y ForwardDiff.derivative x f x, y , x ∂f ∂y f, x, y ForwardDiff.derivative y f x, y , y contourf xs, ys, x,y ∂f ∂x rosenbrock, x, y 1 . xs, ys' contourf xs, ys, x,y ∂f ∂y rosenbrock, x, y . xs, ys' let f x rosenbrock x 1 , x 2 function g dx, x dx 1 ∂f ∂x rosenbrock, x 1 , x 2 dx 2 ∂f ∂y rosenbrock, x 1 , x 2 nothing end sol optimize f, g , 0.0, 0.0 show Optim.minimizer sol sol end md\"\"\" Exercise Fitting our parameters \"\"\" function m x, a, b, c, d return sin a x b cos c x d end xs1 0.0 0.01 2π ys1 m. xs1, 0.3, 1.2, 0.5, 0.7 Mean squared error function mse ŷ, y sum ŷ . y .^2 length y end sol2 let f coeffs mse ys1, m. xs1, coeffs... function g dc, c dc 1 ForwardDiff.derivative a f a, c 2 , c 3 , c 4 , c 1 dc 2 0 TODO dc 3 0 TODO dc 4 0 TODO nothing end c₀ 0.9456253412871252, 0.5060263144193908, 0.7313087724591087, 0.5401913661922946 TODO What happens with c₀ rand 2.0 0.1 2.0, 4 optimize f, g , c₀ end Optim.minimizer sol2 ys2 m. xs1, Optim.minimizer sol2 ... let fig Figure ax Axis fig 1,1 lines ax, xs1, ys1, label \"Target curve\" lines ax, xs1, ys2, label \"Your curve, mse mse ys1, ys2 \" axislegend ax fig end hint md\"\"\" ```julia function g dc, c dc 1 ForwardDiff.derivative a f a, c 2 , c 3 , c 4 , c 1 dc 2 ForwardDiff.derivative a f c 1 , a, c 3 , c 4 , c 2 dc 3 ForwardDiff.derivative a f c 1 , c 2 , a, c 4 , c 3 dc 4 ForwardDiff.derivative a f c 1 , c 2 , c 3 , a , c 4 nothing end ``` \"\"\" danger md\"\"\" It is not gurantueed that the optimizer converges to the correct global minimum. Change the initialization of `c₀`. \"\"\" "},{"url":"exercises/exercise_6_pitfalls/","title":"Pitfalls of AD","tags":["module1","track_ad","exercises"],"text":" A Pluto.jl notebook v0.20.13 frontmatter order \"4.2\" exercise number \"6\" title \"Pitfalls of AD\" tags \"module1\", \"track ad\", \"exercises\" layout \"layout.jlhtml\" description \"sample exercise\" using Markdown using InteractiveUtils begin using PlutoTeachingTools, PlutoUI using PlutoUI Slider PlutoUI.TableOfContents depth 4 end import ForwardDiff import Enzyme derivative enzyme f,x Enzyme.autodiff Enzyme.Forward, f, Enzyme.Duplicated x, one x | only md\"\"\" Type constraints \"\"\" f x Float64 x^2 ForwardDiff.derivative f, 2.0 question box md\"\"\" 1. How would you need to change `f` to avoid this error? 2. Does this error also occur with Enzyme? \"\"\" hint md\"\"\" ```julia f x x^2 ``` \"\"\" hint md\"\"\" The function `derivative enzyme` uses Enzyme. ```julia derivative enzyme f, 2.0 ``` \"\"\" md\"\"\" Mixing of types \"\"\" function g x A zeros 2, 2 A 1,1 x A 2,2 x det A end ForwardDiff.derivative g, 2.0 question box md\"\"\" 1. How would you need to change `g` to avoid this error? 2. Does this error also occur with Enzyme? \"\"\" hint md\"\"\" ```julia function g x A zeros typeof x , 2, 2 A 1,1 x A 2,2 x det A end ``` \"\"\" md\"\"\" Perturbation confusion Recall our small AD system from the lecture. \"\"\" begin struct Dual T Number Number value T deriv T end Dual x Real, y Real Dual promote type typeof x , typeof y promote x, y ... Base.convert Type Dual T , x Real where T Real Dual x, zero T Base.promote rule Type Dual T , Type Real where T Real Dual T Base. x Dual, y Dual Dual x.value y.value, x.deriv y.deriv Base. x Dual, y Dual Dual x.value y.value, x.deriv y.deriv Base. x Dual, y Dual Dual x.value y.value, x.value y.deriv x.deriv y.value Base. x Dual, y Dual Dual x.value y.value, x.deriv y.value x.value y.deriv y.value^2 Base.sin x Dual Dual sin x.value , cos x.value x.deriv Base.cos x Dual Dual cos x.value , sin x.value x.deriv Base.log x Dual Dual log x.value , x.deriv x.value Base.exp x Dual Dual exp x.value , exp x.value x.deriv NEW Base.one x Dual Dual one x.value , zero x.deriv Base. x Dual Dual x.value, x.deriv end begin derivative f, x f Dual x, one x .deriv derivative f x derivative f, x end md\"\"\" Higher order derivatives We can define higher order derivatives as a recursive application of AD. \"\"\" function nth derivative f F, x, Val N where F, N if N 0 return f x else return derivative y nth derivative f, y, Val N 1 , x end end nth derivative sin, 1.0, Val 0 sin 1.0 nth derivative sin, 1.0, Val 1 cos 1.0 nth derivative sin, 1.0, Val 2 sin 1.0 nth derivative sin, 1.0, Val 3 cos 1.0 function nth derivative forwarddiff f F, x, Val N where F, N if N 0 return f x else return ForwardDiff.derivative y nth derivative forwarddiff f, y, Val N 1 , x end end md\"\"\" If we have derivatives on different \"levels\" interact with each other we have to be careful. \\frac d dx x \\frac d dy x y \"\"\" md\"\"\" Manually differentiating \\frac d dx x \\frac d dy x y \\frac d dy x y 1 \\frac d dx x 1 1 Yet when we use our `derivative` function, we get a wrong answer of 2 . \"\"\" let D derivative D x x D y x y, 3. , 5. end md\"\"\" ForwardDiff.jl and Enzyme.jl get this right \"\"\" let D ForwardDiff.derivative D x x D y x y, 3. , 5. end let D derivative enzyme D x x D y x y, 3.0 , 5.0 end md\"\"\" Thinking about this in the language of dual numbers. D f, x f x ϵ .deriv We can evaluate this program through substitution. ```julia D x x D y x y, 3 , 5 substitute x with `5 ϵ` 5 ϵ D y 5 ϵ y, 3 .deriv substitute `y` with `3 ϵ` 5 ϵ 5 ϵ 3 ϵ .deriv .deriv substitute `y` with `3 ϵ` 5 ϵ 8 2ϵ .deriv .deriv Apply rule for 5 ϵ 2 .deriv extract deriv 10 2ϵ .deriv Apply rule for 2 Extract deriv ``` \"\"\" md\"\"\" We see an interaction between the inner and the outer derivative. \"\"\" md\"\"\" We fix this by introducing tagged \\epsilon \\epsilon 1^2 0 \\epsilon 2^2 0 \\epsilon 1 \\neq \\epsilon 2 \\neq \\epsilon 1\\epsilon 2 \\neq 0 But better just to use ForwardDiff or Enymze. \"\"\" "},{"url":"exercises/exercise_7_gradient_descent/","title":"Gradient descent","tags":["module1","track_ad","exercises"],"text":" A Pluto.jl notebook v0.20.13 frontmatter order \"4.3\" exercise number \"7\" title \"Gradient descent\" tags \"module1\", \"track ad\", \"exercises\" license \"MIT\" layout \"layout.jlhtml\" description \"sample exercise\" frontmatter.author name \"Valentin Churavy\" image \"https avatars.githubusercontent.com u 145258?v 4\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils begin using PlutoTeachingTools, PlutoUI using PlutoUI Slider PlutoUI.TableOfContents depth 4 end ChooseDisplayMode begin using CairoMakie set theme theme latexfonts fontsize 16, Lines linewidth 2, , markersize 16 end md\"\"\" Simple gradient descent Implementing proximal gradient descent for minimization of a squared error loss function Inspiration https discourse.julialang.org t types and gradients including forward gradient 946 \"\"\" using ForwardDiff gradient, derivative begin using LinearAlgebra using Random end model linear regression w, b, x w x . b loss function mean squared error ŷ, y sum abs2, ŷ . y size y,2 get gradient w.r.t to `w` loss∇w model, loss, w, b, x, y gradient w loss model w, b, x , y , w get derivative w.r.t to `b` `derivative` is used instead of `gradient` because `b` is a scalar instead of an array lossdb model, loss, w, b, x, y derivative b loss model w, b, x , y , b optimization algorithm function proximal gradient descent model, loss, w, b, x, y lr .1 w lmul lr, loss∇w model, loss, w, b, x, y b lr lossdb model, loss, w, b, x, y return w, b end function main T Array, n 100000 Random.seed 0 p 25 x randn n,p ' y sum x 1 5, dims 1 . randn n ' 0.1 w 0.0001 randn 1,p b 0.0 x T x y T y w T w model linear regression loss mean squared error errs loss model w,b,x ,y time for i 1 p w, b proximal gradient descent model, loss, w, b, x, y push errs, loss model w,b,x ,y end errs end errs main Array lines errs md\"\"\" Exercises \"\"\" md\"\"\" Exercise Customization Play around with this code and define your own model or loss function \"\"\" md\"\"\" Exercise GPU Arrays \"\"\" question box md\"\"\" Cast your mind back to the lecture on parallel computing. This code only uses array based abstractions. Is it generic enough that it just works with different array types? \"\"\" md\"\"\" Try using something like a `CuArray` or other GPU types. note If it doesn't work with your library of choice? What do you need to fix? Maybe you could file an issue or pull request? \"\"\" md\"\"\" Exercise Different AD frameworks We are using `ForwardDiff.gradient` here, as we discussed in class this is not efficient for many parameters. Can you use `Enzyme.gradient Reverse, ... `? \"\"\" "},{"url":"exercises/exercise_8_github/","title":"Github","tags":["module2","track_principles","exercises"],"text":" A Pluto.jl notebook v0.20.13 frontmatter order \"5.1\" exercise number \"8\" title \"Github\" tags \"module2\", \"track principles\", \"exercises\" license \"MIT\" layout \"layout.jlhtml\" description \"sample exercise\" frontmatter.author name \"Valentin Churavy\" image \"https avatars.githubusercontent.com u 145258?v 4\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils begin using PlutoTeachingTools, PlutoUI using PlutoUI Slider PlutoUI.TableOfContents depth 4 end md\"\"\" Exercise Github \"\"\" md\"\"\" Learn Git Branching Learn Git Branching https learngitbranching.js.org is a niffty little website to explore Git concepts. If you want you may spend a few minutes exploring it. \"\"\" md\"\"\" Getting started on Github 1. Create a Github account 2. On Windows I have few personal experiences, but I suspect you might want to stick to Github Desktop or the VSCode integration 3. On Mac Linux Optional Setup a SSH key https docs.github.com en authentication connecting to github with ssh generating a new ssh key and adding it to the ssh agent \"\"\" md\"\"\" Setup a repository for your Project Minimum features README.md LICENSE.md .gitignore Most likely Basic Julia project structure `src ` Project.toml Julia focused CI infrastructure Tests Documentation Benchmarking \"\"\" md\"\"\" Contribute to another Github repository Grab a friend neighbor and open a pull request to their repository \"\"\" "},{"url":"exercises/exercise_9_debugging/","title":"Debugging","tags":["module2","track_principles","exercises"],"text":" A Pluto.jl notebook v0.20.13 frontmatter order \"7.1\" exercise number \"9\" title \"Debugging\" tags \"module2\", \"track principles\", \"exercises\" license \"MIT\" layout \"layout.jlhtml\" description \"sample exercise\" frontmatter.author name \"Valentin Churavy\" image \"https avatars.githubusercontent.com u 145258?v 4\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils import ForwardDiff using PlutoTeachingTools md\"\"\" Exercise Debugging directional derivatives \"\"\" function jvp F, u, v, ϵ sqrt eps F u ϵ . v F u . ϵ end function jacobian F, u w F u v similar u J similar u, length u , length w for i in length u v i 1 J , i jvp F, u, v end J end F1 x x 1 ^4 3 exp x 2 2 jacobian F1, 1.0, 2.0 let J jacobian F1, 1.0, 2.0 ref J ForwardDiff.jacobian F1, 1.0, 2.0 if J ≈ ref J correct elseif J 1,1 ≈ 4.0 && J 2,2 ≈ exp 2.0 keep working md\"The answer is not quite right Check your diagonal entries\" elseif J 1,2 0.0 keep working md\"The answer is not quite right Think back to one hot vectors\" else keep working end end F2 x x 1 ^4 3 exp x 2 2 log x 1 x 2 ^2 jacobian F2, 1.0, 2.0 let success false try J jacobian F2, 1.0, 2.0 success true catch end if success keep working md\"Your function is erroring \" else ref J ForwardDiff.jacobian F2, 1.0, 2.0 if J ≈ ref J correct elseif J 1,1 ≈ 4.0 && J 2,2 ≈ exp 2.0 keep working md\"The answer is not quite right Check your diagonal entries\" elseif J 3,1 ≈ 1.0 && J 3,2 ≈ 4.0 keep working md\"The new entries don't match my expectation.\" elseif J 1,2 0.0 keep working md\"The answer is not quite right Think back to one hot vectors\" else keep working end end end md\"\"\" Exercise Linked List \"\"\" mutable struct LinkedList T const value T next Union Nothing, LinkedList T LinkedList value T where T new T value, nothing end function collect ll LinkedList values ll.value while ll.next nothing push values, ll.next.value end values end ll LinkedList 1.0 collect ll md\"\"\" Let's write a function that appends values \"\"\" function append ll LinkedList, value Find tail while ll.next nothing ll ll.next end ll.next LinkedList value end let ll LinkedList 1.0 append ll, 2.0 TODO collect ll hangs ll end \"\"\" insert ll, after, value Insert a value after another \"\"\" function insert ll LinkedList, after, value while ll nothing if ll.value after ll.next LinkedList value break else ll ll.next end end end function delete ll LinkedList, value while ll nothing if ll.value value ll.next nothing break else ll ll.next end end end let ll LinkedList 1.0 append ll, 3.0 insert ll, 1.0, 2.0 OOPS where did 3.0 go? ll end let ll LinkedList 1.0 append ll, 2.0 append ll, 3.0 delete ll, 2.0 OOPS where did 3.0 go, and why is 2.0 still here? ll end "},{"url":"mod1_introduction/automatic_differentiation/","title":"Automatic Differentiation","tags":["module1","track_ad"],"text":" A Pluto.jl notebook v0.20.13 frontmatter chapter \"1\" section \"4\" order \"4\" title \"Automatic Differentiation\" date \"2025 04 16\" tags \"module1\", \"track ad\" layout \"layout.jlhtml\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils This Pluto notebook uses bind for interactivity. When running this notebook outside of Pluto, the following 'mock version' of bind gives bound variables a default value instead of an error . macro bind def, element format off return quote local iv try Base.loaded modules Base.PkgId Base.UUID \"6e696c72 6542 2067 7265 42206c756150\" , \"AbstractPlutoDingetjes\" .Bonds.initial value catch b missing end local el esc element global esc def Core.applicable Base.get, el ? Base.get el iv el el end format on end begin using PlutoUI, PlutoTeachingTools using PlutoUI Slider PlutoUI.TableOfContents depth 4 end ChooseDisplayMode begin using CairoMakie set theme theme latexfonts fontsize 16, Lines linewidth 2, , markersize 16 end md\"\"\" Introduction to AD with material from Hendrick Ranocha, Adrian Hill, and Alan Edelman \"\"\" md\"\"\" What is a derivative? From your calculus classes, you might recall that a function f \\mathbb R \\rightarrow \\mathbb R is differentiable at \\tilde x if there is a number f' \\tilde x such that \\lim h \\rightarrow 0 \\frac f \\tilde x h f \\tilde x h f' \\tilde x \\quad . This number f' \\tilde x is called the derivative of f at \\tilde x . Let's visualize this on a simple scalar function \"\"\" f x x^2 5 sin x 10 you can change this function bind x̂ Slider 5 0.2 5, default 1.5, show value true let fig Figure ax Axis fig 1,1 , xlabel L\"x\" Plot function xs range 5, 5, 50 ymin, ymax extrema f. xs ylims ax, ymin 5, ymax 5 lines ax, xs, f, label L\"Function f x \" Obtain the function f′ f′ derivative f Plot f′ x lines ax, xs, f′ label L\"Derivative f′ x \" Plot 1st order Taylor series approximation taylor approx x f x̂ f′ x̂ x x̂ f x ≈ f x̃ f′ x̃ x x̃ lines ax, xs, taylor approx label L\"Taylor approx. around \\tilde x \" Show point of linearization vlines ax, x̂ color grey, linestyle dash, label L\"\\tilde x \" axislegend ax, position ct fig end md\"\"\" Partial derivatives For a multi variable function like `f x, y ` we define \\frac \\partial \\partial x f as the rate of change in the x direction, and likewise \\frac \\partial \\partial y f as the rate of change in the y direction. \"\"\" f2 x, y x^2 y^3 5 sin x 5 cos y x 20 you can change this function bind x̂₁ Slider 5 0.2 5, default 1.5, show value true bind ŷ₁ Slider 5 0.2 5, default 1.5, show value true let fig Figure ax Axis3 fig 1,1 , xlabel L\"x\", ylabel L\"y\", zlabel L\"z\", title \"f x,y \" ax4 Axis3 fig 2,1 , xlabel L\"x\", ylabel L\"y\", zlabel L\"z\", title \"Taylor approx. around x̂ x̂₁, ŷ ŷ₁\" ax2 Axis fig 1, 2 , title \"Partial deriv. in x at ŷ ŷ₁\", xlabel L\"x\" ax3 Axis fig 2, 2 , title \"Partial deriv. in y at x̂\", xlabel L\"y\" Plot function xs range 5, 5, 50 ys range 5, 5, 50 zmin, zmax extrema f2. xs, ys' zlims ax, zmin 5, zmax 5 surface ax, xs, ys, f2 lines ax, xs, f, label L\"Function f x \" Obtain the partial derivative functions f′ f x′ derivative x f2 x, ŷ₁ f y′ derivative y f2 x̂₁, y Plot partial f′ x lines ax2, xs, f x′ label L\"Derivative f x′ x \" lines ax3, ys, f y′ label L\"Derivative f x′ x \" Plot 1st order Taylor series approximation taylor approx x, y f2 x̂₁, ŷ₁ f x′ x̂₁ x x̂₁ f y′ ŷ₁ y ŷ₁ surface ax4, xs, ys, taylor approx label L\"Taylor approx. around \\tilde x ,\\tilde y \" fig end md\"\"\" Motivating example In many areas of science we encounter models with unkown parameters Curve fitting Machine learning f x p \\cdots We are often interested in learning these parameters given a training dataset or real world observation. Below we study a model that is the combination of a `sin` and `cos` function with four parameters. \"\"\" function m x, a, b, c, d return sin a x b cos c x d end xs 0.0 0.01 2π function multi slider names, values return PlutoUI.combine do Child inputs md\"\"\" name Child name, Slider vals show value true, default first vals last vals 2 \"\"\" for name, vals in zip names, values md\"\"\" inputs \"\"\" end end bind coeffs multi slider \"a\", \"b\", \"c\", \"d\" , 1.0 0.1 2.0, 1.0 0.1 2.0, 1.0 0.1 2.0, 1.0 0.1 2.0, let fig Figure ax Axis fig 1,1 lines ax, xs, sin, label \"sin\" lines ax, xs, cos, label \"cos\" lines ax, xs, m. xs, coeffs... , label \"model\" axislegend ax fig end question box md\"\"\" Given an \"observed\" evaluation of `m` can we \"learn\" the values of `a`, `b`, `c`, `d`? \"\"\" ys m. xs, 0.3, 1.2, 0.5, 0.7 lines xs, ys md\"\"\" We could try some random values \"\"\" coeffs guess rand 2.0 0.1 2.0, 4 ys guess m. xs, coeffs guess... md\"\"\" We need to define a \"loss\" a function that measures how far away we are from our solution. The Mean Squared Error is a common choice. \"\"\" Mean squared error function mse ŷ, y sum ŷ . y .^2 length y end mse ys, ys guess let fig Figure ax Axis fig 1,1 , title \"MSE mse ys, ys guess \" lines ax, xs, ys lines ax, xs, ys guess fig end md\"\"\" So how do we improve our guess systematically? \"\"\" let neighborhood 0.3 0.1 0.3 fig Figure ax Axis fig 1,1 for offset in neighborhood ys m. xs, coeffs guess 1 offset, coeffs guess 2 end ... lines ax, xs, ys, label \"Offset a offset , MSE mse ys, ys \" end axislegend ax fig end tip md\"\"\" We want to change a parameter, such that we are improving. Randomly trying certainly would get us \"somewhere\" Monte carlo methods Evolutionary algorithms But what we are after is the \"rate of change\" in the error given for a parameter or all parameters . \"\"\" let fig Figure ax Axis fig 1,1 , xlabel L\"a\" Plot function as range 2, 2, 100 f a mse ys, m. xs, a, coeffs guess 2 end ... ymin, ymax extrema f. as ylims ax, ymin 5, ymax 5 lines ax, as, f, label L\"Function f mse ys, m x a, b, c, d \" Obtain the function f′ f′ derivative f Plot f′ x lines ax, as, f′ label L\"Derivative f′ a \" â coeffs guess 1 Plot 1st order Taylor series approximation taylor approx a f â f′ â a â f x ≈ f x̃ f′ x̃ x x̃ lines ax, as, taylor approx label L\"Taylor approx. around \\tilde a \" Show point of linearization vlines ax, â color grey, linestyle dash, label L\"\\tilde a \" axislegend ax, position ct fig end tip md\"\"\" Today we are focusing on \"forward mode\" automatic differentiation. This means that we conceptualize calculating derivatives by applying an infinitesimal perturbation to an argument of a function. As we will see for a multi variable problem like ours this means we need to calculate the partial derivative for each parameter \"seperatly\". In machine learning one often encounters \"reverse mode\" automatic differentiation, which conceptually applies a perturbation to the \"return value\" of a function. We will revisit this in the Lecture \"Automatic Differentiation and Machine Learning\" \"\"\" md\"\"\" Finite differences There are several ways to compute a function and its derivative on a computer. If you use a computer algebra system CAS , you can compute derivatives analytically, e.g., using Wolfram Alpha https www.wolframalpha.com . Another option you should have seen in the introduction to numerical analysis are finite differences. Since the derivative is defined as f' x \\lim h \\to 0 \\frac f x h f x h , it makes sense to use the forward difference \\frac f x h f x h \\approx f' x and approximate the limit by taking a small h 0 . However, this leads to round off error since we typically represent real numbers via floating point numbers with fixed precision . \"\"\" nextfloat 1.0 1.0 eps 1.0 eps 1.0f0 nextfloat 1.234e5 1.234e5 eps 1.234e5 md\"\"\" Thus, there will be two regimes if h is too big, the error of the limit approximation dominates if h is too small, the floating point error dominates We illustrate this for different functions f at x 1 . We use different types of floating point numbers and compute the error of the finite difference approximation. \"\"\" using DoubleFloats bind f diff Select sin \"f x sin x \", cos \"f x cos x \", exp \"f x exp x \", x sin 100 x \"f x sin 100 x \", x sin x 100 \"f x sin x 100 \", bind FloatType Select Float32, Float64, Double64 default Float64 let fig Figure ax Axis fig 1, 1 xlabel L\"Step size h \", ylabel \"Error of the forward differences\", xscale log10, yscale log10 f f diff x one FloatType f′x derivative f, Float64 x h FloatType. 10.0 .^ range 20, 0, length 500 fd error h max abs f x h f x h f′x , eps x 100 lines ax, h, fd error. h label \"\" h def sqrt eps x scatter ax, h def , fd error h def color gray text ax, \"sqrt eps x \" position 5 h def, fd error h def , space data fig end md\"\"\" Next, we use the central difference \\frac f x h f x h 2 h \\approx f' x . \"\"\" let fig Figure ax Axis fig 1, 1 xlabel L\"Step size h \", ylabel \"Error of the central differences\", xscale log10, yscale log10 f f diff x one FloatType f′x, derivative f, Float64 x h FloatType. 10.0 .^ range 20, 0, length 500 fd error h max abs f x h f x h 2 h f′x , eps x 100 lines ax, h, fd error. h label \"\" h def cbrt eps x scatter ax, h def , fd error h def color gray text ax, \"cbrt eps x \" position 5 h def, fd error h def , space data fig end md\"\"\" Forward mode AD for scalars There is a well know proverb Differentiation is mechanics, integration is art Luckily, we are just interested in differentiation for now. Thus, all we need to do is to implement the basic rules of calculus like the product rule and the chain rule. Before doing that, let's consider an example. \"\"\" g x log x^2 exp sin x md\"\"\"We can compute the derivative by hand using the chain rule.\"\"\" g′ x 1 x^2 exp sin x 2 x exp sin x cos x md\"We can think of the function as a kind of computational graph obtained by dividing it into steps.\" function g graph x c1 x^2 c2 sin x c3 exp c2 c4 c1 c3 c5 log c4 return c5 end g 1.0 ≈ g graph 1.0 md\"To compute the derivative, we have to apply the chain rule multiple times.\" function g graph derivative x c1 x^2 c1 ε 2 x c2 sin x c2 ε cos x c3 exp c2 c3 ε exp c2 c2 ε c4 c1 c3 c4 ε c1 ε c3 ε c5 log c4 c5 ε c4 ε c4 return c5, c5 ε end g graph derivative 1.0 g 1.0 , g′ 1.0 let x 1.0, h sqrt eps g x h g x h end md\"\"\" We would like to automate this To do so, we introduce so called dual numbers https en.wikipedia.org wiki Dual number . They carry both a `value` and derivative called `deriv`, the ε part above . Formally, a dual number can be written as x \\varepsilon y, \\qquad x, y \\in \\mathbb R , quite similar to a complex number z x \\mathrm i y, \\qquad x, y \\in \\mathbb R . However, the new basis element \\varepsilon satisfies \\varepsilon^2 0 instead of \\mathrm i ^2 1 . Thus, the dual number have the algebraic structure of an algebra instead of a field like the complex numbers \\mathbb C . In our applications, the \\varepsilon part contains the derivative. Indeed, the rule \\varepsilon^2 0 yields a \\varepsilon b c \\varepsilon d a c \\varepsilon a d b c , which is just the product rule of calculus. You can code this as follows. \"\"\" begin struct Dual T Real Number value T deriv T end Dual x Real, y Real Dual promote x, y ... end md\"Now, we can create such dual numbers.\" Dual 5, 2.0 md\"Next, we need to implement the required interface methods for numbers.\" Base. x Dual, y Dual Dual x.value y.value, x.deriv y.deriv Dual 1, 2 Dual 2.0, 3 Base. x Dual, y Dual Dual x.value y.value, x.deriv y.deriv Dual 1, 2 Dual 2.0, 3 Base. x Dual, y Dual Dual x.value y.value, x.value y.deriv x.deriv y.value Dual 1, 2 Dual 2.0, 3 Base. x Dual, y Dual Dual x.value y.value, x.deriv y.value x.value y.deriv y.value^2 Dual 1, 2 Dual 2.0, 3 md\"We also need to tell Julia how to convert and promote our dual numbers.\" Base.convert Type Dual T , x Real where T Real Dual x, zero T Base.promote rule Type Dual T , Type Real where T Real Dual T Dual 1, 2 3.0 md\"Next, we need to implement the well know derivatives of special functions.\" Base.sin x Dual Dual sin x.value , cos x.value x.deriv sin Dual π, 1.0 Base.cos x Dual Dual cos x.value , sin x.value x.deriv cos Dual π, 1.0 Base.log x Dual Dual log x.value , x.deriv x.value log Dual 1.0, 1 Base.exp x Dual Dual exp x.value , exp x.value x.deriv exp Dual 1.0, 1 Base.abs x Dual Dual abs x.value , sign x.value md\"Finally, we can differentiate the function `f` we started with \" let g dual g Dual 1.0, 1.0 g dual.value, g dual.deriv . g 1.0 , g′ 1.0 end let g dual g Dual 1.0, 1.0 g dual.value, g dual.deriv . g graph derivative 1.0 end md\"This works since the compiler basically performs the transformation `f` \\to `f graph derivative` for us. We can see this by looking at one stage of the Julia compilation process as follows.\" code typed g Dual 1.0, 1.0 code typed g graph derivative 1.0 md\"Since the compiler can see all the different steps, it can generate very efficient code.\" using BenchmarkTools benchmark g graph derivative Ref 1.0 benchmark g Dual Ref 1.0 , 1.0 md\"Now, we have a versatile tool to compute derivatives of functions depending on a single variable.\" derivative f, x Real f Dual x, one x .deriv derivative g, 1.0 derivative x 3 x^2 4 x 5, 2 derivative 3 do x sin x log x end md\"We can also get the derivative as a function itself.\" derivative f x derivative f, x let dg derivative g x range 0.1, 10.0, length 10 dg. x g′. x end let fig Figure ax Axis fig 1,1 lines ax, xs, f diff, label \"f\" lines ax, xs, derivative f diff , label \"f′\" axislegend ax fig end tip md\"\"\" Julia has a robust ecosystem of automatic differentiation tools Do not handroll your own library like we did here, but instead use ForwardDiff.jl https github.com JuliaDiff ForwardDiff.jl or Enzyme.jl https github.com EnzymeAD Enzyme.jl . \"\"\" md\"\"\" Popping up the stack \"\"\" dm a x, a, b, c, d m x, Dual a, one a , b, c, d dm b x, a, b, c, d m x, a, Dual b, one b , c, d dm c x, a, b, c, d m x, a, b, Dual c, one c , d dm d x, a, b, c, d m x, a, b, c, Dual d, one d mse ys, dm a. xs, coeffs guess... mse ys, dm b. xs, coeffs guess... mse ys, dm c. xs, coeffs guess... mse ys, dm d. xs, coeffs guess... md\"\"\" We can also write this in vector form \"\"\" const one hot vectors Dual 0.0, 1.0 , 0.0, 0.0, 0.0 , 0.0 ,Dual 0.0, 1.0 , 0.0, 0.0 , 0.0, 0.0, Dual 0.0, 1.0 , 0.0 , 0.0, 0.0, 0.0, Dual 0.0, 1.0 dm loss, ys, xs, coeffs map v loss ys, m. xs, v... .deriv, map v v. coeffs, one hot vectors dm mse, xs, ys, coeffs guess... md\"\"\" note The need for 4 function evaluation since we have 4 function arguments. \"\"\" begin learning rate 0.01 steps 1000 plot every 100 end let fig Figure ax Axis fig 1,1 coeffs rand 4 errs Float64 for i in 1 steps ys m. xs, coeffs... err mse ys, ys push errs, err if mod1 i, plot every 1 lines xs, ys, label \"Epoch i\" end dcoeffs dm mse, ys, xs, coeffs coeffs . learning rate . dcoeffs end ys m. xs, coeffs... err mse ys, ys lines xs, ys, label \"Epoch steps 1 \" lines xs, ys, label \"Goal\" axislegend ax ax2 Axis fig 2,1 lines ax2, errs fig end "},{"url":"mod1_introduction/matrices/","title":"Matrix multiply","tags":["module1","track_performance","indepth"],"text":" A Pluto.jl notebook v0.20.13 frontmatter chapter \"1\" section \"3\" order \"3.1\" title \"Matrix multiply\" tags \"module1\", \"track performance\", \"indepth\" layout \"layout.jlhtml\" indepth number \"1\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils using PlutoUI, PlutoTeachingTools md\"\"\" Experiments with Memory Access and Matrices based on https github.com stevengj 18a01 blob master memory matrices.ipynb In many problems, especially problems accessing lots of data and doing relatively simple computations on each datum, the performance bottleneck is memory rather than computational speed. Because memory is arranged into a memory hierarchy of larger slower and smaller faster memories, it turns out that changing the order of memory access can have a huge impact on performance. \"\"\" md\"\"\" Benchmarking Matrix Multiplication One of the most basic building blocks of numerical linear algebra is the computation of matrix multiplication given an m \\times n matrix A and an n \\times p matrix B , compute the m \\times p matrix C AB . The entries of C are given by the exact formula C ik \\\\sum j 1 ^n A ij B jk but there are many ways to implement this computation. \\approx 2mnp flops floating point additions and multiplications are required, but they can re ordered arbitrarily, leading to \\sim mnp possible orderings. It turns out that the ordering of the operations in the matrix multiplication has a huge impact on performance, along with low level details of the inner loops. Basically, three factors make the implementation of efficient matrix multiplication highly nontrivial Caches https en.wikipedia.org wiki CPU cache the matrix accesses must be reordered to obtain temporal locality https en.wikipedia.org wiki Locality of reference and hence efficient memory cache usage. Registers https en.wikipedia.org wiki Processor register at the lowest level, the CPU registers form a kind of ideal cache. The innermost loops of the matrix multiplication need to be unrolled in order to load many values into registers and perform as much work with them as possible essentially a small submatrix multiplication . It turns out that a lot of tricks http cscads.rice.edu workshops july2007 autotune slides 07 Frigo.pdf are required to do this well. SIMD instructions https en.wikipedia.org wiki SIMD modern CPUs include special instructions that can perform several arithmetic operations at once e.g. 2, 4, or even 8 `Float64` operations , and to get the full benefit of these operations typically requires hand coding. As a consequence, there is a huge performance gap between the most obvious three loop matrix multiplication code and highly optimized code. This gap has become the central factor in the design of dense linear algebra libraries for several decades, especially the industry standard free open source the LAPACK https en.wikipedia.org wiki LAPACK library nearly all dense linear algebra is now organized around highly optimized BLAS https en.wikipedia.org wiki Basic Linear Algebra Subprograms libraries Because Julia benefits from fast compilers, we can illustrate this performance gap fairly with simple Julia code. In contrast, similar implementation in Matlab or Python would be orders of magnitude slower, and would demonstrate mostly language rather than the algorithmic effects. \"\"\" md\" Naive algorithm The following is the simplest, most obvious, matrix multiplication algorithm just three nested loops, implementing a dot product for each output C ik The only concessions we have made to performance concerns here are 1 we implement an in place matmul variant that operates on a pre existing C array, to avoid benchmarking the memory allocation deallocation and 2 we use the ` inbounds` macro to turn off array bounds checking in Julia for the inner loop. Together, these make less than a factor of two difference in speed. \" compute C A B, using naive matrix multiplication algorithm, with a pre allocated output array C. \" \" is a Julia convention for functions that modify their arguments. function matmul C, A, B m,n size A n,p size B size C m,p || error \"incorrect dimensions \", size C , \" ≠ p\" for i 1 m for k 1 p c zero eltype C for j 1 n inbounds c A i,j B j,k end inbounds C i,k c end end return C end a wrapper that allocates C of an appropriate type matmul A, B matmul Array promote type eltype A , eltype B undef, size A,1 , size B,2 , A, B using LinearAlgebra, Random begin correctness check A rand 5,6 B rand 6,7 norm matmul A,B A B end md\"\"\" Benchmarking naive `matmul` Here, we will benchmark our `naive` matmul implementation against the highly optimized OpenBLAS library that Julia uses for its built in matrix multiplication. Like `matmul `, we will call OpenBLAS with pre allocated output via `mul C, A, B ` instead of the simpler `A B`. By default, OpenBLAS uses multiple CPU cores, which gives it an \"unfair\" parallel speedup, but we can disable this for benchmarking purposes \"\"\" for benchmarking, use only single threaded BLAS begin BLAS.set num threads 1 blas threads nothing end function logspace start, stop, length exp10. range start stop, length end begin blas threads N round. Int, logspace 1, log10 3000 , 60 60 sizes from 10 to 3000 alternatively, use N 10 1000 to see some interesting patterns due to cache associativity etc. t Float64 t0 Float64 for n in N local A zeros n,n local B zeros n,n preallocate output C so that allocation is not included in timing C zeros n,n push t, elapsed matmul C,A,B push t0, elapsed LinearAlgebra.mul C,A,B println \"finished n n slowdown of \", t end t0 end end end md\"\"\" Now, we will plot the results. Since the number of flops is 2n^3 , we will plot 2n^3 t for time t in microseconds in order to plot the gigaflops rate billions of flops per second . If you naively think of a CPU as a box that performs floating point instructions at a fixed rate, with all other instructions being negligible a picture that may have been true circa 1985 , this would be a flat horizontal line independent of n , but we will see that reality is quite different. The OpenBLAS library gets an \"unfair\" factor of 8 speedup on typical modern Intel processors thanks to hand coded support for AVX 512 https en.wikipedia.org wiki Advanced Vector Extensions SIMD instructions, which perform 8 double precision floating point operations simultaneously. \"\"\" using CairoMakie begin blas threads peakflops 1e 9 end let fig Figure ax Axis fig 1, 1 , title \"\", xlabel \"matrix size n\", ylabel L\"\\text gigaflops~ \\frac 2n^3 t \", yscale Makie.pseudolog10, yticks 0, 1, 2, 3, 4, 5, 25, 50, 75 lines ax, N, 2N.^3 . t . 1e 9, label \"naive matmul\" lines ax, N, 2N.^3 . t0 . 1e 9, label \"BLAS matmul\" axislegend ax, position rc fig end md\"\"\" Cache oblivious matrix multiplication As a first step in the right direction, we'll implement a cache oblivious algorithm https en.wikipedia.org wiki Cache oblivious algorithm for matrix multiplication divide the matrices into four submatrices which are multiplied recursively until a sufficiently large base case is reached large enough to amortize the recursion overhead . This strategy erases the steep performance drop off that occurs for large n where the matrix goes out of cache, at the cost of ~25 lines of code rather than ~10 for the naive loops. It still doesn't match the OpenBLAS performance because it fails to address the other two problems unrolling and optimizing the base cases to optimize register utilization, and coding for SIMD instructions. \"\"\" function add matmul rec m,n,p, i0,j0,k0, C,A,B if m n p 64 base case naive matmult for sufficiently large matrices for i 1 m for k 1 p c zero eltype C for j 1 n inbounds c A i0 i,j0 j B j0 j,k0 k end inbounds C i0 i,k0 k c end end else m2 m ÷ 2 n2 n ÷ 2 p2 p ÷ 2 add matmul rec m2, n2, p2, i0, j0, k0, C, A, B add matmul rec m m2, n2, p2, i0 m2, j0, k0, C, A, B add matmul rec m2, n n2, p2, i0, j0 n2, k0, C, A, B add matmul rec m2, n2, p p2, i0, j0, k0 p2, C, A, B add matmul rec m m2, n n2, p2, i0 m2, j0 n2, k0, C, A, B add matmul rec m2, n n2, p p2, i0, j0 n2, k0 p2, C, A, B add matmul rec m m2, n2, p p2, i0 m2, j0, k0 p2, C, A, B add matmul rec m m2, n n2, p p2, i0 m2, j0 n2, k0 p2, C, A, B end return C end function matmul rec C, A, B m,n size A n,p size B size C m,p || error \"incorrect dimensions \", size C , \" ≠ p\" fill C, 0 return add matmul rec m,n,p, 0,0,0, C,A,B end matmul rec A, B matmul rec Array promote type eltype A , eltype B undef, size A,1 , size B,2 , A, B let correctness check A rand 50,60 B rand 60,70 norm matmul rec A,B A B end begin tco Float64 for n in N local A zeros n,n local B zeros n,n preallocate output C so that allocation is not included in timing C zeros n,n push tco, elapsed matmul rec C,A,B println \"finished n n slowdown of \", tco end t0 length tco end end let fig Figure ax Axis fig 1, 1 , title \"\", xlabel \"matrix size n\", ylabel L\"\\text gigaflops~ \\frac 2n^3 t \", yscale Makie.pseudolog10, yticks 0, 1, 2, 3, 4, 5, 25, 50, 75 lines ax, N, 2N.^3 . t . 1e 9, label \"naive matmul\" lines ax, N, 2N.^3 . t0 . 1e 9, label \"BLAS matmul\" lines ax, N, 2N.^3 . tco . 1e 9, label \"Cache Oblivious matmul\" axislegend ax, position rc fig end "},{"url":"mod1_introduction/parallelism/","title":"Parallelism","tags":["module1","track_parallel"],"text":" A Pluto.jl notebook v0.20.13 frontmatter chapter \"1\" section \"2\" order \"2\" title \"Parallelism\" date \"2025 04 23\" tags \"module1\", \"track parallel\" layout \"layout.jlhtml\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils using PlutoUI, PlutoTeachingTools using CairoMakie ChooseDisplayMode PlutoUI.TableOfContents depth 4 md\"\"\" Parallelism \"\"\" md\"\"\" Why do we need parallelism in the first place? \"\"\" md\"\"\" What has your processor done for you recently As programmers we have the mental model that a processor executes our program in linear order Processors are Out of order Superscalar Predictive Micro ops \"\"\" md\"\"\" An idealized processor RobustLocalResource \"https s3.amazonaws.com media p.slid.es uploads 779853 images 4327544 pasted from clipboard.png\", \"arch.png\" \"\"\" function my dot a, b acc zero eltype a inbounds for i in eachindex a,b acc a i b i end return acc end with terminal do code native debuginfo none my dot zeros 3 , zeros 3 end md\"\"\" End of Moore's Law Roughly Processors keep getting faster, but until when? \"\"\" RobustLocalResource \"https www.alleywatch.com wp content uploads 2023 01 screen shot 2017 03 03 at 1 59 21 pm.png\", \"moore.png\" md\"\"\" Notions of scalability \"\"\" md\"\"\" Speedup Strong scalability Given a program that can execute with some compute units N , we define t N as the total time the program takes. note Computers have many different ways of measure time, total time or wall clock time is the 'human' experienced time. \\text Speedup \\frac t 1 t N The time it takes for a program to execute with one compute unit, vs N compute units. \"\"\" md\"\"\" Amdahl's law Given a function f that consists of a parallel portion p and a serial portion s . In 1967, Amdahl pointed out that the speedup is limited by the fraction of the serial part of the software that is not amenable to parallelization. \\text Speedup \\frac 1 S P N Amdahl’s law states that, for a fixed problem, the upper limit of speedup is determined by the serial fraction of the code. \"\"\" speedup s, p, N 1 s p N let fig, ax lines 1 16, N speedup 0, 1, N , label \"0%\" lines ax, 1 16, N speedup .01, .99, N , label \"1%\" lines ax, 1 16, N speedup .05, .95, N , label \"5%\" lines ax, 1 16, N speedup .1, .9, N , label \"10%\" lines ax, 1 16, N speedup .2, .8, N , label \"20%\" lines ax, 1 16, N speedup .4, .6, N , label \"40%\" lines ax, 1 16, N speedup .6, .4, N , label \"60%\" lines ax, 1 16, N speedup .8, .2, N , label \"80%\" lines ax, 1 16, N speedup 1, 0, N , label \"100%\" fig 1, 2 Legend fig, ax, \"Serial\", framevisible false fig end md\"\"\" note For strong scalability the problem size stays constant, and we only vary the amount of compute available. \"\"\" md\"\"\" Weak Scaling Efficiency Given a program that can do N work with N compute units , we define t N as the total time the program takes. \\text Efficiency \\frac t 1 t n note The definition of t N includes the amount of work being scaled up \"\"\" RobustLocalResource \"https raw.githubusercontent.com eth cscs ImplicitGlobalGrid.jl master docs src assets images fig parEff HM3D Julia CUDA all Daint extrapol.png\", \"parallel efficiency.png\" md\"\"\" Node level This notebook uses Threads.nthreads threads \"\"\" with terminal do Sys.cpu summary end using Hwloc with terminal do Hwloc.topology end TODO \"Explain Hyper threading\" md\"\"\" Shared memory parallelism Julia is task based `M N` threading, green threads `Channel`, locks and atomics ```julia function pfib n Int if n 1 return n end t Threads. spawn pfib n 2 return pfib n 1 fetch t Int end ``` ```julia using Base.Threads threads function prefix threads ⊕, y AbstractVector l length y k ceil Int, log2 l do reduce phase for j 1 k threads for i 2^j 2^j min l, 2^k inbounds y i y i 2^ j 1 ⊕ y i end end do expand phase for j k 1 1 1 threads for i 3 2^ j 1 2^j min l, 2^k inbounds y i y i 2^ j 1 ⊕ y i end end return y end A fill 1, 500 000 prefix threads , A ``` From Nash et al., 2021 . Basic Threading Examples in JuliaLang v1.3. JuliaCon Proceedings, 1 1 , 54, https doi.org 10.21105 jcon.00054 \"\"\" function myfun s 0.0 N 10000 for i in 1 N s det randn 3,3 end s N end function bench f, N 10 a zeros N Threads. threads for i in 1 length a a i f end a end using BenchmarkTools, LinearAlgebra function bench Serial f, N 10 a zeros N for i in 1 length a a i f end a end time bench Serial myfun, 1000 time bench myfun, 1000 benchmark bench myfun, 1000 samples 10 evals 3 question box md\"What are potential issues with `myfun`\" answer box md\"\"\" Lot's of memory allocation in the hot loop How is `det` implemented? \\ ` which det zeros 3,3 ` `det lufact A ` `det` calls into BLAS \"\"\" md\"\"\" Composable parallelism \"\"\" question box md\"Who is in charge of parallelism?\" md\"\"\" Potential answers The user? The library? Are they all written in Julia? \"The system\" \"\"\" md\"\"\" As an example for linear algebra Julia uses OpenBLAS. OpenBLAS manages it's own thread pool `BLAS.set num threads`. We can run into contention issues when we use Julia own parallelism using tasks and system libraries. \"\"\" using StaticArrays function myfun improved s 0.0 N 10000 for i in 1 N s det randn SMatrix 3,3 end s N end benchmark bench myfun improved, 1000 samples 10 evals 3 benchmark bench Serial myfun improved, 1000 samples 10 evals 3 md\"\"\" Accelerated \"\"\" md\"\"\" A GPU has many \"lightweight\" threads \"\"\" md\"\"\" CPU die shot RobustLocalResource \"https s3.amazonaws.com media p.slid.es uploads 779853 images 4399827 800px coffee lake die quad core annotated .png\", \"quad core.png\" \"\"\" md\"\"\" GPU block diagram RobustLocalResource \"https devblogs.nvidia.com parallelforall wp content uploads 2016 04 gp100 block diagram 1 624x368.png\", \"p100.png\" \"\"\" md\"\"\" Bottlenecks RobustLocalResource \"https s3.amazonaws.com media p.slid.es uploads 779853 images 4399848 pasted from clipboard.png\", \"gpu system.png\" \"\"\" md\"\"\" Composable infrastructure Core GPUCompiler.jl Takes native Julia code and compiles it directly to GPUs GPUArrays.jl High level array based common functionality KerneAbstractions.jl Vendor agnostic kernel programming language Adapt.jl Translate complex structs across the host device boundary Vendor specific CUDA.jl AMDGPU.jl oneAPI.jl Metal.jl \"\"\" md\"\"\" Different layers of abstraction Vendor specific ```julia using CUDA function saxpy a,X,Y i blockIdx .x 1 blockDim .x threadIdx .x if i length Y inbounds Y i a X i Y i end return nothing end cuda threads 32 blocks cld length Y , 32 saxpy a, X, Y ``` KernelAbstractions ```julia using KernelAbstractions using CUDA kernel function saxpy a, Const X , Y I index Global inbounds Y I a X I Y I end saxpy CUDABackend a, X, Y, ndrange length Y ``` Array abstractions ```julia Y . a . X . Y ``` \"\"\" md\"\"\" How to use KernelAbstractions Use ` kernel function mykernel args... end` to write a GPU style program Instantiate kernel for a backend `kernel mykernel backend ` Backends come from Vendor specific libraries `KA.allocate backend, ... ` to obtain memory Launch kernel `kernel args..., ndrange ... ` while specifying the grid to execute over. \"\"\" TwoColumn md\"\"\" ```julia function vadd a, b, c for i in eachindex c c i a i b i end end a rand N b rand N c similar a vadd a, b, c ``` \"\"\", md\"\"\" ```julia import KernelAbstractions as KA kernel function vadd a, b, c i index Global c i a i b i end backend CUDABackend a KA.allocate backend, Float32, N b KA.allocate backend, Float32, N c similar a vadd kernel vadd backend vadd kernel a, b, c ndrange size c ``` \"\"\" md\"\"\" note GPU execution is asynchronous We will discuss the details in the later GPU lecture. When benchmarking you need to synchronize the device ```julia benchmark begin vadd kernel a, b, c ndrange size c KA.synchronize backend end ``` Otherwise you are only measuring the launch of the kernel. \"\"\" md\"\"\" High level array based programming Julia and GPUArrays.jl provide support for an efficient GPU programming environment build around array abstractions and higher order functions. Vocabulary of operations `map`, `broadcast`, `scan`, `reduce`, ... Map naturally onto GPU execution models Compiled to efficient code multiple dispatch, specialization Write generic, reusable applications BLAS matrix multiply, ... , and other libraries like FFT Array operators using multiple dispatch a design methodology for array implementations in dynamic languages doi 10.1145 2627373.2627383 Rapid software prototyping for heterogeneous and distributed platforms doi 10.1016 j.advengsoft.2019.02.002 \"\"\" md\"\"\" Array types where memory resides and how code is executed. | | | | | | | `A Matrix Float64 undef, 64, 32 ` | CPU | | `A CuMatrix Float64 undef, 64, 32 ` | Nvidia GPU | | `A ROCMatrix Float64 undef, 64, 32 ` | AMD GPU | info Data movement is explicit. \"\"\" md\"\"\" What makes an application portable? 1. Can I run it on a different compute architecture 1. Different CPU architectures 2. We live in a mult GPU vendor world 2. Does it compute the same thing? 1. Can I develop on one platform and move to another later? 3. Does it achieve the same performance ? 4. Can I take advantage of platform specific capabilities? Productivity meets Performance Julia on A64FX doi 10.1109 CLUSTER51413.2022.00072 \"\"\" md\"\"\" Adapt.jl Adapt.jl https github.com JuliaGPU Adapt.jl is a lightweight dependency that you can use to convert complex structures from CPU to GPU. ```julia using Adapt adapt CuArray, Adjoint Array Adjoint CuArray ``` ```julia struct Model T Number, AT AbstractArray T data AT end Adapt.adapt structure to, x Model Model adapt to, x.data cpu Model rand 64, 64 using CUDA gpu adapt CuArray, cpu Model Float64, CuArray Float64, 2, CUDA.Mem.DeviceBuffer ... ``` \"\"\" md\"\"\" Multi Node Distributed \"\"\" md\"\"\" Explicit communication between processes using a library like `MPI.jl` \"\"\" md\"\"\" ```julia using MPI MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm size MPI.Comm size comm dst mod rank 1, size src mod rank 1, size N 4 send mesg Array Float64 undef, N recv mesg Array Float64 undef, N fill send mesg, Float64 rank rreq MPI.Irecv recv mesg, comm source src, tag src 32 sreq MPI.Isend send mesg, comm dest dst, tag rank 32 stats MPI.Waitall rreq, sreq MPI.Barrier comm ``` \"\"\" md\"\"\" note Weak scaling is often a more interesting measurement on clusters. \"\"\" TODO \"mention colab.research.google.co\" "},{"url":"mod1_introduction/performance_engineering/","title":"Performance Engineering","tags":["module1","track_performance"],"text":" A Pluto.jl notebook v0.20.13 frontmatter chapter \"1\" section \"3\" order \"3\" title \"Performance Engineering\" date \"2025 04 30\" tags \"module1\", \"track performance\" layout \"layout.jlhtml\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils This Pluto notebook uses bind for interactivity. When running this notebook outside of Pluto, the following 'mock version' of bind gives bound variables a default value instead of an error . macro bind def, element format off return quote local iv try Base.loaded modules Base.PkgId Base.UUID \"6e696c72 6542 2067 7265 42206c756150\" , \"AbstractPlutoDingetjes\" .Bonds.initial value catch b missing end local el esc element global esc def Core.applicable Base.get, el ? Base.get el iv el el end format on end using PlutoUI, PlutoTeachingTools ChooseDisplayMode PlutoUI.TableOfContents depth 4 md\"\"\" Performance Engineering \"\"\" md\"\"\" Goals 1. Understanding the performance characteristics of your program What is the hot path Where is time being spent 2. Define metrics Benchmarks allow us to understand \"is a change good\" 3. How does my program get executed? Language of choice? Compilation Hardware architecture \"\"\" TwoColumn md\"\"\" Benchmarking Focusing on the \"hot loop\" Allows for comparision Different algorithms Different hardware \"\"\", md\"\"\" Profiling Analyse where time is being spent Many tools with different trade offs Different perspectives Language profiler System profiler \"\"\" md\"\"\" Benchmarking \"\"\" md\"\"\" note Premature optimization is the root of all evil & If you don't measure you won't improve \"\"\" md\"\"\" BenchmarkTools.jl Solid package that tries to eliminate common pitfalls in performance measurment. ` benchmark` macro that will repeatedly evaluate your code to gain enough samples Caveat You probably want to escape your input data \"\"\" using BenchmarkTools bind N PlutoUI.Slider 5 12 md\"\"\" N N \"\"\" data rand 10^N function sum X acc 0 for x in X acc x end acc end elapsed sum data Huh Is pluto lying to us belapsed sum data samples 10 evals 3 benchmark sum data samples 10 evals 3 md\"\"\" note We will talk about the Julia compiler in detail in a future lecture. One important thing to know, that Julia performs type inference based on the argument types, and quality of type inference determines performance characteristics. \"\"\" md\"\"\" ```julia function sum X Vector Float64 acc 0 Int64 for x in X acc x Float64 end acc Union Int64, Float64 end ``` \"\"\" with terminal do code warntype sum data end md\"\"\" Caveats of micro benchmarking BenchmarkTools tries to approximate a function execution a top level. It thus measures the cost of accessing global variables. Use the interpolation syntax ` ` to avoid that cost for very cheap functions. \"\"\" begin a 3.0 b 4.0 end benchmark sin a b benchmark sin a b benchmark sin a b md\"\"\" warning Did we get to fast? \"\"\" code typed do sin 3.0 4.0 end | only md\"\"\" Julia can constant fold expressions. \"\"\" benchmark sin Ref a Ref b md\"\"\" warning Be careful with benchmarks whose time doesn't change with an increase in complexity Likely that the compiler got clever and turned it into a constant time expression or constant folded it. \"\"\" function count N acc 0 for i in 1 N acc 1 end return acc end benchmark count 10 benchmark count 1000000 md\"\"\" warning How would you benchmark `sort `? \"\"\" let v rand Int, 1024 benchmark sort v end md\"\"\" `sort ` is mutating its input. Therefore we need to set `evals 1` and provide a `setup` to re initialize the data everytime \"\"\" benchmark sort v setup v rand Int, 1024 evals 1 md\"\"\" Sources of noise Computers are noisy systems. The operating system manages resources and distribute them to programs. 1. Heat The temperature of your processor influences the frequency it is targetting When benchmarking a function may be faster in the beginning and slower afterwards 2. Other programs The OS is splitting the CPU time into slices So a program may be descheduled 3. Input Output IO Disk Network access is variable The OS will also put you to sleep, when waiting for data 4. The CPU CPU are \"learning\" \"predicitive\" systems. See https discourse.julialang.org t psa microbenchmarks remember branch history 17436 \"\"\" md\"\"\" Profiling \"\"\" md\"\"\" Most profilers we will use are stochastic profilers. They sample the running program at a fixed interval. Julia uses a default of `0.001s` and it also uses a fixed size buffer. See `Profile.init`. note Sampling artifacts can occur when profiling multi threaded applications. During a sample the thread we are sampling from is paused. If the thread is in a critical section this may introduce artifacts. \"\"\" Profile.init md\"\"\" Profiler https docs.julialang.org en latest manual profile ProfileView.jl https github.com timholy ProfileView.jl ProfileCanvas.jl https github.com pfitzseb ProfileCanvas.jl PProf.jl https github.com JuliaPerf PProf.jl \"\"\" using Profile using ProfileCanvas md\"\"\" The Julia profiler focuses on the execution of Julia code. This leads to two limitations 1. By default it does not show time spent in C functions 2. It does not measure time spent on \"external\" threads BLAS threads GC worker threads \"\"\" function profile test n for i 1 n A randn 100,100,20 m maximum A Am mapslices sum, A dims 2 B A , ,5 Bsort mapslices sort, B dims 1 b rand 100 C B. b end end profview profile test 1 run once to trigger compilation ignore this one profview profile test 10 md\"\"\" note The profiler shows all Julia worker threads. The time spent in `task done hook` is a worker thread idiling. \"\"\" function pfib n Int if n 1 return n end t Threads. spawn pfib n 2 return pfib n 1 fetch t Int end function profile pfib n, k for i 1 n pfib k end end profview profile pfib 1, 4 profview profile pfib 10, 16 md\"\"\" To gain insight into runtime behavior we can turn on C frames \"\"\" profview profile pfib 10, 16 C true md\"\"\" Runtime functions you might see `jl gc alloc` Memory allocations `jl gc collect` Garbage collection `jl safepoint wait gc` Garbage collection waiting for all threads to reach it. \"\"\" md\"\"\" Native profilers \"\"\" md\"\"\" System profilers allow you to gain even more insight into how your program is performing, but come with usability down sides. \"\"\" md\"\"\" VTune https github.com JuliaPerf IntelITT.jl?tab readme ov file running julia under vtune Perf https docs.julialang.org en v1 manual profile External Profiling NSight Systems https cuda.juliagpu.org stable development profiling External profilers `CUDA.jl` has it's own inbuilt ` profile` \"\"\" md\"\"\" Instrumentation It can be hard to correlate profiles with our programs, instrumentation makes it easier to define semantically important portions. TimerOutputs https github.com KristofferC TimerOutputs.jl NVTX https github.com JuliaGPU NVTX.jl Tracy https github.com topolarity Tracy.jl IntelITT.jl https github.com JuliaPerf IntelITT.j \"\"\" using TimerOutputs using Base.Threads threads const to TimerOutput timeit to function prefix threads ⊕, y AbstractVector l length y k ceil Int, log2 l do reduce phase timeit to \"reduce\" for j 1 k threads for i 2^j 2^j min l, 2^k inbounds y i y i 2^ j 1 ⊕ y i end end do expand phase timeit to \"expand\" for j k 1 1 1 threads for i 3 2^ j 1 2^j min l, 2^k inbounds y i y i 2^ j 1 ⊕ y i end end return y end let TimerOutputs.reset timer to for i in 1 10 A fill 1, 500 000 prefix threads , A end to end md\"\"\" Allocation profiler The allocation profiler will collect backtraces at allocation sites with a default `sample rate 0.0001` So about 1 10 000 allocations . PProf.jl https github.com JuliaPerf PProf.jl also has a callgraph view instead of just the \"icile\" view. \"\"\" profview allocs profile test 10 profview allocs profile test 10 sample rate 1.0 md\"\"\" Where does time go? \"\"\" TwoColumn md\"\"\" Your code Arithmetic operations Special functions Memory accesses Memory layout Type instabilities Bad algorithm choices Lack of parallelism \"\"\", md\"\"\" Runtime Memory allocation Garbage collection finding unused memory Waiting for the OS Network Filesystem ... Concurrency Lock conflicts Function dispatch Compiling code \"\"\" md\"\"\" Performance annotation in Julia https docs.julialang.org en v1 manual performance tips Julia does bounds checking by default `ones 10 11 ` is an error ` inbounds` Turns of bounds checking locally ` fastmath` Turns of strict IEEE749 locally – be very careful this might not do what you want ` simd` and ` simd ivdep` provide stronger gurantuees to encourage LLVM to use SIMD operations \"\"\" function my sum X acc zero eltype X for x in X acc x end return acc end benchmark my sum w setup w rand 2048 function my sum2 X acc zero eltype X simd for x in X acc x end return acc end benchmark my sum2 w setup w rand 2048 md\"\"\" note ` simd` allows for re ordering of reduction operations. \"\"\" using CairoMakie md\"\"\" Example Matrix addition Matrix addition is an interesting case because it has no data re use, so there is no possible temporal locality, but depending on what order you use for the loops and how matrices are stored in memory, you may or may not get spatial locality that takes advantage of cache lines . Here let's implement matrix addition in two different ways. We'll use a pre allocated output array so that our benchmark does not include the time for memory allocation \"\"\" function matadd1 C, A, B size C size A size B || throw DimensionMismatchmatch m,n size A for i 1 m simd for j 1 n inbounds C i,j A i,j B i,j end end return C end matadd1 A, B matadd1 similar A, promote type eltype A , eltype B , A, B function matadd2 C, A, B size C size A size B || throw DimensionMismatch m,n size A for j 1 n simd for i 1 m inbounds C i,j A i,j B i,j end end return C end matadd2 A, B matadd2 similar A, promote type eltype A , eltype B , A, B let A rand 5,6 B rand 5,6 A B ≈ matadd1 A,B ≈ matadd2 A,B end function logspace start, stop, length exp10. range start stop, length end begin Na round. Int, logspace 1, log10 3000 , 60 60 sizes from 10 to 3000 alternatively, use N 10 1000 to see some interesting patterns due to cache associativity etc. t1 Float64 t2 Float64 for n in Na local A zeros n,n local B zeros n,n preallocate output C so that allocation is not included in timing C zeros n,n matadd1 C,A,B add once just to make sure we are in cache if A and B are small push t1, elapsed matadd1 C,A,B push t2, elapsed matadd2 C,A,B println \"finished n n ratio t1 t2 of \", t1 end t2 end end end let fig Figure ax Axis fig 1, 1 , title \"\", xlabel \"matrix size n\", ylabel L\"\\text gigaflops~ \\frac n^2 t \" lines ax, Na, Na.^2 . t1 . 1e 9, label \"by row\" lines ax, Na, Na.^2 . t2 . 1e 9, label \"by column\" axislegend ax, position rt fig end let fig Figure ax Axis fig 1, 1 , title \"Ratio of matrix addition algorithms\", xlabel \"matrix size n\", ylabel \"by row time by column time\" lines ax, Na, t1 . t2 fig end md\"\"\" note The reason for this is that Julia stores matrices with consecutive columns , which is known as column major storage format. \"\"\" let A zeros Int, 10, 3 for i in 1 length A A i i end A end md\"\"\" Memory layout of Objects \"\"\" using About md\"\"\" Julia has two types of struct types. 1. Immutable 2. Mutable Immutable datatypes have fields that can't be mutated and they do not have object identity . Think numbers, and similar objects. Mutable datatypes can be updated in place and they posses object identity. \"\"\" struct MyImmutable a Float64 end mutable struct MyMutable a Float64 end let x MyImmutable 1.0 y MyImmutable 1.0 x y end let x MyMutable 1.0 y MyMutable 1.0 x y end with terminal do about Float64 ℯ end with terminal do about 1.0 0.0im end with terminal do about MyImmutable end with terminal do about MyMutable end begin x1 fill MyImmutable 0.0 , 10 x1 1 MyImmutable 1.0 x1 end begin x2 fill MyMutable 0.0 , 10 x2 1 .a 1.0 x2 end md\"\"\" warning `fill` with a mutable object will lead to aliasing. \"\"\" begin x3 MyMutable 0.0 for in 1 10 x3 1 .a 1.0 x3 end with terminal do about 1.0, 2.0 end with terminal do about 1.0 0.0im, 2.0 1.0im end with terminal do about MyMutable 0.0 , MyMutable 1.0 end md\"\"\" info Mutable objects are stored in fields and arrays as references pointers . Immutable objects may be stored inline . \"\"\" "},{"url":"mod1_introduction/research_software_engineering/","title":"Research Software Engineering","tags":["module1","track_principles"],"text":" A Pluto.jl notebook v0.20.13 frontmatter chapter \"1\" section \"1\" order \"1\" title \"Research Software Engineering\" date \"2025 04 16\" tags \"module1\", \"track principles\" layout \"layout.jlhtml\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils This Pluto notebook uses bind for interactivity. When running this notebook outside of Pluto, the following 'mock version' of bind gives bound variables a default value instead of an error . macro bind def, element format off return quote local iv try Base.loaded modules Base.PkgId Base.UUID \"6e696c72 6542 2067 7265 42206c756150\" , \"AbstractPlutoDingetjes\" .Bonds.initial value catch b missing end local el esc element global esc def Core.applicable Base.get, el ? Base.get el iv el el end format on end html\" button onclick 'present ' Toggle presentation mode button \" using PlutoUI PlutoUI.TableOfContents depth 4 md\"\"\" Who am I? B.Sc in Cognitive Science University of Osnabrueck 2011 2014 Applied ML research Okinawa Institute of Technology, Japan 2015 2017 M.Sc & PhD in Computer Science Massachussetts Institute of Technology, USA 2017 2024 Research Software Engineer PostDoc University of Mainz University of Augsburg Working on GPU computing in Julia since 2014 – First Julia version 0.3 dev I work on the Julia compiler and runtime. My goal is to make HPC and GPU based programming easier and more accessible, I work with science teams to help them maximise their usage of Julia. \"\"\" md\"\"\" What is reasearch software engineering Software development is an essential, integral part of research activity. Research software increasingly supports the acquisition, processing and analysis of empirical data, but also the modeling and the simulation of complex processes. Thus, software has a significant influence on the quality of research results. Resource \"https www.software.ac.uk sites default files images content BetterSoftwareBetterResearchImage.jpg\" Data analysis Classical data science statistics Visualization Data generation Modelling Why does it matter Code is science Writing programms is akin to writing paper a way of scientific communication Academic recognition of this is still a challenge Science is built on \"trust but verify\" Reproduction of code is a challenge Too often \"works only on one system\" Building scientific software together Leads to building communities Reuse \"\"\" md\"\"\" Community https researchsoftware.org https society rse.org https de rse.org https fg rse.gi.de Conferences RSECon25 https rsecon25.society rse.org https de rse.org de events \"\"\" md\"\"\" Different kind of projects Small 1 user developer 1 developer developer user Most likely what you will build during this class or for your thesis Reproducibility for papers our future selves Medium Small number of developers Equal number of users Slightly more users than developers Most research projects Large Small number of developers Many users Multiple organization invested Hopefully self sustaining note In the end we have a choice between letting a small project grow if it is useful to not just us , or Maintainership Bus factor Crudly How many people could be hit by a bus, before the project becomes umaintainble Often a one to many relationship It can be very easy to become overwhelmed Many things vying for attention Slack Github Issues Discourse Going from small to medium Giving up some measure of control Communities User community Developer community Open Source Necessary for Open Science All to often people are afraid of releasing their code It's not ready yet1 It's ugly I don't want someone to scoop me. Often Release the source code that was used to produce results in a paper But It is worthwhile to think about converting something from a \"project\" to \"a package\" Encourage reuse Separate concerns Then release a project that uses a package to do something The role of Git Github Git is a version control system Keeps track of previous state of the project Don't email tarballs \"Branches\" can be used to keep track of concurrent developments GitHub is a collaboration platform Supports \"pull request\" Keeping track of \"issues\" Software releases Continous integration Documentation hosting Very much building a community \"\"\" md\"\"\" What are you interested in? \"\"\" md\"\"\" Topics of this Course \"\"\" md\"\"\" Principles of research software engineering Best practices of research software engineering, how to develop scientific software in an reproducible, collaborative way. Sub topics 1. Reproducibility 2. Testing 3. Documentation 4. Accessibility note The goal is to enable you , to build the software you need for your research. \"\"\" md\"\"\" Parallelism Computers are becoming evermore parallel. Multi core Multi node Cluster Accelerated computing GPUs ML accelerators TPU Scientific software often has the characteristics that it can scale. 1. Strong scaling \\ More compute resources will enable a faster time to solution constant problem size 2. Weak scaling \\ Increasing the problem size and compute resources, provides a better solution in the same time note Writing parallel programs takes more consideration than writing a serial problem, but we must to solve problems efficiently Efficiency Time efficiency Power efficiency Cost efficiency \"\"\" md\"\"\" Performance Engineering \"\"\" md\"\"\" Automatic differentiation Why do we want gradients derivatives Derivatives compute the rate of change of a function’s output with respect to input s ```math f' x \\lim h\\rightarrow 0 \\frac f a h f a h ``` Used in many places Optimization problems Parameter estimation Uncertainity quantification Machine learning note Automatic differentiation is the art of taking derivatives not of mathematical functions, but of computer programs that implement mathematical functions. It is a corner stone of both scientfici computing and machine learning. \"\"\" md\"\"\" What's Julia? 🟢 🟣 🔴 Julia is a modern, dynamic, general purpose, compiled programming language. It's interactive \"like Python\" , can be used in a REPL or notebooks, like Jupyter it's the \"Ju\" or Pluto this one🎈 . Julia has a runtime which includes a just in time JIT compiler and a garbage collector GC , for automatic memory management. Julia is mainly used for technical computing, and addresses a gap in the programming language landscape for numerical computing. Main paradigm of Julia is multiple dispatch, what functions do depend on type and number of all arguments. \"\"\" md\"\"\" Why Julia? 😍 From \" My Target Audience https scientificcoder.com my target audience \" by Matthijs Cox Resource \"https cdn.hashnode.com res hashnode image upload v1681735971356 91b6e886 7ce1 41a3 9d9f 29b7b096e7f2.png\" Resource \"https cdn.hashnode.com res hashnode image upload v1681735992315 62fdd58f 4630 4120 8eb4 5238740543e8.png\" Explorable & Understandable Composability thanks to multiple dispatch ask me more about this at the end User defined types are as fast and compact as built ins Code that is close to the mathematics No need to switch languages for performance... ...but you can still call C like shared libraries with simple Foreign Function Interface FFI if you want to MIT licensed free and open source \"\"\" md\"\"\" What is the 2 language problem? You start out proto typing in one language high level, dynamic , but performance forces you to switch to a different one low level, static . For convinience use a scripting language Python, R, Matlab, ... but do all the hard stuff in a systems language C, C , Fortran Pragmatic for many applications, but has drawbacks aren't the hard parts exactly where you need an easier language creates a social barrier a wall between users and developers \"sandwich problem\" layering of system & user code is expensive prohibits full stack optimisations Julia for RSEs? Tearing down barriers of collaboration Fostering collaboration Low barrier from package user to package developer One codebase to rule them all Understandable and explorable performance Julia now Recently released v1.9, comming soon v1.10 Focus on solving latency and infrastructure issues Stable language foundation Vibrant package ecosystem Yearly developer conference, all talks and workshops on Youtube. Excellent native GPU computing support NVIDIA AMD Intel Apple Experimental support for accelerators like Graphcore IPU. \"\"\" md\"\"\" Getting started with Julia info Modern Julia Workflows https modernjuliaworkflows.org is an excellent resource to get started with. Installation In order of preference 1. Use `juliaup` ```shell curl fsSL https install.julialang.org | sh ``` 2. Use the binaries from from https julialang.org downloads 3. ... 4. Use whatever version of Julia is on your cluster 5. Don't use `julia` from your package manager Resources https modernjuliaworkflows.org https discourse.julialang.org https docs.julialang.org https julialang.org community events \"\"\" md\"\"\" A first Julia function \"\"\" function mandel z c z maxiter 80 for n 1 maxiter if abs z 2 return n 1 end z z^2 c end return maxiter end function mandel String \"Hello world\" end mandel \"\" function f a Int, b return \"first method\" end function f a, b Int return \"second method\" end f 1, \"\" f \"\", 1 f 1, 1 function f Int64, Int64 \"third method\" end mandel complex .3, .6 dump 0.3 0.6im π md\"\"\" Real component bind x real PlutoUI.Slider 2 0.01 1 Imaginary component bind x img PlutoUI.Slider 1 0.01 1 \"\"\" mandel complex x real, x img using CairoMakie md\"\"\" Resolution bind resolution PlutoUI.Slider 0.1, 0.01, 0.001 , show value true Real shift bind real shift PlutoUI.Slider 3 resolution 3, show value true, default 0 Img shift bind img shift PlutoUI.Slider 3 resolution 3, show value true, default 0 Zoom bind zoom PlutoUI.Slider 1 100, show value true, default 1 \"\"\" evaluate the mandelbrot set over a complex plane begin reals 2 resolution 1 . real shift . zoom imgs 1 resolution 1 . img shift . zoom plane complex re, img for re, img in Iterators.product reals, imgs end heatmap mandel. plane "},{"url":"mod2_principles_of_rse/debugging/","title":"Debugging","tags":["module2","track_principles"],"text":" A Pluto.jl notebook v0.20.13 frontmatter chapter \"2\" section \"3\" order \"7\" title \"Debugging\" date \"2025 05 28\" tags \"module2\", \"track principles\" layout \"layout.jlhtml\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils begin using PlutoUI, PlutoTeachingTools using PlutoUI Slider PlutoUI.TableOfContents depth 4 end begin using CairoMakie set theme theme latexfonts fontsize 16, Lines linewidth 2, , markersize 16 end ChooseDisplayMode using Asciicast using HypertextLiteral md\"\"\" Debugging \"\"\" md\"\"\" Strategies Ideas \"Scientific method\" Form a hypothesis Aim to invalidate it One change at a time Reduce, reduce, reduce The smaller the section of code, the easier it is to have a mental model of what is happeneing Create a git repository \\ Goal Keep track of changes we have so that we don't lose a bug. Is it \"randomly\" happening? What do we do after we fixed a bug? Turn our MWE into a test The bug likely happened due to some confusion of assumptions Does this pattern repeat elsewhere? \"\"\" md\"\"\" The importance of an MWE Remember last weeks lecture on reproducibility You might need to ask a friend or stranger for help What version of Julia are you version Which packages Project.toml Manifest.toml note Make it as easy as possible for another person to reproduce your setup \"\"\" md\"\"\" Common causes Noteworthy Differences https docs.julialang.org en v1 manual noteworthy differences Undefined memory Julia allows for memory to be unitialized ```julia x Array Float64 undef, 10 or similar x ``` \"\"\" sum Array Float64 undef, 10 let x ones 10 similar x end md\"\"\" this can \"poison\" your calculations \"\"\" md\"\"\" Aliasing of memory and variable scope ```julia a 0, 1, 0 b a b 2 0 ??? a ``` ```julia function f a a 3 end a 2 f a ?a ``` \"\"\" md\"\"\" \"Printf\" debugging When we write code we should check our assumptions and raise appropriate errors \"\"\" 1÷0 1 0 md\"\"\" You can write your own exception types and use `throw` to raise them. \"\"\" function safe div x, y if y zero y throw DivideError end x y end safe div 1, 0 md\"\"\" Or you can use `error` to raise a generic `ErrorException`. \"\"\" error \"Should have thought about this\" md\"\"\" Or ` assert` to check conditions. \"\"\" assert 1 0 md\"\"\" Revise Revise https github.com timholy Revise.jl is a core component of my development workflow. It watches source files and packages for changes and applies them to my running Julia session note If you use `VSCode` it is automatically used when you use the inbuilt Julia REPL. note If the script you are working on is split across multiple files, you need to use `includet` for Revise to track included files. \"\"\" md\"\"\" Julia's logging infrastructure Julia has a rich logging infrastructure https docs.julialang.org en v1 stdlib Logging man logging with different levels ` debug` Hidden by default, can be turned on by setting the environment variable `JULIA DEBUG \"Package\"` ` info` General information, great for passing structured information to the user. ` warn` Something is wrong, but the program is likely continuing ` error` Something is really wrong note ` error` does not imply `error ` it in itself is not an exception. You can use the Logging infrastructure to redirect and manipulate structured log messages. \"\"\" info \"Wow so useful \" lecture true md\"\"\" You can used the reserved field `exception` to recover from errors and pass along the backtrace \"\"\" try error \"Ye who enters, abandons all hope \" catch err info \"Caught error\" exception err end try error \"Ye who enters, abandons all hope \" catch err bt catch backtrace info \"Caught error\" exception err, bt end md\"\"\" This can be useful if you want to figure out in which context a log message was fired \"\"\" function fib x if x 0 || x 1 info \"Basecase\" exception ErrorException \"\" , backtrace return x end fib x 1 fib x 2 end fib 1 fib 3 md\"\"\" Callstacks and backtraces Profilers and debuggers think in backtraces When we execute a program, each function needs a little bit of space on the stack. note Heap and Stack are regions in memory, the `heap` contains all allocated data and the stack contains the local variables & co of the currently executing functions. given the execution of `fib 3 ` ``` fib 3 fib 2 fib 1 fib 0 fib 1 ``` Each function may execute more functions, but we only execute one function at a time. So we visit the call graph in order and thus linearize the execution. ``` fib 3 ``` ``` fib 3 fib 2 ``` ``` fib 3 fib 2 fib 1 ``` ``` fib 3 fib 2 fib 0 ``` ``` fib 3 fib 1 ``` \"\"\" md\"\"\" When we collect a profile, or receive an exception, or are in a debugger at a breakpoint, we almost always will collect the current backtrace stacktrace. Letting us know \"how\" did we get here. Information that is not in a backtrace is as an example loops \"\"\" md\"\"\" Tools \"\"\" md\"\"\" Debugger.jl Julia's native debugger. Built upon JuliaInterpreter.jl https github.com JuliaDebug JuliaInterpreter.jl Pro's Close to \"your code\" Con's Only as fast as JuliaInterpreter and thus can struggle with Example ```julia using Debugger function foo n x n 1 BigInt 1 1 1 0 ^x 2,1 end enter foo 20 ``` \"\"\" asciicinema LocalResource \"recordings debugger.cast\" .src speed 2.0 md\"\"\" Infiltrator.jl Since Debugger.jl needs to interpret all code to ensure breakpoints are triggered it can be slow on a very large code to reach the region of interest ```julia function f x out for i in x push out, 2i infiltrate end out end ``` Tricks Combine with Revise.jl and use `Main. infiltrate` to work on package code \"\"\" asciicinema LocalResource \"recordings infiltrator.cast\" .src speed 2.0 md\"\"\" VS Code Debugger.jl is integrated into VS Code https www.julia vscode.org docs dev userguide debugging it has similar caveats as before, but you may prefer it over the pure text interface. \"\"\" md\"\"\" GDB LLDB GDB and LLDB are native debugger, they are closest to the true execution of Julia. They are very powerful but they require are deeper understanding on how programs work. Pro's Faster than `Debugger.jl` since it debug's native code Faithful Debugging the code doesn't change it's behavior Con's Low level view Doesn't know much about Julia code Tricks `ENABLE GDBLISTENER 1 julia g2` and potentially a debug build of Julia ` ccall jl breakpoint val Any Cvoid` `jl ... ` Breakpoints Watchpoitns Execute commands on breakpoint https docs.julialang.org en v1 devdocs debuggingtips ```julia function example r Ref 0 ccall jl breakpoint r Any Cvoid for i in 1 10 r i end return r end ``` \"\"\" asciicinema LocalResource \"recordings gdb.cast\" .src speed 1.5 md\"\"\" ``` ENABLE GDBLISTENER 1 gdb args julia g2 gdb break jl breakpoint gdb run julia function example r Ref 0 ccall jl breakpoint r Any Cvoid for i in 1 10 r i end return r end example generic function with 1 method julia example Thread 1 \"julia\" hit Breakpoint 1, jl breakpoint v 0x7fffeec5c6a0 at cache build tester amdci5 12 julialang julia release 1 dot 11 src rtutils.c 1475 warning 1475 cache build tester amdci5 12 julialang julia release 1 dot 11 src rtutils.c No such file or directory gdb p jl v Base.RefValue Int64 x 0 gdb bt 3 gdb p int64 t v gdb p x v 3 0x7fffeec5c6a0 gdb watch int64 t 0x7fffeec5c6a0 Hardware watchpoint 2 int64 t 0x7fffeec5c6a0 gdb continue Thread 1 \"julia\" hit Hardware watchpoint 2 int64 t 0x7fffeec5c6a0 Old value 0 New value 55 0x00007fffebcfc518 in julia example 43 at REPL 1 5 ``` \"\"\" md\"\"\" rr note A key challenge when debugging is that we often ask the question \"how did we get here\", but debuggers are really good at answering the question \"where are we going from here\". RR https rr project.org is a time traveling debugger. Using `rr record` we record the execution of a program, and it's children. Using `rr replay` we can re execute the program faith fully. We can then use `gdb` to debug a program being replayed and do things like `reverse continue`, e.g. execute a program backwards in time to answer questions like When did this value in memory get set? Caveat Linux only Works most reliably on Intel CPU boooh Doesn't work with GPU codes. \"\"\" md\"\"\" Cthulhu.jl Cthulhu.jl https github.com JuliaDebug Cthulhu.jl is an unusal debugger in the sense that it doesn't debug values, but it debugs types, and thus what the Julia compiler was able to infer about your program. It takes a static walk though dynamic programs , it works on multiple level of representation, but at it's core it operates on typed intermediate representation . \"\"\" md\"\"\" Motivation Have you ever done ` code typed` Search for a `call g, .... Any` `code typed g, .... ` repeat? \"\"\" md\"\"\" Examples ```julia import NamedDims g NamedDims.unify names a, b , a, ``` ```julia h A,B A . B . A h zeros 3 , zeros 3,3 ``` ```julia struct Interval Number a Float64 b Float64 end Base.eltype Interval Float64 contains I Interval, x Float64 I.a x I.b f I Interval, xs Vector Float64 contains. I, xs f Interval 0.5, 0.6 , rand 4 ``` \"\"\" begin struct Interval Number a Float64 b Float64 end Base.eltype Interval Float64 contains I Interval, x Float64 I.a x I.b f I Interval, xs Vector Float64 contains. I, xs end f Interval 0.5, 0.6 , rand 4 with terminal do code warntype f Interval 0.5, 0.6 , rand 4 end begin function asciicinema src speed 1.0 htl \"\"\" link rel \"stylesheet\" type \"text css\" href \"https cdn.jsdelivr.net npm asciinema player 3.9.0 dist bundle asciinema player.css\" script src \"https cdn.jsdelivr.net npm asciinema player 3.9.0 dist bundle asciinema player.min.js\" script div style \"margin 20px\" script const div document.createElement \"div\" AsciinemaPlayer.create src, div, fit \"width\", cols 80, rows 24, speed speed, preload true, return div script div \"\"\" end function asciicinema cast Asciicast.Cast kwargs... str Asciicast.base64encode Asciicast.collect bytes cast asciicinema \"data text plain base64,\" str kwargs... end end "},{"url":"mod2_principles_of_rse/github/","title":"Software development with Github","tags":["module2","track_principles"],"text":" A Pluto.jl notebook v0.20.13 frontmatter chapter \"2\" section \"1\" order \"5\" title \"Software development with Github\" date \"2025 05 14\" tags \"module2\", \"track principles\" layout \"layout.jlhtml\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils html\" button onclick 'present ' Toggle presentation mode button \" using PlutoUI PlutoUI.TableOfContents depth 4 md\"\"\" Git & Github \"\"\" using Kroki md\"\"\" What is Git? Git https git scm.com is a d istributed v ersion c ontrol system. 1. Version Control System Keeps track of \"state\" of a repository. Keeps a history of previous changes 2. Distributed Local first Many different \"views\" on the state of the project Explicit synchronization \"\"\" md\"\"\" note The GitBook https git scm.com book en v2 is a usefull resource. \"\"\" md\"\"\" Git concepts Repository A top level folder tracked by git. Checkout The current local state of the working directory. Commit A set of changes. Hash SHA The hash of a commit includes the history parents . Tag A named commit. Branch A named \"state\" currently pointing at a commit. \"\"\" md\"\"\" Git commands \"\"\" md\"\"\" `git init` ```sh mkdir MyPackage.jl cd MyPackage.jl git init Initialized empty Git repository in tmp MyPackage.jl .git ``` Creates the `.git` folder and initializes a git repository. note `git` will walk up the directory tree until it finds a `.git` folder. \"\"\" mermaid\"\"\" gitGraph \"\"\" md\"\"\" `git status` ```sh git status On branch main No commits yet nothing to commit create copy files and use \"git add\" to track ``` \"\"\" md\"\"\" `git add` \"\"\" md\"\"\" ```sh echo \" MyPackage\" README.md git status On branch main No commits yet Untracked files use \"git add file ...\" to include in what will be committed README.md nothing added to commit but untracked files present use \"git add\" to track ``` \"\"\" md\"\"\" ```sh git add README.md git status On branch main No commits yet Changes to be committed use \"git rm cached file ...\" to unstage new file README.md ``` \"\"\" md\"\"\" Git has three simultaneous \"states\" The state of the current commit The state of the working directory untracked The \"staged\" state \"Changes to be committed\" \"\"\" md\"\"\" note `git add p` is an extremely useful variant that let's you see what changes you are staging. \"\"\" md\"\"\" `git commit` Git commit creates a new commit. ```sh git commit m \"initial commit\" main root commit d3af9e2 initial commit 1 file changed, 1 insertion create mode 100644 README.md ``` ```sh git status On branch main nothing to commit, working tree clean ``` \"\"\" mermaid\"\"\" gitGraph commit \"\"\" md\"\"\" `git log` ```sh git log commit d3af9e273c4b1d42fca6093f63a9560a3b2cf767 HEAD main Author Valentin Churavy ... Date Tue May 13 18 28 24 2025 0200 initial commit ``` \"\"\" md\"\"\" note If you need to remove a file from Git, you can use the `git rm` command. Note, that this only removes the file from the current commit, but it's previous content is always available in the history. \"\"\" md\"\"\" Working with branches \"\"\" md\"\"\" `git branch` ```sh git branch main ``` ```sh git branch v main 504d898 initial commit ``` \"\"\" md\"\"\" You can use `git branch` to create a new branch ```sh git branch develop ``` ```sh git branch v develop 504d898 initial commit main 504d898 initial commit ``` \"\"\" md\"\"\" Deleting a branch. ```sh git branch D develop ``` \"\"\" mermaid\"\"\" gitGraph commit \"\"\" md\"\"\" `git switch` `git switch` switches branches ```sh git switch develop ``` note You will sometimes see me use `git checkout` instead, which is an older form. Learn more https refine.dev blog git switch and git checkout using git switch vs git checkout \"\"\" md\"\"\" With ` c` git switch creates a new branch. ```sh git switch c feature ``` \"\"\" md\"\"\" `git merge` Given a state that looks like this, we may want to add changes from `feature` back to `main`. \"\"\" mermaid\"\"\" gitGraph commit branch feature commit commit \"\"\" md\"\"\" ```sh git switch main git merge feature ``` \"\"\" mermaid\"\"\" gitGraph commit branch feature commit commit checkout main merge feature \"\"\" md\"\"\" Due to the nature of Git, work may have happen on `main` \"\"\" mermaid\"\"\" gitGraph commit branch feature commit commit checkout main commit \"\"\" md\"\"\" ```sh git switch feature git merge main ``` \"\"\" mermaid\"\"\" gitGraph commit branch feature commit commit checkout main commit checkout feature merge main \"\"\" md\"\"\" `git rebase` Instead of merging a branch we may want to `rebase`. Rebase is particularly useful when working with feature branches and it's variant `git rebase i` allows you to cleanup your messy state. \"\"\" mermaid\"\"\" gitGraph commit id \"A\" branch feature commit id \"B\" commit id \"C\" checkout main commit id \"D\" \"\"\" md\"\"\" ```sh git switch feature git rebase main ``` \"\"\" mermaid\"\"\" gitGraph commit id \"A\" commit id \"D\" branch feature commit id \"B\" commit id \"C\" \"\"\" md\"\"\" `git stash` A more advanced utility command is `git stash`. It \"saves\" the current state of your working directory \"\"\" md\"\"\" ```sh echo \"Hey\" README.md git status On branch main Changes not staged for commit use \"git add file ...\" to update what will be committed use \"git restore file ...\" to discard changes in working directory modified README.md no changes added to commit use \"git add\" and or \"git commit a\" ``` \"\"\" md\"\"\" ```sh git diff diff git a README.md b README.md index f266118..a72832b 100644 a README.md b README.md 1 1 MyPackage Hey ``` \"\"\" md\"\"\" ```sh git stash Saved working directory and index state WIP on main 504d898 initial commit git status On branch main nothing to commit, working tree clean ``` \"\"\" md\"\"\" ```sh git stash list stash 0 WIP on main 504d898 initial commit git stash pop On branch main Changes not staged for commit use \"git add file ...\" to update what will be committed use \"git restore file ...\" to discard changes in working directory modified README.md no changes added to commit use \"git add\" and or \"git commit a\" Dropped refs stash 0 a6287e30602f575adba45fa10ee48ee48e0326fc git stash list ``` \"\"\" md\"\"\" note `git stash` only works for current dirty files. Use `git stash s` for all files. \"\"\" md\"\"\" Working with remotes A remote is a Git repository \"somewhere\" else. Most often this means it's hosted on Github or another Git forge like Gitlab. \"\"\" md\"\"\" The core commands are `git fetch` & `git pull` to synchronize the state of a repository `git push` to publish the state of our local repository \"\"\" md\"\"\" Checking with `git remote` we can see that we currently don't have a remote setup \"\"\" md\"\"\" ```sh git remote ``` \"\"\" md\"\"\" Github \"\"\" md\"\"\" Going to https github.com new https github.com new allows use setup a new repository. Doing that without ticking any of the boxes, presents us with two options. \"\"\" md\"\"\" …or create a new repository on the command line ``` echo \" MyPackage.jl\" README.md git init git add README.md git commit m \"first commit\" git branch M main git remote add origin https github.com vchuravy MyPackage.jl.git git push u origin main ``` …or push an existing repository from the command line ``` git remote add origin https github.com vchuravy MyPackage.jl.git git branch M main git push u origin main ``` \"\"\" md\"\"\" Since we already have a local repository setup we choose the second option. note To authenticate to Github there are two different ways. 1. Passwords 2. SSH Keys \"\"\" md\"\"\" note The Github CLI https cli.github.com simplifies some of these operations. \"\"\" md\"\"\" ```sh git remote add origin git github.com vchuravy MyPackage.jl.git git branch M main git push u origin main Enumerating objects 3, done. Counting objects 100% 3 3 , done. Writing objects 100% 3 3 , 229 bytes | 229.00 KiB s, done. Total 3 delta 0 , reused 0 delta 0 , pack reused 0 from 0 To github.com vchuravy MyPackage.jl.git new branch main main branch 'main' set up to track 'origin main'. ``` \"\"\" md\"\"\" ```sh git remote v origin git github.com vchuravy MyPackage.jl.git fetch origin git github.com vchuravy MyPackage.jl.git push ``` \"\"\" md\"\"\" What to do on a merge conflict ? When working with other people each local repository may diverge from the common root. So sometimes you may run into merge conflicts. \"\"\" md\"\"\" When a merge conflict occurs you will see something like ``` HEAD this is some content to mess with content to append totally different content to merge later new branch to merge later ``` You need to choose which side of the merge you want, and maybe you need to combine them I recommend VS Code for this. \"\"\" md\"\"\" Julia & Github \"\"\" md\"\"\" note Below we will walk through the manual setup of a Julia reposirtory, but you may want to use `PkgTemplates.jl` https github.com JuliaCI PkgTemplates.jl to simplify the initial setup. \"\"\" md\"\"\" Structure of a Julia Github repository ```shell .github workflows src MyPackage.jl test Project.toml runtests.jl docs src Project.toml make.jl benchmark Project.toml benchmarks.jl Project.toml README.md LICENSE.md .gitignore ``` \"\"\" md\"\"\" Licensing There are many licenses to choose from. In short a license protects you and others and allows you to use other people code without being sued later... Chose A License https choosealicense.com The Julia community at large prefers the MIT license https choosealicense.com licenses mit . When contributing to someone elses project check the license The most important thing is that it is something like an OSI Approved License https opensource.org licenses and not something made up. \"\"\" md\"\"\" `Project.toml` ```toml name \"MyPackage\" uuid \"XXXXXXXX XXXX XXXX XXXX XXXXXXXXXXXX\" authors \"Author autor somehost.de \" version \"0.1.0\" deps Enzyme \"7da242da 08ed 463a 9acd ee780be4f1d9\" compat Enzyme \"0.13.41\" ``` \"\"\" md\"\"\" ``` generate MyPackage ``` \"\"\" md\"\"\" Testing `test ` A foundational best practice for research software development is testing. Testing can both be 1. Unit tests Small scale test of functionality 2. End to end tests Testing that a task functions Julia has the `Test.jl` https docs.julialang.org en v1 stdlib Test package that helps to write unit tests. Other helpful package TestItems.jl https www.julia vscode.org docs stable userguide testitems `runtests.jl` ```julia using Test using MyPackage testset \"Group of Tests\" begin test my fun 1 test broken my other fun 2 end ``` `Project.toml` ```toml deps MyPackage \"...\" Test \"8dfed614 e22c 5e08 85e1 65c5234f0b40\" sources MyPackage path \"..\" ``` \"\"\" md\"\"\" Documentation The package `Documenter.jl` https documenter.juliadocs.org turns doc strings and manual written docs into an webpage. Doc strings \"\"\" \"\"\" toss coin p 0.5 Bool Toss a coin to your witcher. Arguments `p` The propability that the coin comes up head. \"\"\" function toss coin p 0.5 rand p end md\"\"\" note The Documenter Guide https documenter.juliadocs.org stable man guide has a full setup. \"\"\" md\"\"\" Benchmarking Using `BenchmarkTools.jl` https github.com JuliaCI BenchmarkTools.jl we can write and define benchmark suites. Using `PkgBenchmark.jl` https github.com JuliaCI PkgBenchmark.jl or `AirSpeedVelocity.jl` https github.com MilesCranmer AirspeedVelocity.jl we can run those benchmarks. For CI `AirSpeedVelocity.jl` https github.com MilesCranmer AirspeedVelocity.jl or `github action benchmark` https github.com benchmark action github action benchmark may be options. Send me feedback \"\"\" md\"\"\" ``` benchmark Project.toml benchmarks.jl runbenchmarks.jl ``` \"\"\" md\"\"\" Project.toml ```toml deps BenchmarkTools \"6e4b80f9 dd63 53aa 95a3 0cdb28fa8baf\" MyPackage \"...\" sources MyPackage path \"..\" ``` \"\"\" md\"\"\" `benchmarks.jl` ```julia using BenchmarkTools using TowerOfEnzyme const SUITE BenchmarkGroup SUITE \"basics\" BenchmarkGroup SUITE \"basics\" \"overhead\" benchmarkable nth derivative sin, 1.0, Val 0 ``` \"\"\" md\"\"\" `runbenchmarks.jl` ```julia For CI using BenchmarkTools include \"benchmarks.jl\" tune SUITE results run SUITE, verbose true BenchmarkTools.save \"output.json\", median results ``` \"\"\" md\"\"\" Continous integration Matrix Triggers Secrets \"\"\" md\"\"\" Example `.github workflows CI.yml` ```yml name Run tests on push branches main pull request needed to allow julia actions cache to delete old caches that it has created permissions actions write contents read jobs test runs on matrix.os strategy matrix julia version 'lts', '1', 'pre' julia arch x64 os ubuntu latest, windows latest, macOS latest steps uses actions checkout v4 uses julia actions setup julia v2 with version matrix.julia version arch matrix.julia arch uses julia actions cache v2 id julia cache uses julia actions julia buildpkg v1 uses julia actions julia runtest v1 name Save Julia depot cache on cancel or failure id julia cache save if cancelled || failure uses actions cache save v4 with path | steps.julia cache.outputs.cache paths key steps.julia cache.outputs.cache key ``` \"\"\" md\"\"\" Example `.github workflows Documenter.yml` \"\"\" md\"\"\" note Read the manual on setting up Documenter with `GitHub Actions` https documenter.juliadocs.org stable man hosting GitHub Actions in particular you will need to setup a secret. \"\"\" md\"\"\" ```yml name Documentation on push branches main tags ' ' pull request jobs build These permissions are needed to Deploy the documentation https documenter.juliadocs.org stable man hosting Permissions Delete old caches https github.com julia actions cache usage permissions actions write contents write pull requests read statuses write runs on ubuntu latest steps uses actions checkout v4 uses julia actions setup julia v2 with version '1' uses julia actions cache v2 name Install dependencies shell julia color yes project docs 0 run | using Pkg Pkg.develop PackageSpec path pwd Pkg.instantiate name Build and deploy run julia color yes project docs docs make.jl env GITHUB TOKEN secrets.GITHUB TOKEN If authenticating with GitHub Actions token DOCUMENTER KEY secrets.DOCUMENTER KEY If authenticating with SSH deploy key ``` \"\"\" md\"\"\" Example `.github workflows TagBot.yml` https github.com JuliaRegistries TagBot ```yml name TagBot on issue comment types created workflow dispatch jobs TagBot if github.event name 'workflow dispatch' || github.actor 'JuliaTagBot' runs on ubuntu latest steps uses JuliaRegistries TagBot v1 with token secrets.GITHUB TOKEN ssh secrets.DOCUMENTER KEY ``` \"\"\" "},{"url":"mod2_principles_of_rse/reproducibility/","title":"Reproducibility","tags":["module2","track_principles"],"text":" A Pluto.jl notebook v0.20.13 frontmatter chapter \"2\" section \"2\" order \"6\" title \"Reproducibility\" date \"2025 05 21\" tags \"module2\", \"track principles\" layout \"layout.jlhtml\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils begin using PlutoUI, PlutoTeachingTools using PlutoUI Slider PlutoUI.TableOfContents depth 4 end begin using CairoMakie set theme theme latexfonts fontsize 16, Lines linewidth 2, , markersize 16 end ChooseDisplayMode md\"\"\" Reproducibility Two questions 1. Does my code execute on another computer Can someone else do it without my intervention 2. Do we calculate the same results? \"\"\" md\"\"\" Should be easy right? ```sh git clone https github.com vchuravy FancyProject cd FancyProject julia run script.jl ``` What can go wrong? What version of Julia did I use? What Julia packages dig I use? Which version of those Julia packages did I use? Which version of all the transitive dependencies did I use? \"\"\" md\"\"\" How does Julia know which package can be loaded. `LOAD PATH` https docs.julialang.org en v1 base constants Base.LOAD PATH describes the stacked environment. When doing an `import` `using` called a require operation we search the stack for a matching package. In the REPL ```julia repl julia Base.LOAD PATH 3 element Vector String \" \" \" v . \" \" stdlib\" ``` References Code loading https docs.julialang.org en v1 manual code loading code loading \"\"\" Base.LOAD PATH Base.load path doc Base.LOAD PATH md\"\"\" Active project The ` ` entry in the `LOAD PATH` corresponds to the active project ```sh julia project . ``` or ```sh JULIA PROJECT \".\" julia ``` You can query the current active project with `Base.active project ` and switch the active project with `Base.set active project ` warning When you switch projects inside a Julia session, you are combining \"partial state\". You might have loaded incompatible package versions note In many older docs you will see `Pkg.activate` instead of `Base.set active project` which was only added in Julia 1.8. \"\"\" Base.active project md\"\"\" Julia's package manager \"\"\" md\"\"\" note In Pluto the package manager is tightly integrated. This leads to reproducible note books. Open a Pluto notebook in an editor What do you see? \"\"\" md\"\"\" Project.toml A `Project.toml` describes the dependenencies of a project or a package ```toml name \"App\" uuid \"8f986787 14fe 4607 ba5d fbff2944afa9\" deps BenchmarkTools \"6e4b80f9 dd63 53aa 95a3 0cdb28fa8baf\" compat BenchmarkTools \"1.6\" julia \"1.10\" ``` \"\"\" md\"\"\" A package is a project with a name and a UUID. ```toml name \"App\" uuid \"8f986787 14fe 4607 ba5d fbff2944afa9\" ``` \"\"\" md\"\"\" The ` deps ` section maps names to `UUID` ```toml deps BenchmarkTools \"6e4b80f9 dd63 53aa 95a3 0cdb28fa8baf\" ``` \"\"\" md\"\"\" The ` compat ` section restricts the version of dependencies ```toml compat BenchmarkTools \"1.6\" julia \"1.10\" ``` note There is a special entry for Julia. \"\"\" md\"\"\" Manifest.toml The Manifest is an auto generated record of all the transitive dependencies. note Exact reproducibility is only gurantueed when using the same Julia version and the same `Manifest.toml` ```toml This file is machine generated editing it directly is not advised julia version \"1.11.5\" manifest format \"2.0\" project hash \"cb14032aabe4b0376e6dcb58a497b54adc5b92d1\" deps.Artifacts uuid \"56f22d72 fd6d 98f1 02f0 08ddc0907c33\" version \"1.11.0\" ... ``` \"\"\" md\"\"\" note The `Manifest.toml` is version dependent Note the recorded Julia version. Since Julia 1.11, we have access the Julia version specific manifests. `Manifest.toml` Fallback and only option for 1.10 `Manifest v1.11.toml` Will be preferentiably read by 1.11 and ignored by all else. \"\"\" md\"\"\" Pkg.jl \"\"\" md\"\"\" Two modes of interaction 1. In the repl use ` ` to switch 2. `import Pkg Pkg.add ... ` Read Getting started with Pkg.jl https pkgdocs.julialang.org v1 getting started Managing packages https pkgdocs.julialang.org v1 managing packages \"\"\" md\"\"\" Compatibility ```toml compat BenchmarkTools \"1.6\" julia \"1.10\" ``` Julia uses semantic versioning https semver.org `vX.Y.Z`. `X` is the major version, `Y` is the minor version, `Z` is the patch version, with an exception for leading zeros. 1. MAJOR version when you make incompatible API changes 2. MINOR version when you add functionality in a backward compatible manner 3. PATCH version when you make backward compatible bug fixes When you write a version specifier `1.2.3` it is assumed to be compatible with ` 1.2.3 2.0.0 ` where ` ` is a non inclusive upper bound. Julia has two specifiers 1. Caret specifier `^1.2.3` the default and equivalent to `1.2.3` 2. Tilde specifier `~1.2.3` Julia also supports sets of multiple version specifiers ``` compat Example \"1.2, 2\" ``` Will result in ` 1.2.0, 3.0.0 `. Leading zeros `0.0.x` and `0.x.y` The first non zero version acts as a major version. ``` compat Example \"0.2.1\" ``` Will result in ` 0.2.1, 0.3.0 `. Caret specifier A caret `^` specifier allows upgrade that would be compatible according to semantic versioning. ``` compat PkgA \"^1.2.3\" 1.2.3, 2.0.0 PkgB \"^1.2\" 1.2.0, 2.0.0 PkgC \"^1\" 1.0.0, 2.0.0 ``` Tilde specifier A tilde specifier provides more limited upgrade possibilities. When specifying major, minor and patch versions, or when specifying major and minor versions, only the patch version is allowed to change. If you only specify a major version, then both minor and patch versions are allowed to be upgraded `~1` is thus equivalent to `^1` . For example ``` compat PkgA \"~1.2.3\" 1.2.3, 1.3.0 PkgB \"~1.2\" 1.2.0, 1.3.0 PkgC \"~1\" 1.0.0, 2.0.0 ``` Other specifier You can also express equality and inequality , as well as ranges. See Compatibility https pkgdocs.julialang.org v1 compatibility for more information. \"\"\" md\"\"\" Workflow \"\"\" md\"\"\" 1. Clone repository 2. Check if the repository has a `Project.toml` If not, yell at the author Check the ` deps ` section, is there an entry for every dependency? 3. Check the project for a `Manifest.toml` ```sh head n 3 Manifest.toml This file is machine generated editing it directly is not advised julia version \"1.11.5\" ``` 4. Instantiate the manifest `julia 1.11 project . e \"import Pkg Pkg.instantiate \"` note Instantiate will not resolve the environment again, it will just install all the dependencies listed in the `Manifest.toml`. Make sure to match the version. \"\"\" md\"\"\" Numerical reproducibility What every Computer Scientist should know about Floating Point arithmetic https dl.acm.org doi 10.1145 103162.103163 \"\"\" md\"\"\" Randomness We sometimes \"desire\" randomness, but true randomness is hard in a computer. In computer programs we talk about \"pseudo random number generators\", given a seed they generate a \"unpredicatble\" sequence of numbers with a very large period. \"\"\" using Random rand rand let Random.seed 182 rand end let Random.seed 182 rand end walk x rand x 1, x 1 function walk nsteps N steps 0 for in 2 N push steps, walk steps end end return steps end let fig Figure ax Axis fig 1, 1 , xlabel \"Position\", ylabel \"Steps\", title \"Unseeded\" N 100 lines ax, walk nsteps N , 1 N lines ax, walk nsteps N , 1 N lines ax, walk nsteps N , 1 N lines ax, walk nsteps N , 1 N fig end let fig Figure ax Axis fig 1, 1 , xlabel \"Position\", ylabel \"Steps\", title \"Seeded\" N 100 Random.seed 12 lines ax, walk nsteps N , 1 N lines ax, walk nsteps N , 1 N lines ax, walk nsteps N , 1 N lines ax, walk nsteps N , 1 N fig end md\"\"\" Order of summing Kahan summation https en.wikipedia.org wiki Kahan summation algorithm Pairwise summation used in Julia https github.com JuliaLang julia pull 4039 \"\"\" 0.1 0.2 0.3 0.1 0.2 0.3 md\"\"\" note My first \"silly question\" https stackoverflow.com questions 21872854 floating point math in different programming languages \"\"\" md\"\"\" Floating point arithmetic doesn't quite follow the rules of real arithmetic. Instead the rules are defined in a standard called `IEEE 754`. Because summation in different order will give different answers, the compiler has no lease to reorder operations for us. This is in particular relevant for vectorization and parallel computation. Code may give different answers due to different execution order. \"\"\" md\"\"\" 2046 floating pointer numbers that sum to almost anything Thanks to Stefan Karpinski https discourse.julialang.org t array ordering and naive summation 1929 \"\"\" md\"\"\" The `sumsto` function always returns the same 2046 floating point numbers but returns them in a different order based on `x` for any 64 bit float value of `x` from 0 up to but not including 2^970, the naive left to right sum of the vector returned by `sumsto x ` is precisely `x` \"\"\" function sumsto x Float64 0 x exp2 970 || throw ArgumentError \"sum must be in 0,2^970 \" n, p₀ Base.decompose x integers such that `n exp2 p₀ x` floatmax exp2 p for p in 1074 969 if iseven n p p₀ floatmax exp2 p for p in 1074 969 if isodd n p p₀ end foldl , sumsto 0.0 foldl , sumsto eps 0.0 foldl , sumsto 1.23 foldl , sumsto pi 0 foldl , sumsto 6.0221409e23 foldl , sumsto 9.979201547673598e291 md\"\"\" When adding the values from left to right, all the powers of two after `floatmax ` but before ` floatmax ` have no effect on the sum. Then ` floatmax ` cancels `floatmax ` out, bringing the sum back to zero so the remaining powers of two are added up as expected, giving the final result, `x`. There are 2044 powers of two that can be represented as 64 bit floating point value below 2^970 since the smallest representable power of 2 is `eps 0.0 exp2 1074 5.0e 324`. So the number of values we’re summing is 1074 970 2, one for each power of two and two more for `floatmax ` and ` floatmax `. \"\"\" md\"\"\" Fast math Simon Byrne notes on Fast Math https simonbyrne.github.io notes fastmath Towards Useful Fast Math https llvm.org devmtg 2024 10 slides techtalk Kaylor Towards Useful Fast Math.pdf \"\"\" function foo A 1.0f0 C 1.0f0 Find the smallest value A 2^k for which A 1 A 1 while C 1.0f0 A 2.0f0 C A 1.0f0 A end return A end foo md\"\"\" warning Do not execute `foo fast` It will loop forever. \"\"\" function foo fast A 1.0f0 C 1.0f0 Find the smallest value A 2^k for which A 1 A 1 fastmath while C 1.0f0 A 2.0f0 C A 1.0f0 A end return A end with terminal do code llvm foo fast end md\"\"\" Precision of mathematical implementations CPU vs GPU? `muladd` can given different answer expansion of precsion... \"\"\" md\"\"\" Extending precision, can change the answer, as an example in Swift, operations may be executed in extended precision and one might get different answers under transformations. Take the code below and run it in https swiftwasm.org ```swift func foo x Float16 Float16 let s Float16 1024 let t Float16 3 s let rx Float16 x x t t return rx print foo x 0.5 0.0 func bar x Float16 Float16 let s Float16 1024 let t Float16 3 s let rx Float16 x bar plus a x, b t t return rx inline never func bar plus a Float16, b Float16 Float16 return a b print bar x 0.5 0.5 ``` \"\"\" md\"\"\" Our goal in Julia for `Float16` is to emulate what hardware with real Float16 support would do. \"\"\" md\"\"\" This behavior is important when we implement our own trigonometic functions https github.com JuliaLang julia blob 8987f7ac6d284cb512d336279a74b5414c635757 base special trig.jl L764 If we look at `foo` we would think that `x x t t ` should always return `0.0`, but we choose `t` specially for that to not be the case. \"\"\" maxintfloat Float16 2 Float16 1024 function foo x T where T s maxintfloat T 2 t 3 s rx x x t t end foo Float32 0.5 foo Float64 0.5 foo Float16 0.5 function bar x Float16 s Float16 1024 t 3 s rx x bar plus x,t t end noinline function bar plus a, b return a b end bar Float16 0.5 md\"\"\" So what happens is that Swift and Julia in olden times internally expanded `Float16` operations to `Float32` \"\"\" begin extend x Float16 Float32 x extend x Float32 Float64 x end function foo extended x T where T s maxintfloat T 2 t extend 3 s extend precision rx T x x t t end foo extended Float16 0.5 foo extended Float32 0.5 md\"\"\" This answer is seemingly \"more\" correct since it calculates our \"real\" understanding of x x t t , but it is more useless since it does not accuratly implement floating point semantics. \"\"\" md\"\"\" Implementation of Float16 \"\"\" md\"\"\" ```julia abstract type Number end abstract type Real Number end abstract type AbstractFloat Real end primitive type Float64 AbstractFloat 64 end primitive type Float32 AbstractFloat 32 end primitive type Float16 AbstractFloat 16 end ``` \"\"\" methods cbrt md\"\"\" First attempt Naively lowering Float16 to LLVM’s half type. What to do on platforms with no limited hardware support Extended precision thanks x87 rears it’s ugly head Lesson In order to implement numerical routines that are portable we must be very careful in what semantics we promise. Solution On targets without hardware support for `Float16`, truncate after each operation. GCC 12 supports this as ` fexcess precision 16` \"\"\" md\"\"\" On x86 \"\"\" md\"\"\" ```llvm define half julia muladd half %0, half %1, half %2 top %3 fmul half %0, %1 %4 fadd half %3, %2 ret half %4 ``` \"\"\" md\"\"\" turns into \"\"\" md\"\"\" ``` define half julia muladd half %0, half %1, half %2 top %3 fpext half %0 to float %4 fpext half %1 to float %5 fmul float %3, %4 %6 fptrunc float %5 to half %7 fpext half %6 to float %8 fpext half %2 to float %9 fadd float %7, %8 %10 fptrunc float %9 to half ret half %10 ``` \"\"\" md\"\"\" On your machine? \"\"\" with terminal do code llvm muladd, Float16, Float16, Float16 , optimize false end with terminal do code llvm muladd, Float16, Float16, Float16 end md\"\"\" Stochastic rounding 🎲 Sometimes randomness can be quite useful Real numbers constitue a continuous set \\mathbb R , but finite precision numbers used in computers are a part of a discrete set F \\subset \\mathbb R . When computers do operations involving floating point numbers in F , the true result x \\in \\mathbb R will be approximated by a number \\hat x \\in F , which is typically chosen deterministically to be the nearest number in F this is called \"nearest rounding\". Stochastic rounding is an alternative rounding mode to classic deterministic rounding, which randomly rounds a number x \\in \\mathbb R to either of the two nearest floating point numbers of the result \\lfloor x \\rfloor previous number in F or \\lceil x \\rceil following number in F with the following rule ```math \\mathrm round x \\begin cases \\lfloor x \\rfloor & \\text with probability P x \\\\ 6pt \\lceil x \\rceil & \\text with probability 1 P x \\end cases ``` Common choices are P x 1 2 or, more interestingly, ```math P x \\frac x \\lfloor x \\rfloor \\lceil x \\rceil \\lfloor x \\rfloor ``` In the following we'll always talk about the latter probability function P x . Resource \"https nickhigham.files.wordpress.com 2020 06 stoch round fig2.jpg\" source \" What Is Stochastic Rounding? https nhigham.com 2020 07 07 what is stochastic rounding \" by Nick Higham Stochastic rounding is useful because the average result of operations matches the expected mathematical result. In a statistical sense, it retains some of the information that is discarded by a deterministic rounding scheme, smoothing out numerical rounding errors due to limited precisions. This is particularly important when using low precision floating point numbers like `Float16`. For contrast, deterministic rounding modes like nearest rounding introduce a bias, which is more severe as the precision of the numbers is lower. The IPU is one of the very few processors available with hardware support for stochastic rounding. Let's do an exercise on the CPU with classical nearest rounding. We define a function to do the naive sequential sum of a vector of numbers, because the `sum` function in Julia uses pairwise summation https en.wikipedia.org wiki Pairwise summation , which would have better accuracy. From Mosè Giordano talk on using Julia on IPU https giordano.github.io talks 2024 06 20 julia ipu ornl b11731af d8f3 4714 839d 1535f459a278 \"\"\" naive sum v foldl , v x fill Float16 0.9 , 3000 naive sum x eps Float16 2048 naive sum x ≈ x 1 length x using StochasticRounding using Statistics x sr Float16sr. x naive sum x sr sums sr map naive sum x sr , 1 10000 extrema sums sr mean Float64. sums sr std Float64. sums sr median sums sr let fig Figure ax Axis fig 1,1 stephist ax, sums sr, label \"Stochastic Rounding\", color green vlines ax, naive sum x , , label \"Nearest Rounding\" vlines ax, x 1 length x , label \"True value\" axislegend ax position ct fig end "},{"url":"mod3_parallelism/accelerated/","title":"GPU Computing","tags":["module3","track_parallel"],"text":" A Pluto.jl notebook v0.20.13 frontmatter chapter \"3\" section \"2\" order \"9\" title \"GPU Computing\" date \"2025 06 11\" tags \"module3\", \"track parallel\" layout \"layout.jlhtml\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils This Pluto notebook uses bind for interactivity. When running this notebook outside of Pluto, the following 'mock version' of bind gives bound variables a default value instead of an error . macro bind def, element format off return quote local iv try Base.loaded modules Base.PkgId Base.UUID \"6e696c72 6542 2067 7265 42206c756150\" , \"AbstractPlutoDingetjes\" .Bonds.initial value catch b missing end local el esc element global esc def Core.applicable Base.get, el ? Base.get el iv el el end format on end using PlutoUI, PlutoTeachingTools using CairoMakie ChooseDisplayMode PlutoUI.TableOfContents depth 4 md\"\"\" Accelerated Computing \"\"\" md\"\"\" Accelerated \"\"\" md\"\"\" A GPU has many \"lightweight\" threads \"\"\" md\"\"\" CPU die shot RobustLocalResource \"https s3.amazonaws.com media p.slid.es uploads 779853 images 4399827 800px coffee lake die quad core annotated .png\", \"quad core.png\" \"\"\" md\"\"\" GPU block diagram RobustLocalResource \"https devblogs.nvidia.com parallelforall wp content uploads 2016 04 gp100 block diagram 1 624x368.png\", \"p100.png\" \"\"\" md\"\"\" Bottlenecks RobustLocalResource \"https s3.amazonaws.com media p.slid.es uploads 779853 images 4399848 pasted from clipboard.png\", \"gpu system.png\" \"\"\" md\"\"\" Composable infrastructure Core GPUCompiler.jl Takes native Julia code and compiles it directly to GPUs GPUArrays.jl High level array based common functionality KerneAbstractions.jl Vendor agnostic kernel programming language Adapt.jl Translate complex structs across the host device boundary Vendor specific CUDA.jl AMDGPU.jl oneAPI.jl Metal.jl \"\"\" md\"\"\" Different layers of abstraction Vendor specific ```julia using CUDA function saxpy a,X,Y i blockIdx .x 1 blockDim .x threadIdx .x if i length Y inbounds Y i a X i Y i end return nothing end cuda threads 32 blocks cld length Y , 32 saxpy a, X, Y ``` KernelAbstractions ```julia using KernelAbstractions using CUDA kernel function saxpy a, Const X , Y I index Global inbounds Y I a X I Y I end saxpy CUDABackend a, X, Y, ndrange length Y ``` Array abstractions ```julia Y . a . X . Y ``` \"\"\" md\"\"\" Asynchronous operations warn GPU operations are asynchronous with regards to the host They are ordered with respect to each other, but special care must be taken when using Julia's task based programming together with GPU programming. The JuliaGPU ecosystem synchronizes the GPU on access, so when you move data from and to the GPU we wait for all the kernels to finish See the exercise \"Introduction to KernelAbstractions\" for more details on how to measure GPU kernels and performance. \"\"\" md\"\"\" How to use KernelAbstractions Use ` kernel function mykernel args... end` to write a GPU style program Instantiate kernel for a backend `kernel mykernel backend ` Backends come from Vendor specific libraries `KA.allocate backend, ... ` to obtain memory Launch kernel `kernel args..., ndrange ... ` while specifying the grid to execute over. \"\"\" TwoColumn md\"\"\" ```julia function vadd a, b, c for i in eachindex c c i a i b i end end a rand N b rand N c similar a vadd a, b, c ``` \"\"\", md\"\"\" ```julia import KernelAbstractions as KA kernel function vadd a, b, c i index Global c i a i b i end backend CUDABackend a KA.allocate backend, Float32, N b KA.allocate backend, Float32, N c similar a vadd kernel vadd backend vadd kernel a, b, c ndrange size c ``` \"\"\" md\"\"\" note GPU execution is asynchronous We will discuss the details in the later GPU lecture. When benchmarking you need to synchronize the device ```julia benchmark begin vadd kernel a, b, c ndrange size c KA.synchronize backend end ``` Otherwise you are only measuring the launch of the kernel. \"\"\" md\"\"\" High level array based programming Julia and GPUArrays.jl provide support for an efficient GPU programming environment build around array abstractions and higher order functions. Vocabulary of operations `map`, `broadcast`, `scan`, `reduce`, ... Map naturally onto GPU execution models Compiled to efficient code multiple dispatch, specialization Write generic, reusable applications BLAS matrix multiply, ... , and other libraries like FFT Array operators using multiple dispatch a design methodology for array implementations in dynamic languages doi 10.1145 2627373.2627383 Rapid software prototyping for heterogeneous and distributed platforms doi 10.1016 j.advengsoft.2019.02.002 \"\"\" md\"\"\" Array types where memory resides and how code is executed. | | | | | | | `A Matrix Float64 undef, 64, 32 ` | CPU | | `A CuMatrix Float64 undef, 64, 32 ` | Nvidia GPU | | `A ROCMatrix Float64 undef, 64, 32 ` | AMD GPU | info Data movement is explicit. \"\"\" md\"\"\" What makes an application portable? 1. Can I run it on a different compute architecture 1. Different CPU architectures 2. We live in a mult GPU vendor world 2. Does it compute the same thing? 1. Can I develop on one platform and move to another later? 3. Does it achieve the same performance ? 4. Can I take advantage of platform specific capabilities? Productivity meets Performance Julia on A64FX doi 10.1109 CLUSTER51413.2022.00072 \"\"\" md\"\"\" Adapt.jl Adapt.jl https github.com JuliaGPU Adapt.jl is a lightweight dependency that you can use to convert complex structures from CPU to GPU. ```julia using Adapt adapt CuArray, Adjoint Array Adjoint CuArray ``` ```julia struct Model T Number, AT AbstractArray T data AT end Adapt.adapt structure to, x Model Model adapt to, x.data cpu Model rand 64, 64 using CUDA gpu adapt CuArray, cpu Model Float64, CuArray Float64, 2, CUDA.Mem.DeviceBuffer ... ``` \"\"\" md\"\"\" Writing a custom sum function with KernelAbstractions \"\"\" using KernelAbstractions, BenchmarkTools begin import CUDA import Metal Work around https github.com JuliaGPU oneAPI.jl issues 445 . ENV \"SYCL PI LEVEL ZERO BATCH SIZE\" \"1\" import oneAPI sigh Currently crashes inside Pluto import AMDGPU end let available backends KernelAbstractions.Backend CPU CI get ENV, \"GITHUB ACTIONS\", \"false\" \"true\" Always shows all backend when rendering the notebook on GitHub Actions, notebooks will be static anyway. if CUDA.functional || CI push available backends, CUDA.CUDABackend end if oneAPI.functional || CI push available backends, oneAPI.oneAPIBackend end if Metal.functional || CI push available backends, Metal.MetalBackend end if AMDGPU.functional || CI push available backends, AMDGPU.ROCBackend end bind backend Select available backends end md\"\"\" The below code implements a custom eigenvalue solver. We are interested in implementing porting it to the GPU. For a momement let's forget about `GPUArrays` and all the high level primitives it provides. \"\"\" md\"\"\" ``` f A MyMatrix λ 1 mapreduce v v^2 v λ , , A.v f′ A MyMatrix λ mapreduce v v^2 v λ ^2, , A.v function LinearAlgebra.eigmax A MyMatrix tol eps 2.0 , debug false x0 maximum A.v maximum A.v ^2 δ f A x0 f′ A x0 while abs δ x0 tol x0 δ δ f A x0 f′ A x0 debug && println \"x x0, δ δ\" Debugging end x0 end ``` \"\"\" question box md\"\"\" 1. What fundamental operations do we need to implement? 2. Is there a synchronization point within this code? \"\"\" hint md\"\"\" 1. `maximum v ` can also be written as `reduce max, v, init typemin eltype v ` 2. `reduce op, v ` can also be written as `mapreduce identity, op, f ` So the only operation we need to implement is a `mapreduce` Yes, there are several synchronization points, most notably both `f` and `f′` have to wait for the GPU kernel to finish to return a scalar GPU value. An interesting optimization might be to better pipeline this code and try to avoid a bubble between `f` and `f′`. Or perhaps write a fully custom kernel \"\"\" md\"\"\" We can only communicate with the GPU through Global Memory. A GPU kernel does not return anything. Thus we allocate a single \"value\" storage array. note \"Shared Virtual Memory Unified Virtual Memory\" SVM UVM make memory on the GPU CPU mutually accesibly, we could thus not use an array, but perhaps a `RefValue`, though this is not portable and would mean we miss the implicit syncronization semantics. For our custom eigenvalue solver we need `mapreduce`, but for now let's just implement a non generic `sum`. \"\"\" kernel function naive sum val, data I index Global x data I val 1 x end data KernelAbstractions.ones backend, Int, 1024^2 sum data let x KernelAbstractions.zeros backend, eltype data , 1 naive sum backend x, data, ndrange length data Array x 1 end md\"\"\" note \"Data race\" This looks similar to our issues from last week Data races galore Let's use Atomix again to avoid these issues. \"\"\" using Atomix atomic, atomicswap, atomicreplace kernel function naive atomic sum val, data I index Global x data I atomic val 1 x end let x KernelAbstractions.zeros backend, eltype data , 1 oops I used similar here... naive atomic sum backend x, data, ndrange length data Array x 1 end let x KernelAbstractions.zeros backend, eltype data , 1 benchmark begin naive atomic sum backend x, data, ndrange length data KernelAbstractions.synchronize backend end end benchmark sum data if backend isa CUDA.CUDABackend CUDA. profile sum data end md\"\"\" Currently we have a big bottle neck. The atomic summation on the global data KernelAbstraction provides a ` localmem` region that implements a memory region that is local to a work group So we can load all the data in parallel, then sum it up, and perform much fewer atomic stores. \"\"\" kernel function naive shmem sum val, data AbstractArray T where T I index Global i index Local N uniform prod groupsize tile localmem T N, tile i data I synchronize if i 1 acc tile 1 for i in 2 N acc tile i end atomic val 1 acc end end let x KernelAbstractions.zeros backend, eltype data , 1 naive shmem sum backend, 64 x, data, ndrange length data Array x 1 end let x KernelAbstractions.zeros backend, eltype data , 1 benchmark begin naive shmem sum backend, 64 x, data, ndrange length data KernelAbstractions.synchronize backend end end md\"\"\" info There are improvements we can do on this algorithm As an example one thread is doing all the \"sum\" work per compute group. We probably want to share that work. In CUDA there are also shuffle operations that can be used instead of the `localmem`. For now let's write a \"mapreduce\" code that applies an operation `f` to each element. \"\"\" md\"\"\" warn The code below contains an loop that uses atomic operations to update a value... Developing this is a pain since it can easily hang your system. \"\"\" md\"\"\" note Sometimes things are broken. Originally the plan was to implement a generic `mapreduce` kernel using atomic operations, but this did not work out generally. Instead we implement only `mapsum` ```julia kernel function naive shmem mapreduce val, data AbstractArray T , f F, op O where T, F, O I index Global i index Local N uniform prod groupsize tile localmem T N, tile i f data I synchronize if i 1 acc tile 1 for i in 2 N acc op acc, tile i end TODO Generic atomic breaks atomic val 1 op acc val 1 op val 1 , acc v atomic val 1 success false while success v, success atomicreplace val 1 v op v, acc end end end ``` \"\"\" kernel function naive shmem mapsum val, data AbstractArray T , f F where T, F I index Global i index Local N uniform prod groupsize tile localmem T N, tile i f data I synchronize if i 1 acc tile 1 for i in 2 N acc tile i end atomic val 1 acc end end using GPUArraysCore function my mapsum f, data backend get backend data x KernelAbstractions.zeros backend, eltype data , 1 naive shmem mapsum backend, 64 x, data, f, ndrange length data GPUArraysCore. allowscalar x 1 end my mapsum x x^2, data mapreduce x x^2, , data using Adapt struct MyMatrix A v A end Adapt.adapt structure to, x MyMatrix MyMatrix adapt to, x.v adapt backend, MyMatrix X X randn 1024 md\"\"\" \"\"\" f A MyMatrix λ 1 my mapsum v v^2 v λ , A.v f′ A MyMatrix λ my mapsum v v^2 v λ ^2, A.v using LinearAlgebra function LinearAlgebra.eigmax A MyMatrix tol eps 2.0 , debug false x0 my mapreduce identity, max, A.v my mapreduce identity, max, A.v ^2 x0 maximum A.v maximum A.v ^2 Cheating by using GPUArrays δ f A x0 f′ A x0 while abs δ x0 tol x0 δ δ f A x0 f′ A x0 debug && println \"x x0, δ δ\" Debugging end x0 end eigmax diagm X X X' eigmax MyMatrix X eigmax adapt backend, MyMatrix X backend "},{"url":"mod3_parallelism/distributed/","title":"Distributed programming with MPI.jl","tags":["module3","track_parallel"],"text":" A Pluto.jl notebook v0.20.13 frontmatter chapter \"3\" section \"3\" order \"10\" title \"Distributed programming with MPI.jl\" date \"2025 06 18\" tags \"module3\", \"track parallel\" layout \"layout.jlhtml\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils This Pluto notebook uses bind for interactivity. When running this notebook outside of Pluto, the following 'mock version' of bind gives bound variables a default value instead of an error . macro bind def, element format off return quote local iv try Base.loaded modules Base.PkgId Base.UUID \"6e696c72 6542 2067 7265 42206c756150\" , \"AbstractPlutoDingetjes\" .Bonds.initial value catch b missing end local el esc element global esc def Core.applicable Base.get, el ? Base.get el iv el el end format on end begin using PlutoUI, PlutoTeachingTools using PlutoUI Slider PlutoUI.TableOfContents depth 4 end ChooseDisplayMode md\"\"\" Distributed computing with MPI.jl \"\"\" using MPI, Serialization, Statistics, StaticArrays md\"\"\" MPI stands for \"Message Passing Interface\" and is one of the oldest and most successful distributed programming paradigms. We normally start an MPI program with `mpiexec` see exercises , but this doesn't play nicely with Pluto. Here I use a ` mpi` macro to execute a block of code as a standalone MPI. `mpiexec` starts a number of independent processes that execute the same program. We can use API calls like `MPI.Comm rank` and `MPI.Comm size` to query which rank we are. \"\"\" md\"\"\" The ` mpi` macro executes a block as an MPI program. note Each block is isolated from another, and as such you need to setup state independently. warning The ` mpi` macro is purely to make MPI work in Pluto for teaching, but should be not used for any real uses. Furthermore, always wrap your blocks in `let` and not `begin` to not confuse Pluto. \"\"\" np 4 macro mpi np, expr path, io mktemp control io path path \".ji\" println io, \"using MPI, Serialization\" println io, \" mpi begin\" println io, expr println io, \"end\" println io, \"\"\" mpi MPI.gather mpi, MPI.COMM WORLD root 0 if MPI.Comm rank MPI.COMM WORLD 0 Serialization.serialize \" control io path\", mpi end \"\"\" close io quote let np esc np path path run ` mpiexec np np Base.julia cmd project Base.active project path ` end v Serialization.deserialize control io path rm control io path all isnothing, v ? nothing v end end md\"\"\" When using MPI we are going to execute N np independent copies of a program. \"\"\" mpi np let MPI.Init n rand println n end md\"\"\" As we can see the output of the program is mangled. note Most often we only print from `rank 0`. \"\"\" mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm if rank 0 println \"Comm size MPI.Comm size comm \" end end md\"\"\" It can also help to print in \"one shot\" \"\"\" mpi np let MPI.Init comm MPI.COMM WORLD print \"\"\" Hello world, I am rank MPI.Comm rank comm of MPI.Comm size comm \"\"\" MPI.Barrier comm wait for all processes to reach this point. end md\"\"\" Initialization and Finalization As you may have noticed when we start an MPI program we have to explicitly call `MPI.Init` In other programming languages you will have to use a corresponding `MPI.Finalize`, this is not necessary in Julia since we automatically execute `MPI.Finalize` when the program shuts down. \"\"\" md\"\"\" Two sided communication \"\"\" md\"\"\" Blocking \"\"\" mpi 2 let function pingpong T, sz, iters buff zeros T, sz comm MPI.COMM WORLD rank MPI.Comm rank comm MPI.Barrier comm tic MPI.Wtime for in 1 iters if rank 0 MPI.Send buff, comm dest 1 MPI.Recv buff, comm source 1 else MPI.Recv buff, comm source 0 MPI.Send buff, comm dest 0 end end toc MPI.Wtime return toc tic iters end MPI.Init pingpong Float64, 1024, 10 end md\"\"\" note The program above is limited to two ranks Let's send data around in a ring \"\"\" mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm N MPI.Comm size comm assert N 1 if rank 0 token MPI.recv comm source rank 1 print \"Rank rank received token token.\\n\" else token 1 end MPI.send token, comm dest mod rank 1, N Now rank 0 can receive the token if rank 0 token MPI.recv comm source N 1 print \"Rank rank received token token.\\n\" end token end md\"\"\" Non Blocking It is quite tricky to reason about the order of `recv` and `send` across all processes. Note how we always have to set up some `recv` then perform the matching `send`. Then flip the operations. `Irecv ` and `Isend` are non blocking operations, that means we can issue them and wait later for them to be completed. note \"Tagging\" Since we now can have many operations in flight at once, one might need to tag the operations. \"\"\" mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm N MPI.Comm size comm nonblocking receive from previous rank recv buf Array Float64 undef, 2 recv req MPI.Irecv recv buf, comm source mod rank 1, N , tag 0 nonblock send to next rank send buf Float64 rank, rank send req MPI.Isend send buf, comm dest mod rank 1, N , tag 0 block until communication is complete MPI.Waitall recv req, send req print \" rank Received recv buf\\n\" end md\"\"\" Ghost cells \"\"\" md\"\"\" | 1 | 2 11 | 12 | | | | | | left | mine | right | \"\"\" mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm N MPI.Comm size comm M 10 data zeros M 2 Computation phase interior data 2 M 1 . rank Communication phase begin Receive into ghost cells from neighbors left recv MPI.Irecv view data, 1 , comm source mod rank 1, N , right recv MPI.Irecv view data, M 2 , comm source mod rank 1, N Send to neighbors left send MPI.Isend view data, 2 , comm dest mod rank 1, N , right send MPI.Isend view data, M 1 , comm dest mod rank 1, N block until communication is complete MPI.Waitall left recv, right recv, left send, right send end print \" rank Received data\\n\" Computation phase ... end md\"\"\" Collective communication and synchronization \"\"\" md\"\"\" MPI defines a series of \"collective\" operations, operations that all processes within a communicator partake in. We have already encountered the `MPI.Barrier` operation that forces all processes to wait. \"\"\" md\"\"\" One possible but inefficient way to implement a barrier is the ring communication example from above ```julia if rank 0 token MPI.recv comm source rank 1 print \"Rank rank received token token.\\n\" else token 1 end MPI.send token, comm dest mod rank 1, N Now rank 0 can receive the token if rank 0 token MPI.recv comm source N 1 print \"Rank rank received token token.\\n\" end ``` \"\"\" md\"\"\" We set up a chain of blocking operations and the program can only continue thereafter \"\"\" md\"\"\" Broadcast Sends data from 1 to N \"\"\" mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm if rank 0 x rand 3 else x zeros 3 end MPI.Bcast x, comm root 0 print \" rank Received x\\n\" end mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm function my bcast data, comm root 0 rank MPI.Comm rank comm N MPI.Comm size comm if rank root send data to all other ranks for other in 0 N 1 if other root MPI.Send data, comm dest other end end else MPI.Recv data, comm source root end end if rank 0 x rand 3 else x zeros 3 end my bcast x, comm root 0 print \" rank Received x\\n\" end md\"\"\" MPI implementation may choose their own more efficient implementations This implementation is inefficient since it is limited by the bandwidth of the root process. Better would be something like a communication tree \"\"\" bind np bench Select 2, 4, 8, 12 , default 4 avg my bcast mean x x 1 , bcast times avg mpi bcast mean x x 2 , bcast times avg mpi bcast avg my bcast bcast times mpi np bench let function my bcast data, comm root 0 rank MPI.Comm rank comm N MPI.Comm size comm if rank root send data to all other ranks for other in 0 N 1 if other root MPI.Send data, comm dest other end end else MPI.Recv data, comm source root end end function benchmark T, sz, iters comm MPI.COMM WORLD data zeros T, sz my bcast time 0.0 mpi bcast time 0.0 warmup my bcast data, comm MPI.Bcast data, comm for in 1 iters MPI.Barrier comm my bcast time MPI.Wtime my bcast data, comm MPI.Barrier comm my bcast time MPI.Wtime MPI.Barrier comm mpi bcast time MPI.Wtime MPI.Bcast data, comm MPI.Barrier comm mpi bcast time MPI.Wtime end my bcast time iters, mpi bcast time iters end MPI.Init benchmark Float64, 400000, 10 magical return to Pluto value end md\"\"\" Scatter Scatter is very similar to broadcast. Broadcast sends the same data to all processes. Scatter sends chunks of an array to different processes. \"\"\" mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm N MPI.Comm size comm x zeros Int, 3 data rank 0 ? collect 1 N 3 nothing MPI.Scatter data, x, comm root 0 print \" rank Received x\\n\" end md\"\"\" Gather Collect data from N to 1. note Gather is the inverse of scatter. \"\"\" md\"\"\" In MPI communication is explicit Let's generate `np` np random numbers and send them all to the root rank. \"\"\" mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm x rand xs rank 0 ? zeros MPI.Comm size comm nothing if rank 0 show xs end MPI.Gather Ref x , xs, comm root 0 if rank 0 show xs end x magical return to Pluto value end md\"\"\" note `Ref x ` is a one element container, like ` x ` \"\"\" mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm x rand xs MPI.Gather x , comm root 0 if rank 0 show xs end x end mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm x rand xs MPI.gather x, comm root 0 if rank 0 show xs end x end md\"\"\" MPI.jl often has three variants of functions `Gather ` Closest to the underlying MPI implementation, expects input and output buffer. `Gather` Convinience function that allocates on the receiving side, expects and input buffer. `gather` Handles arbitrary Julia objects and allocates on the receiving side. \"\"\" md\"\"\" Reduce `Gather` moves all values from all ranks to one root rank. Instead of copying values and then manually reducing them, MPI supports an `MPI.Reduce` call. \"\"\" md\"\"\" Compute \\int 0^1 \\frac 4 1 x^2 dx 4 atan x 0^1 which evaluates to π \"\"\" pis mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm TODO Support interpolation into ` mpi` macro n 50 000 000 s 0.0 for i MPI.Comm rank comm 1 MPI.Comm size comm n x i .5 n s 4 1 x^2 end mypi s n our π MPI.Reduce mypi, MPI.SUM, comm root 0 if rank 0 println \"Error our π π our π π \" end mypi end our π sum pis mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm sum of ranks MPI.Reduce rank, , comm root 0 if rank 0 println \"sum of ranks sum of ranks \" end sum of ranks end md\"\"\" note We can also use any Julia function and not just the limited set of reduction operators defined by the standard. \"\"\" md\"\"\" Allgather and Allreduce `Gather` and `Reduce` are N to 1 collectives and they have sibling varieties that perform an all to all communication. \"\"\" mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm sum of ranks MPI.Allreduce rank, , comm if rank 0 println \"sum of ranks sum of ranks \" end sum of ranks end mpi np let MPI.Init comm MPI.COMM WORLD rank MPI.Comm rank comm ranks MPI.Allgather rank , comm ranks end md\"\"\" Communicators \"\"\" md\"\"\" So far we have been using ```julia comm MPI.COMM WORLD ``` without much explanation. `MPI.COMM WORLD` is the default communicator and contains all available ranks Other communicators can be used to partition our computation. `MPI.COMM SELF` Just ourselves `MPI.COMM TYPE SHARED` All ranks on one physical node `MPI.COMM NULL` Nobody We can split our communicator \"\"\" mpi np let MPI.Init world rank MPI.Comm rank MPI.COMM WORLD comm MPI.Comm split MPI.COMM WORLD, iseven world rank , world rank rank MPI.Comm rank comm print \"World rank world rank, Split rank rank\\n\" end md\"\"\" note `Comm split` is probably the easiest way of splitting a communicator, but modern MPI uses MPI Groups which are a bit more flexible. \"\"\" mpi np let MPI.Init world rank MPI.Comm rank MPI.COMM WORLD world group MPI.Comm group MPI.COMM WORLD odd ranks convert. Int32, collect 1 2 MPI.Comm size MPI.COMM WORLD odd group MPI.Group incl world group, odd ranks even group MPI.Group difference world group, odd group group iseven world rank ? even group odd group comm MPI.Comm create group MPI.COMM WORLD, group, tag 0 non collective only ranks in group must participate rank MPI.Comm rank comm print \"World rank world rank, Split rank rank\\n\" end md\"\"\" We can also duplicate communicators This can be important if you want to hand off a communicator to a library maybe one implemented in C Fortran and you are unable to negotiate a common tag scheme. Otherwise we might respond to messages meant for the library and not us \"\"\" mpi np let MPI.Init world rank MPI.Comm rank MPI.COMM WORLD comm MPI.Comm dup MPI.COMM WORLD rank MPI.Comm rank comm print \"World rank world rank, Other rank rank\\n\" end md\"\"\" Other examples \"\"\" md\"\"\" Custom ops and custom data types \"\"\" mpi np let using Statistics Define a custom struct This contains the summary statistics mean, variance, length of a vector struct SummaryStat mean Float64 var Float64 n Float64 end function SummaryStat X AbstractArray m mean X v varm X,m, corrected false n length X SummaryStat m,v,n end Define a custom reduction operator this computes the pooled mean, pooled variance and total length function pool S1, S2 n S1.n S2.n m S1.mean S1.n S2.mean S2.n n v S1.n S1.var S1.mean S1.mean m S2.n S2.var S2.mean S2.mean m n SummaryStat m,v,n end MPI.Init comm MPI.COMM WORLD root 0 X randn 10,3 . 1,3,7 ' Perform a scalar reduction summ MPI.Reduce SummaryStat X , pool, comm root if MPI.Comm rank comm root show summ.var end Perform a vector reduction the reduction operator is applied elementwise col summ MPI.Reduce mapslices SummaryStat,X,dims 1 , pool, comm root if MPI.Comm rank comm root col var map summ summ.var, col summ show col var end nothing end md\"\"\" Static Vectors \"\"\" mpi np let using StaticArrays MPI.Init comm MPI.COMM WORLD x ones SVector 3, Float64 sum MPI.Allreduce x , , comm if MPI.Comm rank comm 0 show sum end nothing end "},{"url":"mod3_parallelism/interactive_tuning/","title":"Interactive performance tuning with Julia","tags":["module3","track_parallel","indepth"],"text":" A Pluto.jl notebook v0.20.13 frontmatter chapter \"3\" section \"1\" order \"8.2\" title \"Interactive performance tuning with Julia\" tags \"module3\", \"track parallel\", \"indepth\" layout \"layout.jlhtml\" indepth number \"3\" frontmatter.author name \"Mosè Giordano\" url \"https giordano.github.io \" using Markdown using InteractiveUtils This Pluto notebook uses bind for interactivity. When running this notebook outside of Pluto, the following 'mock version' of bind gives bound variables a default value instead of an error . macro bind def, element format off return quote local iv try Base.loaded modules Base.PkgId Base.UUID \"6e696c72 6542 2067 7265 42206c756150\" , \"AbstractPlutoDingetjes\" .Bonds.initial value catch b missing end local el esc element global esc def Core.applicable Base.get, el ? Base.get el iv el el end format on end TableOfContents md\"\"\" Interactive performance tuning with Julia note \"Thread pinning\" Remember to pin the threads of your programs This notebook runs on a Slurm cluster and uses `ThreadPinning.jl` https github.com carstenbauer ThreadPinning.jl to pin the Julia process to the CPU threads allocated by Slurm by using the `SLURM CPU BIND LIST` environment variable. Part of this presentation will use material from the workshop Julia for HPC UCL 2024 https github pages.arc.ucl.ac.uk julia hpc 2024 run by Carsten Bauer , the course material is available at carstenbauer JuliaUCL24 https github.com carstenbauer JuliaUCL24 . \"\"\" md\"\"\" warn This notebook uses and LIKWID & LinuxPerf.jl, which requires Linux \"\"\" using ThreadPinning ThreadPinning.pinthreads Int. log2. parse. BigInt, first split ENV \"SLURM CPU BIND LIST\" , ',' , Threads.nthreads warn false ThreadPinning.pinthreads cores with terminal ThreadPinning.threadinfo using CpuId cpuinfo Sys.CPU NAME md\"\"\" Simple linear algebra \"\"\" using LIKWID using LinuxPerf using PlutoUI using Chairmarks begin a T pi x rand T, N y rand T, N z zeros T, N end md\"\"\"T bind T Select Float32, Float64 \"\"\" let range trunc. Int, exp10. 0 0.2 5 md\"N bind N Slider range, default 10 000, show value true \" end function axpy z, a, x, y for idx in eachindex z, x, y z idx a x idx y idx end end b axpy z, a, x, y md\"\"\" We can use `LIKWID.jl` https github.com JuliaPerf LIKWID.jl to measure the number of floating point operations performed in the `axpy ` function details below 👇 . \"\"\" N FLOPs first events perf group \"RETIRED SSE AVX FLOPS ALL\" N FLOPs per iteration N FLOPs N with terminal code llvm debuginfo none axpy z, a, x, y with terminal code native debuginfo none axpy z, a, x, y perf group T Float32 ? \"FLOPS SP\" \"FLOPS DP\" metrics, events perfmon perf group axpy z, a, x, y first metrics perf group md\"\"\"We can also access Linux's `perf` https perf.wiki.kernel.org index.php Main Page performance counters via `LinuxPerf.jl` https github.com JuliaPerf LinuxPerf.jl .\"\"\" measure axpy z, a, x, y perf events \" cache references,cache misses \" pstats perf events axpy z, a, x, y md\"\"\" Parallelising the `sum` ...not so fast. \"\"\" data rand 1 000 000 Threads.nthreads sum data b sum data using Base.Threads using ChunkSplitters function sum threads chunks data nchunks nthreads psums zeros eltype data , nchunks threads for c, idcs in enumerate chunks data n nchunks for i in idcs psums c data i end end return sum psums end sum data ≈ sum threads chunks data b sum threads chunks data pstats perf events sum data pstats perf events sum threads chunks data md\"High cache trashing frequency false sharing https en.wikipedia.org wiki False sharing https d3i71xaburhd42.cloudfront.net cb149ebdbe097867a3b307a0f8d24c5867c0aa68 19 Figure4 1.png Good solution Create a new local accumulator inside the `for` loop, and update the per chunk accumulator at the end of the inner loop.\" function sum threads chunks local data nchunks nthreads psums zeros eltype data , nchunks threads for c, idcs in enumerate chunks data n nchunks local s zero eltype data for i in idcs s data i end psums c s end return sum psums end sum data ≈ sum threads chunks local data b sum threads chunks local data pstats perf events sum threads chunks local data md\"\"\" Better solution Don't do the manual accumulation, use tasks. \"\"\" function sum map spawn data nchunks nthreads ts map chunks data, n nchunks do idcs spawn views sum data idcs end return sum fetch. ts end sum data ≈ sum map spawn data b sum map spawn data pstats perf events sum map spawn data let range trunc. Int, exp10. 0 0.2 9 md\"\"\" large\\ N bind large N Slider range, default 10, show value true \"\"\" end "},{"url":"mod3_parallelism/shared-memory/","title":"Shared-memory parallelism","tags":["module3","track_parallel"],"text":" A Pluto.jl notebook v0.20.13 frontmatter chapter \"3\" section \"1\" order \"8\" title \"Shared memory parallelism\" date \"2025 06 04\" tags \"module3\", \"track parallel\" layout \"layout.jlhtml\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils begin using PlutoUI, PlutoTeachingTools using PlutoUI Slider PlutoUI.TableOfContents depth 4 end begin using CairoMakie set theme theme latexfonts fontsize 16, Lines linewidth 2, , markersize 16 end ChooseDisplayMode md\"\"\" Shared Memory parallelism This notebook uses Threads.nthreads threads \"\"\" with terminal do Sys.cpu summary end using Hwloc with terminal do topology info end md\"\"\" CPU architecture \"\"\" md\"\"\" In Introduction to Parallelism .. mod1 introduction parallelism we briefly touched on computer architecture. On of the more peculiar issues are so called Hyperthreads HT . Hyperthread is the Intel brand name for Simultaneous multithreading SMT . SMT N provides N logical CPU cores per physical CPU core. Some platforms may even make N configurable. On Intel and AMD systems N 2, but systems with higher N are available. The core idea is that memory accesses are slow and superscalar CPUs can have many instructions in flight at once. The CPU architecture exposes a limited amount of resources registers that can be used, but the actual implementation in hardware may have an abundance of resources. This is can be used to execute operations can be speculativly executed or we can execute multiple threads on the same physical core. note For our purposes the distinction can be mostly ignored, but we should not expect perfect scaling when exceeding the number of physical cores. \"\"\" using ThreadPinning with terminal do threadinfo end md\"\"\" Julia tasks In Julia concurrent and parallel programming is done through tasks . Tasks are independent and communication units of computation. They execute always concurrently, but not always parallel. info \"Further reading\" https docs.julialang.org en v1 manual asynchronous programming https proceedings.juliacon.org papers 10.21105 jcon.00054 \"\"\" import Base.Threads sync, spawn function fib x if x 1 return 1 else b spawn fib x 2 a fib x 1 return a fetch b Int end end fib 5 md\"\"\" note A core principle at play is the notion of divide and conquer. Can I split a problem into smaller sub problems? Often, in parallel computing we split problems until a grain size is reached for which we execute a base case. Today's exercise \"Parallel sorting\" is an example of divide and conquer \"\"\" md\"\"\" question The Fibonacci function above is just an example to demonstrate basic ideas of parallel programming. How would you implement it if you were interested in getting the result as fast as possible? \"\"\" md\"\"\" Channels \"\"\" md\"\"\" As mentioned Julia tasks are communicating , communication can happen with dedicated programming concepts like `Channel`, or directly through memory shared with another tasks. Channels are first in first out queues that can either be buffered e.g. contain a reservoir for a number of elements , or un buffered blocking. Julia's channels support `put `, `take ` \"\"\" let ch Channel Int Inf buffered sync begin for i in 1 10 spawn put ch, rand Int end end close ch Otherwise collect will wait for more data collect ch end md\"\"\" Race conditions `Channel` is a concurrent data structure and ensure that it safe to use with multiple tasks. When we use our own data structures we have to make sure that we make them safe if necessary, otherwise we will observe data races. \"\"\" mutable struct BrokenCounter x Int end let a BrokenCounter 0 N 10 K 100 000 sync for i in 1 N spawn for i in 1 K a.x 1 GC.safepoint end end a.x, N K, a.x N K end md\"\"\" note We launch `N` tasks each performing `K` updates to our counter. We expect `N K` updates, but we only observe a fraction of them. ```julia a.x 1 Can be written as a.x a.x 1 ``` But in the time between reading and writing `a.x`, there might have been another update. This is a classical race conditions. \"\"\" md\"\"\" Atomics & Locks One way this can be fixed is to use atomics. Atomics allow to express the `read and increment` operation as one operation. info \"Further reading\" https marabos.nl atomics https redixhumayun.github.io systems 2024 01 03 atomics and concurrency.html \"\"\" mutable struct AtomicCounter atomic x Int end let a AtomicCounter 0 N 10 K 100 000 sync for i in 1 N spawn for i in 1 K atomic a.x 1 GC.safepoint end end a.x, N K, a.x N K end let a AtomicCounter 0 a.x 10 end let a AtomicCounter 0 atomic sequentially consistent a.x 1 2 end md\"\"\" We can also fix this using a lock. note `Lockable` was added in Julia 1.11. \"\"\" let a Base.Lockable BrokenCounter 0 , Base.ReentrantLock N 10 K 100 000 sync for i in 1 N spawn for i in 1 K lock a, a .x 1 GC.safepoint end end lock a begin a .x, N K, a .x N K end end md\"\"\" Atomics are limited to things that are \"small\" For fields Julia might put a lock variable next to the field, if the value is \"big\" . And locks are an easier way to provide access to things that should be exclusive access, but are more expensive than atomics. \"\"\" md\"\"\" Atomics on Arrays Not yet part of the base language. Use the package `Atomix`. \"\"\" using Atomix Atomix, atomic, atomicswap, atomicreplace let A ones Int, 3 Atomix. atomic A 1 1 fetch and increment A end md\"\"\" Parallel loops \"\"\" import Base.Threads threads, nthreads, threadid let a zeros Int, nthreads 2 threads for i in 1 length a a i threadid end a end md\"\"\" note `threadid` is a implementation detail, and should not be used as part of algoritms. Tasks migrate across threads, so it is not a stable observation. \"\"\" md\"\"\" Schedulers Julia has diffferent schedulers for parallel for loops ` dynamic` the default . Chunks the iteration space. ` greedy` One task per thread, good for unequal workloads. Iteration space is interpreted as a channel. ` static` One task per thread, equal division of iteration space. Can not be nested. \"\"\" md\"\"\" Parallel primitives \"\"\" using OhMyThreads md\"\"\" While ` threads` is on the surface an acceptable interface, it is often cumbersome to implement reductions. There are several libraries that provide high level parallel primitives based on higher order functions functions that take other functions . `map f, A ` `reduce , A ` `mapreduce f, , A ` `OhMyThreads.jl` provides `tmap` `treduce` `tmapreduce` `tforeach` note It is a bit unfortunate that `map` & co are implicitly parallel on the GPU, but not on the CPU. Fixing this as been a long standing ToDo. \"\"\" let a zeros Int, nthreads 2 tforeach 1 length a do i a i threadid end a end md\"\"\" False sharing Based on the OhMyThreads.jl docs https juliafolds2.github.io OhMyThreads.jl stable literate falsesharing falsesharing . \"\"\" using BenchmarkTools data rand 1 000 000 nthreads md\"\"\" Baseline sequential sum \"\"\" function simple sum data acc zero eltype data for i in eachindex data acc data i end acc end benchmark sum data benchmark simple sum data question box md\"\"\" Note that our simple sum is slower than Julia's sum Why might that be the case? \"\"\" md\"\"\" Naive parallel implementation We allocate space for the intermediate results. \"\"\" function parallel sum falsesharing data nchunks nthreads psums zeros eltype data , nchunks sync for c, idcs in enumerate OhMyThreads.index chunks data n nchunks spawn begin for i in idcs psums c data i end end end return sum psums end sum data ≈ parallel sum falsesharing data benchmark parallel sum falsesharing data md\"\"\" Oof our parallel code is slower than our serial code The core issue is that we directly update the values in memory and this causes the CPU to bounce the cache lines from core to core \"\"\" md\"\"\" Padding to cache line size One way of solving this is to over allocate memory and pad each sum such that it is occuring on different cache lines. `std hardware destructive interference size` https en.cppreference.com w cpp thread hardware destructive interference size.html \"\"\" const CACHE LINE SIZE 64 function parallel sum padded data nchunks nthreads pad each entry stride CACHE LINE SIZE ÷ sizeof eltype data psums zeros eltype data , nchunks stride sync for c, idcs in enumerate OhMyThreads.index chunks data n nchunks spawn begin c idx c 1 stride 1 for i in idcs psums c idx data i end end end return sum psums end benchmark parallel sum padded data md\"\"\" note To make our implementation faster, we should first think about how we could make our `simple sum` implementation faster. \"\"\" md\"\"\" Task local parallel summation Another way of solving this is to do a local reduction first \"\"\" function parallel sum tasklocal data nchunks nthreads psums zeros eltype data , nchunks sync for c, idcs in enumerate OhMyThreads.index chunks data n nchunks spawn begin local s zero eltype data for i in idcs s data i end psums c s end end return sum psums end benchmark parallel sum tasklocal data md\"\"\" Reuse efficient local implementation \"\"\" function parallel sum map data nchunks nthreads psums zeros eltype data , nchunks sync for c, idcs in enumerate OhMyThreads.index chunks data n nchunks spawn begin psums c sum view data, idcs end end return sum psums end benchmark parallel sum map data md\"\"\" Of course we can just use `OhMyThreads.treduce` \"\"\" benchmark treduce , data ntasks nthreads "},{"url":"mod4_automatic_differentiation/ad_in_science/","title":"Examples of Automatic Differentiation in Scientific Computing","tags":["module4","track_ad"],"text":" A Pluto.jl notebook v0.20.13 frontmatter chapter \"4\" section \"2\" order \"12\" title \"Examples of Automatic Differentiation in Scientific Computing\" date \"2025 07 02\" tags \"module4\", \"track ad\" layout \"layout.jlhtml\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils This Pluto notebook uses bind for interactivity. When running this notebook outside of Pluto, the following 'mock version' of bind gives bound variables a default value instead of an error . macro bind def, element format off return quote local iv try Base.loaded modules Base.PkgId Base.UUID \"6e696c72 6542 2067 7265 42206c756150\" , \"AbstractPlutoDingetjes\" .Bonds.initial value catch b missing end local el esc element global esc def Core.applicable Base.get, el ? Base.get el iv el el end format on end begin using PlutoUI, PlutoTeachingTools using PlutoUI Slider PlutoUI.TableOfContents depth 4 end begin using CairoMakie set theme theme latexfonts fontsize 16, Lines linewidth 2, , markersize 16 end begin using Enzyme using Enzyme Reverse import ForwardDiff end using Symbolics using ShortCodes using SparseArrays using LinearAlgebra using Krylov using SparseConnectivityTracer begin using SparseMatrixColorings import Images end md\"\"\" Motivation In science and engineering we often need to solve systems of equations . If the equations are linear then linear algebra tells us a general method to solve them these are now routinely applied to solve systems of millions of linear equations. If the equations are nonlinear then things are less obvious. The main solution methods we know work by... reducing the nonlinear equations to a sequence of linear equations They do this by approximating the function by a linear function and solving that to get a better solution, then repeating this operation as many times as necessary to get a sequence of increasingly better solutions. This is an example of an iterative algorithm . A well known and elegant method, which can be used in many different contexts, is the Newton method . It does, however, have the disadvantage that it requires derivatives of the function. This can be overcome using automatic differentiation techniques. \"\"\" md\"\"\" The Newton method in 1D based of https featured.plutojl.org computational thinking newton.html \"\"\" md\"\"\" We would like to solve equations like f x g x . We rewrite that by moving all the terms to one side of the equation so that we can write h x 0 , with h x f x g x . A point x^ such that h x^ 0 is called a root or zero of h . The Newton method finds zeros, and hence solves the original equation. \"\"\" md\"\"\" The idea of the Newton method is to follow the direction in which the function is pointing We do this by building a tangent line at the current position and following that instead, until it hits the x axis. Let's look at that visually first \"\"\" md\"\"\" n bind n2 Slider 0 10, show value true, default 1 \"\"\" md\"\"\" x₀ bind x02 Slider 10 10, show value true, default 6 \"\"\" md\"\"\" n bind n Slider 0 10, show value true, default 1 \"\"\" md\"\"\" x₀ bind x0 Slider 10 10, show value true, default 6 \"\"\" straight x0, y0, x, m y0 m x x0 function standard Newton f, n, x range, x0, ymin 10, ymax 10 f′ x ForwardDiff.derivative f, x fig Figure size 400, 300 ax Axis fig 1,1 ylims ax, ymin, ymax lines ax, x range, f hlines hlines ax, 0.0, color magenta, linestyle dash lw 3 scatter ax, x0, 0, color green annotation ax, x0, 5, text L\"x 0\" for i in 1 n lines ax, x0, x0 , 0, f x0 , color gray, alpha 0.5 scatter ax, x0, f x0 , color red m f′ x0 lines ax, x range, straight x0, f x0 , x, m for x in x range , color blue, alpha 0.5, linestyle dash lw 2 x1 x0 f x0 m scatter ax, x1, 0, color green , ann x1, 5, L\"x % i\", 10 annotation ax, x1, 5, text L\"x % i\" x0 x1 end fig end let f x x^2 2 standard Newton f, n2, 1 0.01 10, x02, 10, 70 end let f x 0.2x^3 4x 1 standard Newton f, n, 10 0.01 10, x0, 10, 70 end md\"\"\" Mathematics of the Newton method \"\"\" md\"\"\" We can convert the idea of \"following the tangent line\" into equations as follows. You can also do so by just looking at the geometry in 1D, but that does not help in 2D. \"\"\" md\"\"\" Suppose we have a guess x 0 for the root and we want to find a hopefully better guess x 1 . Let's set x 1 x 0 \\delta , where x 1 and \\delta are still unknown. We want x 1 to be a root, so \"\"\" md\"\"\" f x 1 f x 0 \\delta \\simeq 0 \"\"\" md\"\"\" If we are already \"quite close\" to the root then \\delta should be small, so we can approximate f using the tangent line f x 0 \\delta \\, f' x 0 \\simeq 0 and hence \\delta \\simeq \\frac f x 0 f' x 0 so that x 1 x 0 \\frac f x 0 f' x 0 Now we can repeat so that x 2 x 1 \\frac f x 1 f' x 1 and in general x n 1 x n \\frac f x n f' x n . This is the Newton method in 1D. \"\"\" function newton1D f, x0 f′ x ForwardDiff.derivative f, x sequence x0 x x0 for i in 1 10 x f x f′ x end return x end newton1D x x^2 2, 37.0 sqrt 2 md\"\"\" Extending Netwon to 2D T \\mathbb R ^2 \\to \\mathbb R ^2 \"\"\" md\"\"\" We want to find the inverse T^ 1 y , i.e. to solve the equation T x y for x . We use the same idea as in 1D, but now in 2D \"\"\" md\"\"\" T x 0 \\delta \\simeq 0 T x 0 J \\cdot \\delta \\simeq 0, where J DT x 0 is the Jacobian matrix of T at x 0 , i.e. the best linear approximation of T near to x 0 . \"\"\" md\"\"\" Hence \\delta is the solution of the system of linear equations \"\"\" md\"\"\" J \\cdot \\delta T x 0 Then we again construct the new approximation x 1 as x 1 x 0 \\delta . \"\"\" function newton2D step T, x J ForwardDiff.jacobian T, x δ J \\ T x J^ 1 T x return x δ end \"Find ``x`` such that ``T x 0``\" function newton2D T, x0, n 10 x x0 for i in 1 n x newton2D step T, x end return x end md\"\"\" Remember that Newton is designed to look for roots , i.e. places where T x 0 . We want T x y , so we need another layer \"\"\" begin \"Looks for ``x`` such that ``f x y``, i.e. ``f x y 0``\" function inverse f, y, x0 0, 0 return newton2D x f x y, x0 end inverse f y inverse f, y end T α x, y , x α y^2, y α x^2 md\"\"\" α bind α Slider 0.0 0.01 1.0, show value true \"\"\" T α 0.3, 0.4 T α ∘ inverse T α 0.3, 0.4 Essentially I v md\"\"\" Jacobian f \\mathbb R ^n\\rightarrow\\mathbb R ^m ```math J \\begin bmatrix \\frac \\partial f 1 \\partial x 1 & \\cdots & \\frac \\partial f 1 \\partial x n \\\\ 1ex % 1ex more space between rows of matrix \\vdots & \\ddots & \\vdots \\\\ 1ex \\frac \\partial f m \\partial x 1 & \\cdots & \\frac \\partial f m \\partial x n \\end bmatrix ``` \"\"\" md\"\"\" Let's take a function `F` from \\mathbb R ^2\\rightarrow\\mathbb R ^3 as a concrete example. \"\"\" function F X X 1 ^4 3 exp X 2 2 log X 1 X 2 ^2 end md\"\"\" Manual derivation Symbolics \"\"\" begin variables X 2 ex F F X ex J Symbolics.jacobian ex F, X scalarize false end md\"\"\" note Above we have the general form of the Jacobian of `F`, but more often than not we care about the Jacobian of `F` at a point `x`. \"\"\" substitute. ex J, X 1.0, 1.0 function F2 X X 1 ^4 3 X 2 0 ? X 2 exp X 2 2 log X 1 X 2 ^2 end md\"\"\" note When we have functions with control flow that depend on an input, the symbolic approach fails. \"\"\" begin ex F2 F2 X ex J2 Symbolics.jacobian ex F2, X scalarize false end md\"\"\" Using ForwardDiff \"\"\" ForwardDiff.jacobian F, 1.0, 1.0 ForwardDiff.jacobian F2, 1.0, 1.0 md\"\"\" note Automatic differentiation always evaluates things at a point, we can ask for the Jacobian of a function with input dependent control flow. \"\"\" ForwardDiff.jacobian F, 1.0, 1.0 ForwardDiff.jacobian F2, 1.0, 1.0 md\"\"\" Using Enzyme \"\"\" Enzyme.jacobian Forward, F, 1.0, 1.0 | only Enzyme.jacobian Reverse, F, 1.0, 1.0 | only md\"\"\" note Both Jacobian are equivalent, but the object computed by Reverse mode is the transpose. \"\"\" let J Enzyme.jacobian Reverse, F, 1.0, 1.0 | only v 1.0, 0.0 J v end md\"\"\" Mutation \"\"\" begin function G Y, X Y 1 X 1 ^4 3 Y 2 exp X 2 2 Y 3 log X 1 X 2 ^2 end function G X Y similar X, 3 G Y, X Y end end ForwardDiff.jacobian G, 1.0, 1.0 ForwardDiff.jacobian zeros 3, 2 , G , zeros 3 , 1.0, 1.0 J Enzyme.jacobian Reverse, G, 1.0, 1.0 | only md\"\"\" Jacobian vector product \"\"\" md\"\"\" The Jacobian vector product or directional derivative is a powerful primitive. We implicitly perform a Jv operation at the point u . With Enzyme this is a direct application of `autodiff` in Forward mode. \"\"\" function JVP f F, u, v, ϵ NaN where F return only Enzyme.autodiff Forward, f, Duplicated, Duplicated u, v end md\"\"\" Approximation In many texts you will find the directional derivatives approximated using a finite difference operation. \"\"\" md\"\"\" First order finite difference \\frac F u \\epsilon \\cdot v F u \\epsilon \"\"\" First order Taylor series function JVP Finite Diff 1st F, u, v, ϵ sqrt eps F u ϵ . v F u . ϵ end md\"\"\" Second order finite difference \\frac F u \\epsilon \\cdot v F u \\epsilon \\cdot v 2\\epsilon \"\"\" Second order Taylor series Central finite difference function JVP Finite Diff 2nd F, u, v, ϵ cbrt eps 2 F u ϵ . v F u ϵ . v . 2ϵ end md\"\"\" Choosing ϵ warn A core challenge is to choose the right ϵ for the problem at hand. Common For 1st order ϵ \\sqrt \\epsilon mach For 2nd order ϵ \\sqrt 3 \\epsilon mach 2 Input dependent For 1st order ϵ \\frac \\sqrt 1 ||u|| \\epsilon mach ||v|| after DOI \"10.1016 j.jcp.2003.08.010\" For 2nd order ϵ \\frac \\sqrt 1 ||u|| ||v|| \\sqrt 3 \\epsilon mach after DOI \"10.1016 j.cam.2011.09.003\" \"\"\" JVP Finite Diff 1st F, 1.0 1.0 , 1.0 0.0 JVP Finite Diff 2nd F, 1.0 1.0 , 1.0 0.0 md\"\"\" Example Truncated Weierstrass \"\"\" md\"\"\" Truncated Weierstrass f x \\sum k 0 ^ N a^k cos b^k\\pi x f' x \\sum k 0 ^ N a^k b^k \\pi sin b^k\\pi x With a \\in 0,1 and 1 ab \"\"\" f x, a 0.3, b 3.0 N 10 sum a^k cos b^k π x for k in 0 N f′ x, a 0.3, b 3.0 N 10 sum a^k b^k π sin b^k π x for k in 0 N let xs LinRange 1.0, 1.0, 10000 fig Figure size 800, 800 lines fig 1, 1 , xs, x f x, N 3 , axis title \"Function N 3\", ylabel L\" f x \", xlabel L\" x \" lines fig 2, 1 , xs, x f x, N 10 , axis title \"Function N 10\", ylabel L\" f x \", xlabel L\" x \" lines fig 1, 2 , xs, x f′ x, N 3 , axis title \"Derivative N 3\", ylabel L\" f' x \", xlabel L\" x \" lines fig 2, 2 , xs, x f′ x, N 10 , axis title \"Derivative N 10\", ylabel L\" f' x \", xlabel L\" x \" fig end function rel error f, f′, x, method, ϵ y′ f′ x y method f, x, one x , ϵ max eps 4, abs y′ y y′ Avoid error 0 to cause issues in log plot end let ϵs 10.0^ x for x in 30 0.1 0 ϵs BigFloat. 10.0^ x for x in 30 0.1 0 x 0.2 fig Figure size 800, 800 ax Axis fig 1,1 , xscale log10, yscale log10, title \"Relative error N 3\", ylabel \"Relative error\", xlabel \"Difference step size\" ylims ax, 1e 20, 10e1 lines ax, ϵs, ϵ rel error x f x, N 3 , x f′ x, N 3 , x, JVP Finite Diff 1st, ϵ , label \"FFD 1st order\" lines ax, ϵs, ϵ rel error x f x, N 3 , x f′ x, N 3 , x, JVP Finite Diff 2nd, ϵ , label \"CFD 2nd order\" lines ax, ϵs, ϵ rel error x f x, N 3 , x f′ x, N 3 , x, JVP, ϵ , label \"AD\" vlines ax, sqrt eps , label L\" \\sqrt \\epsilon mach \", color \"red\",ymin 1 vlines ax, cbrt eps 2 , label L\" \\cbrt \\epsilon mach 2 \", color \"orange\",ymin 1 axislegend ax, position lt ax Axis fig 2,1 , xscale log10, yscale log10, title \"Relative error N 10\", ylabel \"Relative error\", xlabel \"Difference step size\" ylims ax, 1e 20, 10e1 lines ax, ϵs, ϵ rel error x f x, N 10 , x f′ x, N 10 , x, JVP Finite Diff 1st, ϵ , label \"FFD 1st order\" lines ax, ϵs, ϵ rel error x f x, N 10 , x f′ x, N 10 , x, JVP Finite Diff 2nd, ϵ , label \"CFD 2nd order\" lines ax, ϵs, ϵ rel error x f x, N 10 , x f′ x, N 10 , x, JVP, ϵ , label \"AD\" vlines ax, sqrt eps , label L\" \\sqrt \\epsilon mach \", color \"red\", ymin 1 vlines ax, cbrt eps 2 , label L\" \\cbrt \\epsilon mach 2 \", color \"orange\", ymin 1 axislegend ax, position lt fig end md\"\"\" Extracting a Jacobian \"\"\" ex F question box md\"\"\" Are the zeros here real structural zeros, or are they computational zero? \"\"\" ex J md\"\"\" Inadmissability \"\"\" md\"\"\" note Why is there a NaN? \"\"\" ex F md\"\"\" ``` Base.log x Dual Dual log x.value , x.deriv x.value Base. x Dual, y Dual Dual x.value y.value, x.deriv y.deriv ``` \"\"\" 0.0 0.0 derivative of log x 0.0 0.0 0.0 2.0 md\"\"\" note Finite differences can struggle with \"inadmisable\" values. Recall the definition of `F`. We calculate log X 1 If X 1 0 then the 2nd order method will go a little bit to the left and right of zero. `log 0 eps ` is not defined. \"\"\" ex F md\"\"\" ``` log 0 eps ``` \"\"\" md\"\"\" Using Enzyme generalized w J v ``` autodiff Forward, G , Duplicated out, w , Duplicated in, v ``` w v J ``` autodiff Reverse, G , Duplicated out, copy v , Duplicated in, w ``` note Recall from last week that Enzyme zeros the shadow inputs. Necessitating the `copy v ` \"\"\" md\"\"\" warn That's clearly wrong \"\"\" let out zeros 3 w zeros 2 autodiff Reverse, G , Duplicated out, 0.0,0.0,0.0 , Duplicated 1.0, 1.0 , w w end md\"\"\" Uhm... All the seeds are zero... So why do we have a gradient ? \"\"\" typeof G zeros 3 , 1.0, 1.0 md\"\"\" Due to historic reasons Enzyme inferes function with floating point return as \"active return\" and provides an implicit seed of 1.0. \"\"\" let out zeros 3 w zeros 2 autodiff Reverse, G , Const, Duplicated out, 0.0,0.0,0.0 , Duplicated 1.0, 1.0 , w w end md\"\"\" Sanity restored... \"\"\" md\"\"\" Jacobian Operator Given a problem `F u 0` we say that `F u ` calculates the residual . We want to have an operator `J F, u ` that represents the Jacobian of `F` at `u`. \"\"\" struct JacobianOperator F, A f F F out, in out A in A function JacobianOperator f F, out, in where F return new F, typeof in f, out, in end end begin Base.size J JacobianOperator length J.out , length J.in Base.eltype J JacobianOperator eltype J.in end function assemble jacobian F, u, JVP N length u v zero u out JVP F, u, v M length out J SparseMatrixCSC eltype u , Int undef, M, N for j in 1 N v . 0 v j 1 out JVP F, u, v for i in 1 M if out i 0 J i, j out i end end end J end assemble jacobian F, 1.0 1.0 , JVP assemble jacobian F, 1.0 1.0 , JVP Finite Diff 1st assemble jacobian F, 1.0 1.0 , JVP Finite Diff 2nd assemble jacobian F, 1.0 0.0 , JVP assemble jacobian F, 1.0 0.0 , JVP Finite Diff 1st assemble jacobian F, 1.0 0.0 , JVP Finite Diff 2nd assemble jacobian F, 0.0 1.0 , JVP assemble jacobian F, 0.0 1.0 , JVP Finite Diff 1st assemble jacobian F, 0.0 1.0 , JVP Finite Diff 2nd begin LinearAlgebra.adjoint J JacobianOperator Adjoint J LinearAlgebra.transpose J JacobianOperator Transpose J end let J Enzyme.jacobian Reverse, F, 1.0, 1.0 | only v 1.0, 0.0, 0.0 transpose J v vJ end begin v 1.0, 0.0, 0.0 transpose J v end let out zeros 3 w zeros 2 autodiff Reverse, G , Duplicated out, copy v , Duplicated 1.0, 1.0 , w w end let out zeros 3 w zeros 2 autodiff Reverse, G , Const, Duplicated out, copy v , Duplicated 1.0, 1.0 , w w end function maybe duplicated f, df if Enzyme.Compiler.guaranteed const typeof f return DuplicatedNoNeed f, df else return Const f end end function LinearAlgebra.mul out, J JacobianOperator, v autodiff Forward, maybe duplicated J.f, Enzyme.make zero J.f , Const, Duplicated J.out, out , Duplicated J.in, v return nothing end function LinearAlgebra.mul out, J′ Union Adjoint Any, JacobianOperator , Transpose Any, JacobianOperator , v J parent J′ If out is non zero we get spurious gradients out . 0 autodiff Reverse, maybe duplicated J.f, Enzyme.make zero J.f , Const, Enzyme zeros input derivatives and that confuses the solvers. Duplicated J.out, copy v , Duplicated J.in, out return nothing end function Base.collect JOp JacobianOperator N, M size JOp v zeros eltype JOp , M out zeros eltype JOp , N J SparseMatrixCSC eltype v , Int undef, size JOp ... for j in 1 M out . 0.0 v . 0.0 v j 1.0 mul out, JOp, v for i in 1 N if out i 0 J i, j out i end end end return J end JOp JacobianOperator G , zeros 3 , 1.0, 1.0 let w zeros 3 mul w, JOp, 1.0, 0.0 w end let w zeros 2 mul w, transpose JOp , 1.0, 0.0, 0.0 w end md\"\"\" Newton Krylov F u 0 J F, u \\times x F u Problems of the form F u 0 can be solved with a Newton method, this particular helpful for finding an iterative solution of nonlinear equations. We can use a Krylov solver \"gmres\" to solve the linear system Ax b without \"materializing\" A . This is called a matrix free method. \"\"\" begin function G2 y, x y 1 x 1 ^2 x 2 ^2 2 y 2 exp x 1 1 x 2 ^2 2 end function G2 x y similar x G2 y, x return y end end let xs LinRange 3, 4, 1000 ys LinRange 5, 5, 1000 levels 0.1, 0.25, 0.5 2 10..., 10.0 10 200..., 200 100 4000... fig, ax contour xs, ys, x, y norm G2 x, y levels fig end function newton krylov F , u tol rel 1.0e 6, tol abs 1.0e 12, max niter 50, callback x nothing res similar u F res, u n res norm res callback u tol tol rel n res tol abs iter 0 while n res tol && iter max niter Ax b J JacobianOperator F , res, u x, stats gmres J, res Take a step in the newton direction u . x F res, u n res norm res callback u if isinf n res || isnan n res error \"Solver blew up\" end iter 1 end return u, solved n res tol, iter end trace 1 let x₀ 2.0, 0.5 xs Vector Tuple Float64, Float64 undef, 0 hist x push xs, x 1 , x 2 nothing newton krylov G2 , x₀, callback hist xs end trace 2 let x₀ 2.5, 3.0 xs Vector Tuple Float64, Float64 undef, 0 hist x push xs, x 1 , x 2 nothing newton krylov G2 , x₀, callback hist xs end G2 trace 2 end ... let xs LinRange 3, 4, 1000 ys LinRange 5, 5, 1000 levels 0.1, 0.25, 0.5 2 10..., 10.0 10 200..., 200 100 4000... fig, ax contour xs, ys, x, y norm G2 x, y levels lines ax, trace 1 lines ax, trace 2 fig end md\"\"\" Coloring \"\"\" md\"\"\" Matrix free methods are powerful especially for systems that are 1. Large 2. Unkown structure of the Jacobian But special care is needed to make sure that the number of Jacobian Vector product evaluations is low. In particular for Netwon Krylov one has to use inexact Newton Krylov with an Eisenstatt Walker condition. Matrix coloring is an alternative to reduce the cost of calculating the Jacobian at each iteration. \"\"\" md\"\"\" note Adrian Hill and Guillaume Dalle have done some great work on this in the Julia AD landscape. 1. https iclr blogposts.github.io 2025 blog sparse autodiff 2. https arxiv.org abs 2501.17737 warn Remember the issues we had earlier with \"structural sparsisty\" and \"input dependent computation\". These issues are particularly perniscous for automatic sparsity detection. \"\"\" jacobian sparsity F, 1.0, 1.0 , TracerSparsityDetector jacobian sparsity F, 1.0, 0.0 , TracerSparsityDetector jacobian sparsity F, 1.0, 0.0 , TracerLocalSparsityDetector jacobian sparsity x x 1 x 2 , 1.0, 0.0 , TracerLocalSparsityDetector jacobian sparsity x x 1 x 2 , 0.0, 1.0 , TracerLocalSparsityDetector jacobian sparsity x x 1 x 2 , 1.0, 1.0 , TracerLocalSparsityDetector jacobian sparsity x x 1 x 2 , 0.0, 0.0 , TracerSparsityDetector jacobian sparsity F2, 1.0, 1.0 , TracerSparsityDetector jacobian sparsity F2, 1.0, 1.0 , TracerLocalSparsityDetector md\"\"\" Recall the diffuse example from the beginning of term. \"\"\" function heat 1D du, u, a, Δx , t N length u Enforce the boundary condition u 1 0 u end 0 du 1 0 du end 0 Only compute within for i in 2 N 1 du i a u i 1 2u i u i 1 Δx^2 end return end function heat 1D u du similar u heat 1D du, u, 0.01, 0.01 , 0.01 return du end md\"\"\" What is it's Jacobian? \"\"\" ForwardDiff.jacobian heat 1D, ones 12 assemble jacobian heat 1D, ones 12 , JVP J heat 1D sparsity jacobian sparsity heat 1D, ones 12 , TracerSparsityDetector result coloring J heat 1D sparsity, ColoringProblem , GreedyColoringAlgorithm SparseMatrixColorings.show colors result 1 ncolors result column colors result function assemble jacobian with colors F, u, JVP sparsity jacobian sparsity F, u, TracerSparsityDetector result coloring sparsity, ColoringProblem , GreedyColoringAlgorithm N length u v zero u out JVP F, u, v M length out J Matrix eltype u undef, M, ncolors result for c in 1 ncolors result v . 0 for j in 1 N if c column colors result j v j 1 end end J ,c JVP F, u, v end decompress J, result end assemble jacobian with colors heat 1D, ones 12 , JVP md\"\"\" note Instead of doing `length u ` JVPs, we are only doing `ncolors`. As long as the structure of our Jacobian is stable we can reuse the color information to extract it efficiently. \"\"\" "},{"url":"mod4_automatic_differentiation/lagrangian_ode/","title":"Lagrangian","tags":["module4","track_performance","indepth"],"text":" A Pluto.jl notebook v0.20.8 frontmatter chapter \"4\" section \"2\" order \"12.1\" title \"Lagrangian\" tags \"module4\", \"track performance\", \"indepth\" layout \"layout.jlhtml\" indepth number \"4\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils begin using PlutoUI using CairoMakie, Enzyme, OrdinaryDiffEqTsit5, RecursiveArrayTools using LinearOperators, Krylov import Enzyme Reverse end versioninfo md\"\"\" Acknowledgement Based on material from Simone Chiocchetti \"\"\" md\"\"\" The Euler Lagrange equations Given L kinetic\\ energy potential\\ energy of the system, the Euler Lagrange equations of motion are, by definition of \\dot q \\frac d dt q \\dot q \\frac dL dq \\frac d dt \\frac dL d\\dot q 0 the equation of motion can be cast by chain rule as \\frac dL dq \\frac d \\dot q \\frac dL d\\dot q \\frac d\\dot q dt 0 \\frac dL dq \\frac d^2L d\\dot q ^2 \\frac d\\dot q dt 0 \\frac dL dq H \\frac d\\dot q dt 0 which gives \\frac d\\dot q dt H^ 1 \\frac dL dq \"\"\" md\"\"\" motivation of course this will anyways cost more than actually using a precomputed rhs which does not require inverting the hessian for every rhs evaluation , but sometimes some crazy systems are lightweight enough, yet complicated enough to differentiate by hand that just implementing the lagrangian and forgetting about everything else is a nice bonus. generally writing down the lagrangian of many physical mechanical systems is absolutely trivial, while writing down the equations of motion is a huge amount of work to be done by hand or sympy \"\"\" md\"\"\" First example Elastic rope first example problem an elastic rope made from N point masses attached by springs \"\"\" function lagrangian elastic rope q, qdot, p N length q m, k, g p kinetic energy is just 0.5 mass velocity^2 Eₖ 0.5 sum m . qdot.^2 potential energy is the elastic energy of the springs plus the gravitational potential Eₚ k 1 q 1 ^2 for i in 2 N Eₚ k i q i q i 1 ^2 end Eₚ 0.5 Eₚ g sum m . q return Eₖ Eₚ lagrangian kinetic energy potential energy end function parameters elastic rope N 3 a helper to setup the parameters of the problem N min N, 5 time domain t0 0 tend 50 initial condition q0 1.0, 2.0, 3.0, 4.0, 5.0 qdot0 0.2, 0.3, 0.7, 0.4, 0.5 q0, qdot0 q0 1 N , qdot0 1 N parameters m 1.1, 1.2, 1.3, 1.4, 1.5 k 1.5, 1.3, 1.7, 1.9, 2.1 g 1. m, k m 1 N , k 1 N params m, k, g return params, q0, qdot0, t0, tend end function source reference elastic rope S, Q, p, t what the user would have to implement instead of lagrangian elastic rope to solve the ode source rhs of the ODE q, qdot Q m, k, g p N length q S.q . qdot S.qdot 1 k 2 q 2 k 1 k 2 q 1 for i in range 1, N 1 S 1 i k i q i 1 k i 1 q i 1 k i k i 1 q i S 2 N 1 k N 1 q N 2 q N 1 S N S N m g return S end end md\"\"\" Second Example Celestial Mechanics \"\"\" function lagrangian celestial mechanics q, qdot, params the lagrangian for the celestial mechanics system, this is the function that the user will implement note that it just computes the scalar value of the lagrangian of the system m, g params N length q ÷ 2 x view q, 1 N y view q, N 1 2N u view qdot, 1 N v view qdot, N 1 2N x, y q u, v qdot Eₖ 0.5 sum m . u.^2 . v.^2 kinetic energy is mass times velocity^2 2 potential energy is the sum of the gravitational potentials of all planets wrt all other planets the planet itself can be summed too, just use r r 2 eps as a division operator Eₚ 0.0 for i in 1 N rq x . x i .^2 . y . y i .^2 Eₚ 0.5 g m i sum m . sqrt. rq . rq . 1.0e 28 end return Eₖ Eₚ lagrangian kinetic energy potential energy end function parameters celestial mechanics N 4 a helper to setup the parameters of the problem N min N, 4 time domain t0 0.0 tend 1.0e1 parameters m 1000.0, 100.0, 10.0, 1.0 m m 1 N g 1.0 initial condition x0 0.0, 0.8, 3.7, 5.9 y0 0.0, 0.1, 0.2, 0.3 r0q x0.^2 . y0.^2 v0 sqrt. m 1 . g . sqrt. r0q . r0q . 1.0e 20 u0 0.0, 0.4, 0.5, 0.6 . v0 x0, y0, u0, v0 x0 1 N , y0 1 N , u0 1 N , v0 1 N v0 1 sum m . v0 m 1 u0 1 sum m . u0 m 1 q0 NamedArrayPartition x x0, y y0 qdot0 NamedArrayPartition u u0, v v0 q0 vcat x0, y0 qdot0 vcat u0, v0 params m, g return params, q0, qdot0, t0, tend end , q0, parameters celestial mechanics def source reference celestial mechanics t, Q, params what the user would have to implement instead of lagrangian celestial mechanics to solve the ode source rhs of the ODE N len Q 4 m, g params N , params 1 S np.zeros 4 N q, qdot Q 2 N , Q 2 N x, y q N , q N S 2 N qdot for i in range N irtot m x x i 2 y y i 2 1.5 1.0e 40 S 2 N i g np.sum x x i irtot S 3 N i g np.sum y y i irtot return S md\"\"\" Machinery \"\"\" md\"\"\" We remember that we want to solve \\frac d\\dot q dt H^ 1 \\frac dL dq First we need \\frac dL dq which we can get through automatic differentiation. Particularly efficent is in this case reverse mode. \"\"\" Note on 1.11 there is currently an Enzyme bug with broadcasting https github.com EnzymeAD Enzyme.jl issues 2397 static if VERSION v\"1.11\" function dLdq L, q, qdot, p dq Enzyme.make zero q autodiff Reverse, Const L , Active, Duplicated q, dq , Const qdot , Const p dq end else function dLdq L, q, qdot, p dq Enzyme.make zero q autodiff Reverse, Const L , Active, Duplicated q, dq , Duplicated qdot, Enzyme.make zero qdot , Duplicated p, Enzyme.make zero p dq end end md\"\"\" Secondly we need to solve H^ 1 \\frac dL dq . H is the Hessian and as such we want to do `H \\ b` where b \\frac dL dq . `manual dldqdotqdot` is using finite difference to take the 2nd derivative of L w.r.t \\dot q `dLdqdotdqdot explicit` uses automatic differentiation to perform forward over reverse mode `dLdqdotdqdot matrix free` constructs a matrix free variant that can be used by `Krylov.jl` to solve `H\\b`. \"\"\" function manual dLdqdotdqdot L, q, qdot, params, epsilon1 1.0e 4, epsilon2 1.0e 4 the lagrangian is a function of the lagrangian coordinates q and of the derivatives of the lagrangian coordinates qdot treated as independent variables this function computes the hessian of the lagrangian with respect to qdot brutally using finite differences N length qdot H zeros N, N f zeros 2, 2 for i in 1 N for j in 1 i for ii in 1 2 for jj in 1 2 qdotk copy qdot qdotk i qdotk i 2 ii 1 epsilon1 qdotk j qdotk j 2 jj 1 epsilon2 f ii,jj L q, qdotk, params end end H i,j f 1,1 f 1,2 f 2,1 f 2,2 4.0 epsilon1 epsilon2 end end for i in 1 N for j in i 1 N H i,j H j,i end end return H end function dLdqdotdqdot explicit L, q, qdot, p N length qdot v Enzyme.make zero qdot w similar v H zeros eltype qdot , N, N for i in 1 N v . 0 v i 1 Enzyme.hvp w, Const qdot L q, qdot, p , qdot, v H , i . w end H end function dLdqdotdqdot matrix free L, q, qdot, p N length qdot LinearOperator eltype qdot , N, N, false, false, w, v Enzyme.hvp w, Const qdot L q, qdot, p , qdot, v , nothing, nothing end function source from lagrangian L function source S, Q, p, t q, qdot Q N length q S.q . qdot S 1 N . qdot b dLdq L, q, qdot, p H dLdqdotdqdot matrix free L, q, qdot, p H dLdqdotdqdot explicit L, q, qdot, p x, stats gmres H, b assert stats.solved S.qdot . x S N 1 end . H \\ b return end end f elastic rope source from lagrangian lagrangian elastic rope sol elastic rope let p, q0, qdot0, t0, tend parameters elastic rope u0 NamedArrayPartition q q0, qdot qdot0 prob ODEProblem f elastic rope, u0, t0, tend , p solve prob, Tsit5 end let sol sol elastic rope fig Figure ts sol.t N length first sol.u .q for i in 1 N ax Axis fig 1,i , xlabel \"t\", ylabel L\"q\" ax2 Axis fig 2,i , xlabel \"t\", ylabel L\"\\dot q \" ax3 Axis fig 3,i , xlabel \"q\", ylabel L\"\\dot q \" q map j sol.u j .q i , 1 length ts qdot map j sol.u j .qdot i , 1 length ts lines ax, ts, q lines ax2, ts, qdot lines ax3, q, qdot end fig end f celestial mechanics source from lagrangian lagrangian celestial mechanics sol celestial mechanics let p, q0, qdot0, t0, tend parameters celestial mechanics u0 NamedArrayPartition q q0, qdot qdot0 prob ODEProblem f celestial mechanics, u0, t0, tend , p solve prob, Tsit5 end let sol sol celestial mechanics fig Figure ts sol.t N length first sol.u .q ÷ 2 ax Axis fig 1,1 , xlabel L\"x\", ylabel L\"y\" for i in 1 N x map j sol.u j .q i , 1 length ts y map j sol.u j .q N i , 1 length ts lines ax, x, y end fig end "},{"url":"mod4_automatic_differentiation/machine_learning/","title":"Automatic Differentiation and Machine Learning","tags":["module4","track_ad"],"text":" A Pluto.jl notebook v0.20.13 frontmatter chapter \"4\" section \"1\" order \"11\" title \"Automatic Differentiation and Machine Learning\" date \"2025 06 18\" tags \"module4\", \"track ad\" layout \"layout.jlhtml\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils begin using PlutoUI, PlutoTeachingTools using PlutoUI Slider PlutoUI.TableOfContents depth 4 end begin using CairoMakie set theme theme latexfonts fontsize 16, Lines linewidth 2, , markersize 16 end using BenchmarkTools, LinearAlgebra using Lux, Random, Optimisers begin ENV \"DATADEPS ALWAYS ACCEPT\" true using MLDatasets MNIST, convert2image using ImageShow end using OneHotArrays, Statistics, MLUtils, Zygote using Printf md\"\"\" Automatic Differentiation and Machine Learning \"\"\" md\"\"\" note Recall our example from lecture 4. \"\"\" begin import ForwardDiff import Enzyme end function m x, a, b, c, d return sin a x b cos c x d end xs 0.0 0.01 2π ys m. xs, 0.3, 1.2, 0.5, 0.7 function mse ŷ, y sum ŷ . y .^2 length y end function dm loss, ys, xs, c ForwardDiff.derivative a loss ys, m. xs, a, c 2 , c 3 , c 4 , c 1 ForwardDiff.derivative a loss ys, m. xs, c 1 , a, c 3 , c 4 , c 2 ForwardDiff.derivative a loss ys, m. xs, c 1 , c 2 , a, c 4 , c 3 ForwardDiff.derivative a loss ys, m. xs, c 1 , c 2 , c 3 , a , c 4 end md\"\"\" note We have to perform one ForwardDiff.derivative call per parameter \"\"\" coeffs guess rand 2.0 0.1 2.0, 4 m. xs, coeffs guess... dm mse, xs, ys, coeffs guess... begin learning rate 0.01 steps 1000 plot every 100 end let fig Figure ax Axis fig 1,1 coeffs rand 4 errs Float64 for i in 1 steps ys m. xs, coeffs... err mse ys, ys push errs, err if mod1 i, plot every 1 lines xs, ys, label \"Epoch i\" end dcoeffs dm mse, ys, xs, coeffs coeffs . learning rate . dcoeffs end ys m. xs, coeffs... err mse ys, ys lines xs, ys, label \"Epoch steps 1 \" lines xs, ys, label \"Goal\" axislegend ax ax2 Axis fig 2,1 lines ax2, errs fig end md\"\"\" Instead of using Forward Mode automatic differentiation we can perform reverse mode AD. \"\"\" function dm rev loss, ys, xs, c dc Enzyme.make zero c function f ys, xs, c loss ys, map x m x, c... , xs end Enzyme.autodiff Enzyme.Reverse, f, Enzyme.Const ys , Enzyme.Duplicated xs, Enzyme.make zero xs , Enzyme.Duplicated c, dc dc end dm rev mse, xs, ys, coeffs guess... dm mse, xs, ys, coeffs guess... md\"\"\" note In one call we can obtain the derivatives of all coefficients This is a great help when the number of parameters grow very large \"\"\" md\"\"\" Reverse Mode \"\"\" md\"\"\" Forward vs. reverse mode AD What we have discussed previously is the basic idea of forward mode AD. It is called this way since the information of the derivatives propagates in the same way as the usual computation. Let's now consider a typical optimization problem, e.g., \\min x \\| A x b \\| 2^2, where A and b are given. To use something like gradient descent, we need to compute the derivative with respect to x , i.e., \\nabla x \\| A x b \\| 2^2 2 A x b ^T A. Let's check this with a simple example. \"\"\" let A 1.0 2.0 3.0 4.0 b 5.0, 6.0 f x1, x2 begin y A x1, x2 b return y 1 ^2 y 2 ^2 end x randn 2 result ad ForwardDiff.gradient X f X... , x result 2 A x b ' A abs2 result ad 1 result 1 abs2 result ad 2 result 2 end md\"\"\" Next, let's consider a non square matrix A and set b 0 for simplicity. Then, we have \\nabla x \\| A x \\| 2^2 2 A x ^T A 2 x^T A^T A. If you want to calculate this gradient by hand, you have to choose whether you want to first calculate A^T A and then x^T A^T A first calculate x^T A^T and then x^T A^T A Which order would you choose? \"\"\" order 1 A, x x' A' A order 2 A, x x' A' A md\" 4 \\times 4 matrix\" A1 randn 4, 4 x1 randn size A1, 2 benchmark order 1 A1, x1 benchmark order 2 A1, x1 md\" 2 \\times 8 matrix\" A3 randn 2, 8 x3 randn size A3, 2 benchmark order 1 A3, x3 benchmark order 2 A3, x3 md\"\"\" General introduction The difference between these orders is the basic idea of forward vs. reverse mode AD. To give you a rough idea, consider the chain rule applied to the function x \\mapsto f\\Bigl g\\bigl h x \\bigr \\Bigr , i.e., \\nabla x f\\Bigl g\\bigl h x \\bigr \\Bigr f'\\Bigl g\\bigl h x \\bigr \\Bigr \\cdot g'\\bigl h x \\bigr \\cdot h' x . Forward mode AD is like computing the derivatives from right to left like the usual data flow when computing f \\circ g \\circ h x . In contrast, reverse mode AD is like computing the derivatives from left to right, i.e., in reverse order. To be able to do so, you need to store the indermediate values h x and g\\bigl h x \\bigr . \"\"\" function forward f, f′, g, g′, h, h′, x h x h x h′ x h′ x gh x g h x gh′ x g′ h x h′ x fgh x f gh x fgh′ x f′ gh x gh′ x return fgh x, fgh′ x end function reverse f, f′, g, g′, h, h′, x h x h x gh x g h x fgh x f gh x f′ ghx f′ gh x fg′ hx f′ ghx g′ h x fgh′ x fg′ hx h′ x return fgh x, fgh′ x end const A randn 10, 10^2 const b randn size A, 1 h x AbstractVector A x h′ x AbstractVector A g Ax AbstractVector Ax b g′ Ax AbstractVector I f Ax b AbstractVector sum abs2, Ax b f′ Ax b AbstractVector 2 Ax b' let x randn size A, 2 fwd forward f, f′, g, g′, h, h′, x rev reverse f, f′, g, g′, h, h′, x fwd 1 rev 1 , norm fwd 2 rev 2 end let x randn size A, 2 benchmark forward f, f′, g, g′, h, h′, x end let x randn size A, 2 benchmark reverse f, f′, g, g′, h, h′, x end md\"\"\" Thinking about Reverse Mode \"\"\" function simple mlp input, weight, bias tmp1 input weight tmp2 tmp1 bias return tmp2 end function dsimple mlp input, weight, bias tmp1 input weight tmp2 tmp1 bias return tmp2 start by creating and zero'ing shadow variables for each differentiable input dinput 0.0 dweight 0.0 dbias 0.0 start by creating and zero'ing shadow variables for each differentiable variable dtmp1 0.0 dtmp2 0.0 derivative of return tmp2, set dtmp2 1.0 dtmp2 1.0 derivative of tmp2 tmp1 bias dtmp1 dtmp2 dbias dtmp2 dtmp2 0.0 After a read of a shodow we always must zero derivative of tmp1 input weight dinput dtmp1 weight dweight dtmp1 input dtmp1 0.0 After a read of a shodow we always must zero we're done, return the gradient derivative of all input variable return dinput, dweight, dbias end dsimple mlp 2.0, 0.5, 0.3 import Enzyme Reverse, Active, Duplicated Enzyme.autodiff Reverse, simple mlp, Active 2.0 , Active 0.5 , Active 0.3 function with memory x x x 1 tmp1 x ^ 2 return tmp1 end function dwith memory x x x 1 tmp1 x ^ 2 return tmp1 dx Enzyme.make zero x dx 0.0 dtmp1 0.0 dtmp1 1.0 dx dtmp1 2 x dtmp1 0.0 dx 1 dx note the `accumulate` operation return dx, end dwith memory 2.0 function with memory2 x, y tmp1 x 1 ^ 2 y 1 tmp1 return nothing end md\"\"\" This time let the user provide dx, dy \"\"\" function dwith memory2 x, dx , y, dy tmp1 x 1 ^ 2 y 1 tmp1 dtmp1 0.0 dtmp1 dy 1 dy 1 0.0 dx 1 dtmp1 2 x 1 Note Needs to prove the x is read only and not aliased with y return nothing end let dx 0.0 dy 1.0 dwith memory2 2.0 , dx , 0.0 , dy dx, dy end let dx 0.0 dy 1.0 Enzyme.autodiff Reverse, with memory2, Duplicated 2.0 , dx , Duplicated 0.0 , dy dx, dy end function with control x cond x 0 if cond tmp1 x^2 else tmp1 x end return tmp1 end function dwith control x cond x 0 if cond tmp1 x^2 else tmp1 x end dx 0.0 dtmp1 0.0 dtmp1 1.0 if cond d x^2 dx 2x dx dtmp1 2 x else d x dx 1 dx dtmp1 1 end return dx, end dwith control 2.0 Enzyme.autodiff Enzyme.Reverse, with control, Enzyme.Active 2.0 dwith control 2.0 Enzyme.autodiff Enzyme.Reverse, with control, Enzyme.Active 2.0 md\"\"\" Machine Learning \"\"\" model Chain Dense 128, 256, tanh , Chain Dense 256, 1, tanh , Dense 1, 10 md\"\"\" Dense A Fully Connected feed forward functions Transfer function A non \"\"\" begin rng Random.default rng Random.seed rng, 0 end ps, st Lux.setup rng, model md\"\"\" note Conscious handling of randomness. \"\"\" let Dummy Input x rand rng, Float32, 128, 2 Run the model y, Lux.apply model, x, ps, st end let x rand rng, Float32, 128, 2 train state Lux.Training.TrainState model, ps, st, Adam 0.0001f0 gs, loss, stats, train state Lux.Training.compute gradients AutoEnzyme , MSELoss , x, rand rng, Float32, 10, 2 , train state Training.apply gradients train state, gs train state, gs end let x rand rng, Float32, 128, 2 train state Lux.Training.TrainState model, ps, st, Adam 0.0001f0 gs, loss, stats, train state Training.single train step AutoEnzyme , MSELoss , x, rand rng, Float32, 10, 2 , train state end md\"\"\" MNIST \"\"\" dataset MNIST convert2image dataset, 10 convert2image dataset, 12 function loadmnist batchsize, train split Load MNIST N 1500 nothing dataset MNIST split train if N nothing imgs dataset.features , , 1 N labels raw dataset.targets 1 N else imgs dataset.features labels raw dataset.targets end Process images into H, W, C, BS batches x data Float32. reshape imgs, size imgs, 1 , size imgs, 2 , 1, size imgs, 3 y data onehotbatch labels raw, 0 9 x train, y train , x test, y test splitobs x data, y data at train split return Use DataLoader to automatically minibatch and shuffle the data DataLoader collect. x train, y train batchsize, shuffle true, partial false , Don't shuffle the test data DataLoader collect. x test, y test batchsize, shuffle false, partial false , end lux model Chain Conv 5, 5 , 1 6, relu , MaxPool 2, 2 , Conv 5, 5 , 6 16, relu , MaxPool 2, 2 , FlattenLayer 3 , Chain Dense 256 128, relu , Dense 128 84, relu , Dense 84 10 , const lossfn CrossEntropyLoss logits Val true function accuracy model, ps, st, dataloader total correct, total 0, 0 st Lux.testmode st for x, y in dataloader target class onecold Array y predicted class onecold Array first model x, ps, st total correct sum target class . predicted class total length target class end return total correct total end function train model rng Random.default rng , kwargs... train dataloader, test dataloader loadmnist 128, 0.9 ps, st Lux.setup rng, model vjp AutoEnzyme Oof Enzyme takes forever to come up with the gradient 105s vjp AutoZygote train state Training.TrainState model, ps, st, Adam 3.0f 4 Lets train the model nepochs 10 tr acc, te acc 0.0, 0.0 for epoch in 1 nepochs stime time for x, y in train dataloader , , , train state Training.single train step vjp, lossfn, x, y , train state end ttime time stime tr acc accuracy model, train state.parameters, train state.states, train dataloader 100 te acc accuracy model, train state.parameters, train state.states, test dataloader 100 printf \" %2d %2d \\t Time %.2fs \\t Training Accuracy %.2f%% \\t Test Accuracy \\ %.2f%%\\n\" epoch nepochs ttime tr acc te acc end return tr acc, te acc end tr acc, te acc train lux model "},{"url":"mod5_performance_engineering/compilers/","title":"Compilers","tags":["module5","track_performance"],"text":" A Pluto.jl notebook v0.20.13 frontmatter chapter \"5\" section \"1\" order \"14\" title \"Compilers\" date \"2025 07 08\" tags \"module5\", \"track performance\" layout \"layout.jlhtml\" frontmatter.author name \"Valentin Churavy\" url \"https vchuravy.dev\" using Markdown using InteractiveUtils This Pluto notebook uses bind for interactivity. When running this notebook outside of Pluto, the following 'mock version' of bind gives bound variables a default value instead of an error . macro bind def, element format off return quote local iv try Base.loaded modules Base.PkgId Base.UUID \"6e696c72 6542 2067 7265 42206c756150\" , \"AbstractPlutoDingetjes\" .Bonds.initial value catch b missing end local el esc element global esc def Core.applicable Base.get, el ? Base.get el iv el el end format on end begin using PlutoUI using PlutoTeachingTools using ShortCodes end using JuliaSyntax using IRViz using BenchmarkTools using Unitful begin using LLVM using LLVM.Interop end html\" button onclick 'present ' Toggle presentation mode button \" PlutoUI.TableOfContents depth 4 begin using JET using AllocCheck end md\"\"\" Compilation \"\"\" md\"\"\" 1. Parsing 2. Linearization 3. Abstract interpretation based type inference 4. High level optimizations 5. LLVM based optimizations 6. Emission of native code \"\"\" func \"\"\" function example X acc zero eltype X for x in X acc x end return acc end \"\"\" md\"\"\" Parsing \"\"\" md\"\"\" note Parsing turns text into expressions. \"\"\" expr Meta.parse func parsestmt SyntaxNode, func parsestmt Expr, func md\"\"\" note To go from Expr to usable function we need to resolve symbols in a namespace and add the function to the system. `eval` will do that for us. \"\"\" example eval expr example ones 10 md\"\"\" Linearization \"\"\" md\"\"\" note `Expr` form an abstract syntax tree AST . Julia macros are functions from `Expr` to `Expr`. To simplify things later we want to linearize that tree. In Julia we call this lowering. \"\"\" CL code lowered example, Array Float64 , | only md\"\"\" note Lowered code is in single static assignment SSA form with memory. `%8 %7 nothing` is a SSA statement, and ` 3` are variables e.g. memory . \"\"\" md\"\"\" note The lowered code forms a control flow graph CFG, with a set of basic blocks. \"\"\" viz CL md\"\"\" note Control flow is implemented using `goto %n` and `goto %n if not %c` statements. Julia's IR uses implicit fallthrough, so a basic block ending in a `goto %n if not %c` statement has a second implicit successor in the subsequent basic block. \"\"\" md\"\"\" Abstract interpretation based type inference \"\"\" md\"\"\" As soon as we have an AST we could have written an interpreter. Perhaps we would use the lowered form to make our lifes easier. \"\"\" import Core.Compiler as CC function interpret naively CL, args code CL.code slots Vector Any undef, length CL.slotnames stmts Vector Any undef, length code SSA copy args into vars for name, value in args id findfirst name , CL.slotnames slots id value end resolve val CC.SSAValue stmts val.id resolve val CC.SlotNumber slots val.id function local eval stmt if stmt isa GlobalRef return eval stmt elseif stmt isa CC.SSAValue || stmt isa CC.SlotNumber return resolve stmt elseif stmt isa Expr head stmt.head if head call resolve args map local eval, stmt.args f first resolve args f args resolve args 2 end if f isa Core.IntrinsicFunction || f isa Core.Builtin || f Base.iterate return f f args... end argtypes ntuple i Core.Typeof f args i , length f args CL f code lowered f, argtypes | only f args′ collect zip CL f.slotnames 2 end , f args try return interpret naively CL f, f args′ catch exc error \"Interpretation failed\" exception exc f f args return f f args... end elseif head show stmt error \" in rhs expression\" else show stmt error \"Unknown expression\" end else return stmt end end interpreter pc 1 while true stmt code pc if stmt isa GlobalRef || stmt isa CC.SSAValue || stmt isa CC.SlotNumber stmts pc local eval stmt elseif stmt isa Expr head stmt.head if head lhs stmt.args 1 rhs stmt.args 2 assert lhs isa CC.SlotNumber val local eval rhs slots lhs.id val stmts pc val else stmts pc local eval stmt end elseif stmt isa CC.GotoIfNot cond local eval stmt.cond if cond pc stmt.dest else pc 1 end continue elseif stmt isa CC.GotoNode pc stmt.label continue elseif stmt isa CC.ReturnNode return local eval stmt.val else error \"Unknown stmt\" end pc 1 end end interpret naively CL, X ones 3 , md\"\"\" Our interpreter isn't all that fast, but it mostly works Of course if you want to support all of Julia life becomes a bit more complicated. Here I am skipping things like foreign function calls, and I fall back onto the native execution scheme if I have no other options. \"\"\" md\"\"\" note To accelerate execution we would need to cache lookups and perhaps perform optimizations such as inlining. We need to carefully consider under which assumptions we may do so \"\"\" md\"\"\" What makes a programming language dynamic? 1. Dynamic typing vs static typing 2. Open world vs closed world 3. Dynamic structs vs dynamic \"\"\" md\"\"\" Dynamic typing vs static typing The common argument JavaScript vs TypeScript reminds one a bit of Vim vs Emacs, or spaces vs tabs. Laurence Tratt A modest attempt to help prevent unnecessary static dynamic typing debates https tratt.net laurie blog 2010 a modest attempt to help prevent unnecessary static dynamic typing debates.html Laurence Tratt Dynamically Typed Languages https tratt.net laurie research pubs papers tratt dynamically typed languages.pdf a dynamically typed language is one that does not check or enforce type safety at compile time. But is there all that is to it? \"\"\" md\"\"\" Some terminology Types A type or in Object Oriented programming a class defines a set of values ```julia abstract type Fruit end struct Apples Fruit end abtract type StoneFruit end struct Peaches StoneFruit end ``` Compile time Work done before the user's program Run time Work done during the execution of the program \"\"\" md\"\"\" Static typying In statically typed languages a variable can only be assigned a type once. Explicitly Java ```java int i 3 String s \"4\" int x i s ``` Implicitly Haskell ```haskell let i 3 s \"4\" in i s ``` note This allows us to perform type checking before executing a program. Type errors occur at compile time. \"\"\" md\"\"\" Dynamic typying In a dynamically typed language type checks are deferred until runtime. \"\"\" function f i 3 s \"4\" i s end md\"\"\" note We can still write analyzers that perform these checks ahead of execution. \"\"\" md\"\"\" Dynamic typing also means that functions can return differently typed values depending on program execution. In Julia we denotate those as `Union Int, Float64 ` \"\"\" g rand ? 1 1.0 md\"\"\" Strong vs weakly typed Yet another delination is strong vs weakly typed. As an example `C` is a statically typed language, but values are weakly typed. ```c void f void p int ints int p float floats float p ints 0 floats 1 ``` \"\"\" md\"\"\" Open world vs closed world Many dynamic programing languages have the notion of `eval`. `Eval` allows one to add code at runtime. Julia is \"define by running\" code get's added incrementaly by running code. note Read Eval Print Line REPL is a powerful tool, as are use cases like notebooks. Open world semantics means code can be added or change at runtime. Closed world semantics mean that the program is fixed before running. \"\"\" md\"\"\" Interpretation vs Compilation question Is Python interpreted \"\"\" let content md\"\"\" answer Compiled to Bytecode then interpreted and there are full Python compiler \"\"\" HTML \" details html content details \" end md\"\"\" question Is C compiled? \"\"\" let content md\"\"\" answer Yes, but there are also C interpreters such as Cling. \"\"\" HTML \" details html content details \" end md\"\"\" note Interpretations vs Compilation is an implementation choice not part of the language semantics. \"\"\" md\"\"\" The curse of leaky abstractions In Python I can create a class and later modify the fields of an object of said class. ```python class MyClass ... x 5 ... c MyClass c.x 5 c.y 3 c.y 3 c MyClass c.y Traceback most recent call last File \" stdin \", line 1, in module AttributeError 'MyClass' object has no attribute 'y' ``` \"\"\" md\"\"\" Similarily in Ruby ```ruby class A end a A.new a.age 20 Raises undefined method `age ' Inject attribute accessors into instance of A class a attr accessor age end a.age 20 Succeeds a.age Returns 20 b A.new b.age Raises undefined method, as expected, since we only modified one instance of A ``` \"\"\" md\"\"\" note In both Python and Ruby fields are dynamically typed. \"\"\" md\"\"\" While Julia has inheritance, only abstract types can be inherited from. Concrete types are final. ```julia struct A age Int data Untyped field end ``` Types can also be parametric. important Since concrete types are final we can take advantage of this information, without having to rely on guard checks. \"\"\" md\"\"\" So how do we make dynamic programs run fast Julia Avoiding runtime uncertainty Sophisticated type system Type inference Multiple dispatch Specialization JIT compilation Julia Dynamism and Performance Reconciled by Design doi 10.1145 3276490 \"\"\" md\"\"\" Type inference In Julia, type inference is the process of propagating type information from arguments. This utilizes Abstract Interpretation as a technique. Iteration until fixed point is reach Convergence Types form a lattice Bottom `Union ` Top `Any` Imprecise answers are permisible E.g. `Any` is always correct Care must be taken for recursive functions and loops Requires heuristics to prevent it from running forever note Type inference allows us to aggressivly de virtualize calls. \"\"\" function mysum X acc 0 for x in X acc x end return acc end md\"\"\" The analysis for `mysum` begins with a specific signature type. `Tuple typeof mysum , Vector Float64 ` ```julia function mysum X Vector Float64 acc 0 for x in X acc x end return acc end ``` \"\"\" md\"\"\" Examining line 1 we can state that `acc Int64` due to it being set to a constant. ```julia function mysum X Vector Float64 acc 0 Int State X Vector Float64 , acc Int for x in X acc x end return acc end ``` \"\"\" md\"\"\" Let's for the moment assume that we can deduce `x in X` to imply that `x Float64` ```julia function mysum X Vector Float64 acc 0 Int for x Float64 in X State X Vector Float64 , acc Int, x Float64 acc x end return acc end ``` \"\"\" md\"\"\" `acc x` is syntax sugar for `acc acc x` Julia has a user extendable promotion scheme for arithmetic ops `Base.promote op , Int64, Float64 ` Base.promote op , Int64, Float64 ```julia function mysum X Vector Float64 acc 0 Int State X Vector Float64 , acc Int for x Float64 in X acc Float64 acc Int x Float64 State X Vector Float64 , acc Float64, x Float64 end return acc end ``` \"\"\" md\"\"\" If `length X 0` the for loop might never be executed. So after the loop we have to unify the abstract interpretation state before the loop with after the loop iteration. State Before ` X Vector Float64 , acc Int64 ` State After ` X Vector Float64 , acc Float64 ` Unification ` X Vector Float64 , acc Union Int64, Float64 ` We haven't yet reached a fix point. So we re enter the loop with our new state. ```julia function mysum X Vector Float64 acc 0 Int for x Float64 in X X Vector ... , acc Union Int64, Float64 , x Float64 acc Union Float64 acc Union Int64, Float64 x Float64 State X Vector Float64 , acc Float64, x Float64 end return acc end ``` \"\"\" md\"\"\" State Before ` X Vector Float64 , acc Union Int64, Float64 ` State After ` X Vector Float64 , acc Float64 ` Unification ` X Vector Float64 , acc Union Int64, Float64 ` Fixed point is reached and we can conclude that after the loop `acc Union Int64, Float64 ` and thus `mysum X Vector Float64 Union Int, Float64 ` We can use Julia's introspection tools to automatically query the system ` code lowered` ` code typed optimize false` \"\"\" code lowered mysum rand 3 code typed optimize false mysum rand 3 md\"\"\" What information did we use? Julia has `eval`. So what happens if someone changes the definition of say ` ` while the code is running? Julia uses the so called \"world age\" system to track the validity of method definitions and method compilations. \"\"\" TwoColumn md\"\"\" Julia 0.3 ```julia repl julia f 1 julia g f julia g 1 julia f 2 julia g 1 ``` \"\"\", md\"\"\" Julia 1.0 ```julia repl julia f 1 julia g f julia g 1 julia f 2 julia g 2 ``` \"\"\" md\"\"\" World age in Julia optimizing method dispatch in the presence of eval doi 10.1145 3428275 \"\"\" md\"\"\" Optimizations Optimizations are unobservable program transformations Except when they fail Julia as many other modern programming languages uses a multi stage optimization pipeline. High level optimizations Constant propagation Concrete evaluation Effect analysis Inlining Dead code elimination DCE Scalar replacement of aggregates SROA Future Interprocedural Escape Analyis Future Alias analysis Future Loop based optimizations such as loop invariant code motion LICM \"\"\" begin IR, rt only Base.code ircode mysum, Vector Float64 , , optimize until \"compact 1\" IR end Core.Compiler.ssa inlining pass Core.Compiler.copy IR , Core.Compiler.InliningState Core.Compiler.NativeInterpreter , false md\"\"\" Or simply \"\"\" code typed mysum rand 4 md\"\"\" LLVM based code generation LLVM is a widely used compiler framework Common set of \"middle end\" optimizations based on LLVM IR Specific backend optimizations Frontends like Julia generate LLVM intermediate representation IR \"\"\" code llvm optimize false debuginfo none mysum rand 30 md\"\"\" Optimizations include Simplifications Control flow graph CFG Instructions Julia specific passes allocation optimizations ellision heap to stack ... Loop optimization LICM Vectorization Unrolling Peeling Loop Unswitch We also perform legalization . Turning the LLVM full or Julia specific concepts into a legal representation. note Each successive lowering simplifies the program and throws information away. This opens the question When to perform what optimization. \"\"\" md\"\"\" Interpretation vs JIT vs JAOT vs AOT There are many flavours of executing programs. Interpreters Byte code interpreters Compilation Ahead of Time AOT Just in Time JIT Tracing JIT Just Ahead of Time JAOT Just in the nick of Time JIT compilers are commonly used for Java or JavaScript, they are often tiered and responsive. As an example Java may compile your program first with the `C1` compiler tuned for latency, then later if it detects your function has executed many times it will use the `C2` compiler. In JavaScript it will assome the most common type and de optimize upon uncommon types. \"\"\" md\"\"\" Currently Julia compiles every function before it's first execution. note This initial latency or time to first X, is why sometimes folks say \"Julia is fast the second time\" Over the years more and more attention has been payed on amortizing this compilation cost by performing caching. Native code caching for Julia 1.9 had a drastic impact on how \"snappy\" the language feels. \"\"\" YouTube \"jFhL8EVrz7s\", 10, 5 md\"\"\" How to make it correct? In order to execute dynamic programs fast we need to obtain information make assumptions. It is crucial that the information remains correct. info The language semantics are dynamic, optimizations and static analysis doesn't change that. So we need to track the validity of the information. Python compilers and other JITs often use guard checks whereas Julia thanks to final concrete types can use world ages to convert open world semantics to closed world semantics. Julia also uses edges to limit the invalidation effect. Methods and compile results have a pair of world ages first valid and last valid . important On disk caches must be carefully validated as well. \"\"\" md\"\"\" So why bother? The usability improvement of dynamic programming language is not to undervalue Python, JavaScript and co are very popular for accessibility reasons Dynamic Programming Languages often have a higher level of abstractions When I work on hard science, I don't want to worry about memory lifetimes Dynamic programming languages can be fast The presenence of a JIT allows for value specializationn We can start dynamic and regain information at call boundaries note Julia is a fast, high level dynamic programming language. It's not magic, but rather clever language design. \"\"\" md\"\"\" Going fast nowhere \"\"\" bind N Slider 1 9 data rand 10^N function sum X Vector Float64 acc 0 Int64 for x in X acc x Float64 end acc Union Int64, Float64 end benchmark sum data md\"\"\" Figuring out what is happening The stages of the compiler ` code lowered` ` code typed` & ` code warntype` ` code llvm` ` code native` Where is a function defined ` which` & ` edit` \"\"\" code lowered sum data md\"\"\" A simple example counting \"\"\" function f N acc 0 for i in 1 N acc 1 end return acc end f code typed f K 100 000 000 result benchmark f K begin t time minimum result u\"ns\" in ns pFreq round typeof 1u\"PHz\" , K t end md\"\"\" So we are doing K 1 000 000 million additions in t ... That would mean our processor operates at pFreq We wish... Let's do a basic check, 10x bigger input. \"\"\" benchmark f Ref 10 K md\"\"\" Let's explore what code we are actually running. Using Julia's reflection macros we can see all of the stages of code generation. \"\"\" code lowered f K code typed optimize false f K code typed optimize true f K with terminal do code llvm optimize false f K end with terminal do code llvm optimize true f K end with terminal do code native f K end md\"\"\" Conclusion LLVM realised that our loop ```julia for i in 1 N acc 1 end ``` Just ended up being acc 1 N \"\"\" md\"\"\" Can we actually measure the speed of our original code? \"\"\" \"\"\" clobber Force the compiler to flush pending writes to global memory. Acts as an effective read write barrier. \"\"\" inline clobber asmcall \"\", \"~ memory \", true \"\"\" escape val The `escape` function can be used to prevent a value or expression from being optimized away by the compiler. This function is intended to add little to no overhead. See https youtu.be nXaxk27zwlk?t 2441 \"\"\" inline escape val T where T asmcall \"\", \"X,~ memory \", true, Nothing, Tuple T , val function k Type T , N where T acc zero T for i in 1 N acc one T clobber end return acc end with terminal do code llvm debuginfo none k Int64, 10 end with terminal do code native debuginfo none k Int64, 10 end function m Type T , N where T acc zero T for i in 1 N acc one T escape acc end return acc end with terminal do code llvm debuginfo none m Int64, 10 end result2 benchmark m Int64, K benchmark m Int64, K 10 begin t2 time minimum result2 u\"ns\" in ns pFreq2 round typeof 1u\"MHz\" , K t2 end md\"\"\" Frequency estimation pFreq2 ~ round typeof 1u\"GHz\" , pFreq2 Note Benchmarking is hard, careful evalutaion of what you are trying to benchmark. If we were just interesting in how fast f N was we would have been fine with our first measurement But we were interested in the speed of addition as a proxy of perfromance Integer math on a computer is associative, Floating Point math is not. \"\"\" md\"\"\" SIMD \"\"\" TODO md\"\"\" LICM \"\"\" "}]