What it does

It records the country you sleep in, every night, without you opening it. iOS wakes the app when the phone moves a few hundred metres or settles somewhere, the app works out the country, and the next morning it writes one row to your sheet: the date, the country, and a place for evidence like a flight number.

Then it counts. The interesting part is that every rule counts a day differently. UK settlement only counts whole days outside the country, so the day you leave and the day you come back do not count. Schengen counts any part of a day, so they both do. The tax residence test looks at where you are at midnight. Daybound knows the definition for each rule, and when you open a day it tells you which rules it counts for and why.

The settlement screen goes one step further and tells you the earliest day you can apply, starting from the day your visa started, and starting over if you ever broke the 180-day rule. I learned from it that I had broken it in 2023 without noticing.

Why a spreadsheet

Because these numbers matter for years, and an app is only as durable as the app. Your history is an ordinary Google Sheet in your own Drive. You can open it, send it to a lawyer, back it up, or delete the app; the rows stay. The app creates the sheet itself and can only see the files it created, nothing else in your Drive.

There is a second sheet with every location fix the phone ever took, and you can import your Google Maps Timeline export into it, so the past is yours as well.

The one manual button

I removed every way of typing days in by hand. There is exactly one: “Correct this day”, and it asks why. The reason goes into the sheet next to the day, so if anyone ever asks, the record explains itself.

Getting it

Daybound is free on the App Store, with no subscription and no account with me. Every screen works with demo data before you sign in with Google, so you can look around first. It is not legal or tax advice; check the rules that apply to you.

What it does while you drive

Every few kilometres it quietly looks up what is around you: Wikipedia, in the local language because that is usually where the good articles are, travel guides, and the web for the things encyclopedias miss. Then it decides what is worth mentioning. The bar I aimed for is what a good local guide sitting in the passenger seat would point out: famous enough, interesting enough, close enough to the road.

You can also tell it what you care about in the settings. If you say you like railways, it will happily talk about a small station that anyone else would drive past in silence.

It tries to get the timing right too. A place is mentioned once, a little before it comes into view, and it tells you which side to look. In the screenshots: 1.1 km, straight ahead.

And if something makes you curious, you hold a button and ask. It answers, and looks things up if it needs to.

The practical bits

Your OpenAI key stays on your phone and is only used to talk to OpenAI. An hour of driving costs somewhere between 15 and 40 cents. Everything else comes from free sources like Wikipedia.

It runs in the browser, so it keeps your screen on while it works, like a navigation app does. There is also an iPhone version of the same thing.

One honest note about the screenshots: they come from the app’s built-in test drive, on the road from Monteriggioni to Siena. The car and the voice are simulated so I could replay the same drive while building the app, but the map and the places are real.

Why a spreadsheet

Because your data stays yours. Your tasks are just rows in a normal spreadsheet. You can open it in Google Sheets and edit it there whenever you want, and it will still be there, perfectly readable, in ten years, even if this app disappears. The app can only see the files it created or the ones you picked. It cannot look at anything else in your Drive.

Letting your agents use it

This is the reason I built it. I wanted a place where an AI agent could leave something for me, and where I could leave something for it, that is not a chat log and not a file buried in some repo.

If you use claude.ai or Claude Code, you can add Memoria as a connector, sign in with Google, and from then on your agent can read the board, add tasks, move them around and take notes. It works even when the agent runs on a schedule, without you there.

Trying it

The easiest way is the hosted version at memoria-board.vercel.app. If you would rather run your own copy, the code is open source and takes about fifteen minutes to deploy, all on free tiers.

Source: github.com/RCambier/Memoria. MIT.

Bayes for a set of random events

My own way of understanding Bayesian Inference. Heavily based on: - My examples are partially taken from: https://www.bayesrulesbook.com/chapter-6.html - My understanding is coming from: https://xcelab.net/rm/statistical-rethinking/

probabilities = pd.DataFrame({
    'events': ['Rain', 'NoRain'],
    'prior_knowledge': [0.5, 0.5],
});

probabilities

# We observed the data when calling our friend
likelihood = pd.DataFrame({\
    'events': ['Rain', 'NoRain'],
    'likelihood': [2/3, 1/3], # We use our system, here logical thinking from what is said in the prompt, to link the observations to each event
});

likelihood

bayes = pd.concat([probabilities, likelihood.drop(columns=['events'])], axis=1)
bayes

bayes['posterior_unnormalized'] = bayes['prior_knowledge'] * bayes['likelihood']
bayes

bayes['posterior'] = bayes['posterior_unnormalized'] / bayes['posterior_unnormalized'].sum()
bayes

Bayes for more categories

From 2 categories, we can move to more categories

From : https://www.bayesrulesbook.com/chapter-2#michelle-simple

For example, suppose you’re watching an interview of somebody that lives in the United States. Without knowing anything about this person, U.S. Census figures provide prior information about the region in which they might live: the Midwest ( M ), Northeast ( N ), South ( S ), or West ( W ).16 This prior model is summarized in Table 2.5.17 Notice that the South is the most populous region and the Northeast the least (P(S) > P(N)). Thus, based on population statistics alone, there’s a 38% prior probability that the interviewee lives in the South

Now that we have the prior probabilities for each event, we need to observe some data. Here we will observe

# We observered the data when the person said "pop"
probabilities = pd.DataFrame({
    'events': ['M', 'N', 'S', 'W'],
    'prior_knowledge': [0.21, 0.17, 0.38, 0.24],  # The likelihoods are derived from analysing data in each region.
});

probabilities

But then, you see the person point to a fizzy cola drink and say “please pass my pop.” Though the country is united in its love of fizzy drinks, it’s divided in what they’re called, with common regional terms including “pop,” “soda,” and “coke.” This data, i.e., the person’s use of “pop,” provides further information about where they might live. To evaluate this data, we can examine the pop_vs_soda dataset in the bayesrules package (Dogucu, Johnson, and Ott 2021) which includes 374250 responses to a volunteer survey conducted at popvssoda.com

likelihood = pd.DataFrame({
    'events': ['M', 'N', 'S', 'W'],
    'likelihood': [0.6447, 0.2734, 0.07922, 0.2943], # We use our system, here obserbing data in each region, to link the observations to each event
});

likelihood

bayes = pd.concat([probabilities, likelihood.drop(columns=['events'])], axis=1)
bayes

bayes['posterior_unnormalized'] = bayes['prior_knowledge'] * bayes['likelihood']
bayes

bayes['posterior'] = bayes['posterior_unnormalized'] / bayes['posterior_unnormalized'].sum()
bayes

Bayes for random variable, still discrete

We now have a system where, if we have prior events with prior probabilities, and some “data” event happening that relate to the prior events with some probabilities, we can derive a posterior where we update our knowledge on the prior events.

The Trick

Magically, we can use this exact system with a Statistical Model, and adapt it the following way: - The “prior events” are going to be a random variable of a statistical model - The “data events” are going to be observations you make of the statistical model outputs

probabilities = pd.DataFrame({
    'events': ['0.2', '0.5', '0.8'],
    'prior_knowledge': [0.1, 0.25, 0.65],
});

probabilities

the binomial model is the “system” we use to find the likelihood at each event, at each potential value of PI

# We observed 1 victory out of 6 games
likelihood = pd.DataFrame({
    'events': ['0.2', '0.5', '0.8'],
    'likelihood': [
        binom.pmf(k=1, n=6, p=0.2),  # We use our system, here the Binomial distribution, to link the observations to each event
        binom.pmf(k=1, n=6, p=0.5),  # We use our system, here the Binomial distribution, to link the observations to each event
        binom.pmf(k=1, n=6, p=0.8)], # We use our system, here the Binomial distribution, to link the observations to each event
});

likelihood

bayes = pd.concat([probabilities, likelihood.drop(columns=['events'])], axis=1)
bayes

bayes['posterior_unnormalized'] = bayes['prior_knowledge'] * bayes['likelihood']
bayes

bayes['posterior'] = bayes['posterior_unnormalized'] / bayes['posterior_unnormalized'].sum()
bayes

Bayes for continuous random variable

Instead of considering only 3 possible events, 3 possible values to the PI, we can define a continuous set of possible events.

Let’s say that PI represents the chances of winning a chess Game.

The observable “data” is how many out of 8 games have been won.

For example if 1 out of 8 games are won, we can estimate the likelihood of that event happening for every possible PI with Binomial(k=1, n=8, p=PI)

linspace = np.linspace(0, 1, 30)
pdf = beta.pdf(x=linspace, a=3, b=3)
pdf = pdf/pdf.sum()
plt.title('prior')
plt.stem(linspace, pdf)

probabilities = pd.DataFrame({
    'events': linspace,
    'prior_knowledge': pdf,
});

probabilities

# We observed 1 win out of 8 games
likelihood = pd.DataFrame({
    'events': linspace,
    'likelihood': [
        binom.pmf(k=1, n=8, p=probability) # We use our system, here the PMF of the Binomial, to link the observations to each event
        for probability in linspace],
});

likelihood

plt.title('likelihood')
plt.stem(likelihood['events'], likelihood['likelihood'])

bayes = pd.concat([probabilities, likelihood.drop(columns=['events'])], axis=1)
bayes

bayes['posterior_unnormalized'] = bayes['prior_knowledge'] * bayes['likelihood']
bayes

bayes['posterior'] = bayes['posterior_unnormalized'] / bayes['posterior_unnormalized'].sum()
bayes

plt.title('posterior')
plt.stem(linspace, bayes['posterior'])

Using it for a regression

https://www.bayesrulesbook.com/chapter-9

So far, this is neat, but not very useful. How can we use this system for a regression ?

The idea is to assign things in the following way: - For a regression Yi = Gaussian(mu=Alpha + Beta * Xi, sigma=variance) (for a single Sample!!) - The “events” are going to be the possible values of Alpha and Beta - The “system” linking a likelihood to each possible events is going to be applying the linear system with the chosen Alpha, Beta and the observed Xi and Yi fo this sample.

The situation will be the following:

We have a dataset coming from a bike rental agencies. For a set of sample days, we have the number of clients and the temperature.

We will start with a trivial dataset with a single sample: - temperatures = [15] - n_clients [5]

We want to build a regression to know how those two values relate.

We will then extend to a less trivial use case with multiple samples: - temperatures = [15, 23, 22, 12] - n_clients = [5, 7, 6, 2]

from scipy.stats import expon, beta, norm

linspace = np.linspace(-3, 6, 40)

alpha_pdf = norm.pdf(x=linspace, loc=0, scale=5)
alpha_pdf = alpha_pdf / alpha_pdf.sum()

beta_pdf = norm.pdf(x=linspace, loc=2, scale=5)
beta_pdf = beta_pdf / beta_pdf.sum()

gaussian_variance_pdf = expon.pdf(x=linspace, scale=0.2)
gaussian_variance_pdf = gaussian_variance_pdf / gaussian_variance_pdf.sum()

plt.title('priors')
plt.stem(linspace, alpha_pdf, label='Alpha', basefmt=" ")
plt.stem(linspace, beta_pdf, 'g', label='Beta', basefmt=" ")
plt.legend()
plt.show()


plt.stem(linspace, gaussian_variance_pdf, 'r', label='Variance', basefmt=" ")
plt.legend()
plt.show()

from IPython.display import display

probabilities_alpha = pd.DataFrame({
    'event_alpha': linspace,
    'prior_knowledge_alpha': alpha_pdf,
});

probabilities_beta = pd.DataFrame({
    'event_beta': linspace,
    'prior_knowledge_beta': beta_pdf,
});

probabilities_variance = pd.DataFrame({
    'event_variance': linspace,
    'prior_knowledge_variance': gaussian_variance_pdf,
});

display(probabilities_alpha.head())
display(probabilities_beta.head())
display(probabilities_variance.head())

all_possible_events= pd.merge(
    left=pd.merge(left=probabilities_alpha,
                  right=probabilities_beta,
                  how='cross'),
    right=probabilities_variance,
    how='cross')

all_possible_events = all_possible_events[all_possible_events.columns.sort_values()]
all_possible_events['prior'] = all_possible_events['prior_knowledge_alpha'] * all_possible_events['prior_knowledge_beta'] * all_possible_events['prior_knowledge_variance']
all_possible_events

# We image a system where each day, the chance of rain is represented by number_of_clients = alpha + beta * temperature
# We observed a day with 15 degrees, and 5 clients
# What is the likelihood of these observations at each possible "event" of alpha, beta and variance ?
all_possible_events_one_day = all_possible_events.copy()
likelihood = pd.DataFrame({
    'likelihood': [
         norm.pdf(x=5,                                             # We observe 5 clients
                 loc=row['event_alpha'] + row['event_beta'] * 15,  # We had 15 degrees, and every possible combinations of alpha, beta and variance
                 scale=row['event_variance'])
         for _, row in all_possible_events_one_day.iterrows()],
}, index=all_possible_events_one_day.index);

all_possible_events_one_day['likelihood'] = likelihood
all_possible_events_one_day

/private/tmp/claude-501/-Users-rodolphecambier-Projects-rcambier-github-io--claude-worktrees-blog-post-review-8a57f4/0010ddf8-6e73-46b1-8c34-53b653ade511/scratchpad/bayes-venv/lib/python3.12/site-packages/scipy/stats/_distn_infrastructure.py:2079: RuntimeWarning: divide by zero encountered in divide
  x = np.asarray((x - loc)/scale, dtype=dtyp)

all_possible_events_one_day.sort_values('likelihood', ascending=False).head()

all_possible_events_one_day['posterior_unnormalized'] = all_possible_events_one_day['prior'] * all_possible_events_one_day['likelihood']
all_possible_events_one_day['posterior'] = all_possible_events_one_day['posterior_unnormalized'] / all_possible_events_one_day['posterior_unnormalized'].sum()
all_possible_events_one_day

The posterior is the 3 dimensional joint posterior probability for every sample (alpha, beta, variance).

We can also get the marginal posterior distributions for each dimension:

all_possible_events_one_day.groupby(['event_alpha'])['posterior'].sum()

event_alpha
-3.000000    0.002512
-2.769231    0.005188
-2.538462    0.012265
-2.307692    0.040148
-2.076923    0.101899
-1.846154    0.119673
-1.615385    0.061342
-1.384615    0.019318
-1.153846    0.007543
-0.923077    0.003718
-0.692308    0.001918
-0.461538    0.001131
-0.230769    0.000881
 0.000000    0.000998
 0.230769    0.001564
 0.461538    0.002950
 0.692308    0.005899
 0.923077    0.013508
 1.153846    0.042828
 1.384615    0.105282
 1.615385    0.119758
 1.846154    0.059455
 2.076923    0.018135
 2.307692    0.006858
 2.538462    0.003274
 2.769231    0.001636
 3.000000    0.000934
 3.230769    0.000705
 3.461538    0.000772
 3.692308    0.001172
 3.923077    0.002140
 4.153846    0.004145
 4.384615    0.009193
 4.615385    0.028230
 4.846154    0.067214
 5.076923    0.074052
 5.307692    0.035608
 5.538462    0.010519
 5.769231    0.003853
 6.000000    0.001781
Name: posterior, dtype: float64

plt.title('marginal posterior alpha')
marginal_alpha = all_possible_events_one_day.groupby(['event_alpha'])['posterior'].sum()
plt.stem(marginal_alpha.index, marginal_alpha)
plt.show()

plt.title('marginal posterior beta')
marginal_beta = all_possible_events_one_day.groupby(['event_beta'])['posterior'].sum()
plt.stem(marginal_beta.index, marginal_beta)
plt.show()

plt.title('marginal posterior variance')
marginal_variance = all_possible_events_one_day.groupby(['event_variance'])['posterior'].sum()
plt.stem(marginal_variance.index, marginal_variance)
plt.show()

all_possible_events_one_day_ = all_possible_events_one_day.dropna()
sampled_parameters = choice(all_possible_events_one_day_.index, size=100, p=all_possible_events_one_day_['posterior'])
sampled_parameters = all_possible_events_one_day_.loc[sampled_parameters]

def mxline(intercept, slope, start=-2, end=7):
    y1 = slope*start + intercept
    y2 = slope*end + intercept
    plt.plot([start, end], [y1, y2])

for _, row in sampled_parameters[['event_alpha', 'event_beta']].iterrows():
  mxline(row['event_alpha'], slope=row['event_beta'])
plt.show()

import seaborn as sns

# Let's cheat and see what the alpha and beta values could be
sns.regplot(x=[0, 1, 5,  7,  6,  2 ], y=[5, 6, 15, 17, 18, 12], truncate=False)

import math

temperatures = [0, 1, 5,  7,  6,  2 ] # x
n_clients    = [5, 6, 15, 17, 18, 12] # y

# Finding the likelihood of observing those values at every possible alpha,beta,variance events
all_possible_events_mutliple_days = all_possible_events.copy()
for i, sample in enumerate(zip(temperatures, n_clients)):
  likelihood = pd.Series([
          norm.pdf(x=sample[1],
                  loc=row['event_alpha'] + row['event_beta'] * sample[0],
                  scale=row['event_variance'])
          for _, row in all_possible_events_mutliple_days.iterrows()],
  );
  all_possible_events_mutliple_days[f'likelihood_{i}'] = likelihood

# The likelihood is the product of all the daily likelihoods !
all_possible_events_mutliple_days['likelihood'] = math.prod([all_possible_events_mutliple_days[f'likelihood_{i}'] for i in range(len(temperatures))])
all_possible_events_mutliple_days

all_possible_events_mutliple_days['posterior_unnormalized'] = all_possible_events_mutliple_days['prior'] * all_possible_events_mutliple_days['likelihood']
all_possible_events_mutliple_days['posterior'] = all_possible_events_mutliple_days['posterior_unnormalized'] / all_possible_events_mutliple_days['posterior_unnormalized'].sum()

all_possible_events_mutliple_days = all_possible_events_mutliple_days.dropna()

# Once we know the posterior of each 3-dimensional event, we can gather the marginal posterior distributions
plt.title('marginal posterior alpha')
marginal_alpha = all_possible_events_mutliple_days.groupby(['event_alpha'])['posterior'].sum()
plt.stem(marginal_alpha.index, marginal_alpha)
plt.show()

plt.title('marginal posterior beta')
marginal_beta = all_possible_events_mutliple_days.groupby(['event_beta'])['posterior'].sum()
plt.stem(marginal_beta.index, marginal_beta)
plt.show()

plt.title('marginal posterior variance')
marginal_variance = all_possible_events_mutliple_days.groupby(['event_variance'])['posterior'].sum()
plt.stem(marginal_variance.index, marginal_variance)
plt.show()

/private/tmp/claude-501/-Users-rodolphecambier-Projects-rcambier-github-io--claude-worktrees-blog-post-review-8a57f4/0010ddf8-6e73-46b1-8c34-53b653ade511/scratchpad/bayes-venv/lib/python3.12/site-packages/scipy/stats/_distn_infrastructure.py:2079: RuntimeWarning: divide by zero encountered in divide
  x = np.asarray((x - loc)/scale, dtype=dtyp)
/private/tmp/claude-501/-Users-rodolphecambier-Projects-rcambier-github-io--claude-worktrees-blog-post-review-8a57f4/0010ddf8-6e73-46b1-8c34-53b653ade511/scratchpad/bayes-venv/lib/python3.12/site-packages/scipy/stats/_distn_infrastructure.py:2079: RuntimeWarning: invalid value encountered in divide
  x = np.asarray((x - loc)/scale, dtype=dtyp)

all_possible_events_mutliple_days

sampled_parameters = choice(all_possible_events_mutliple_days.index, size=100, p=all_possible_events_mutliple_days['posterior'])
sampled_parameters = all_possible_events_mutliple_days.loc[sampled_parameters]

def mxline(intercept, slope, start=-2, end=7):
    y1 = slope*start + intercept
    y2 = slope*end + intercept
    plt.plot([start, end], [y1, y2])

for _, row in sampled_parameters[['event_alpha', 'event_beta']].iterrows():
  mxline(row['event_alpha'], slope=row['event_beta'])
plt.show()

Doing the same with PyMC

import pymc as pm
import arviz as az

basic_model = pm.Model()


# Reminder: This is how we did it manually

# linspace = np.linspace(0, 1, 30)
# alpha_pdf = beta.pdf(x=linspace, a=3, b=3)
# alpha_pdf = alpha_pdf / alpha_pdf.sum()

# beta_pdf = beta.pdf(x=linspace, a=3, b=1)
# beta_pdf = beta_pdf / beta_pdf.sum()

# gaussian_variance_pdf = expon.pdf(x=linspace, scale=0.2)
# gaussian_variance_pdf = gaussian_variance_pdf / gaussian_variance_pdf.sum()

Y = [5] # The number of clients
X1 = [15] # The temperature

with basic_model:

    # Let's do the same with PyMC
    alpha = pm.Normal("alpha", 0, 5)
    beta = pm.Normal("beta", 2, 5)
    sigma = pm.Exponential("sigma", scale=0.2)

    # Expected value of outcome
    mu = alpha + beta * X1

    # Likelihood (sampling distribution) of observations
    Y_obs = pm.Normal("Y_obs", mu=mu, sigma=sigma, observed=Y)

    trace = pm.sample(5000)

az.plot_trace(trace, figsize=(10, 10));

Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (4 chains in 4 jobs)
NUTS: [alpha, beta, sigma]

/private/tmp/claude-501/-Users-rodolphecambier-Projects-rcambier-github-io--claude-worktrees-blog-post-review-8a57f
4/0010ddf8-6e73-46b1-8c34-53b653ade511/scratchpad/bayes-venv/lib/python3.12/site-packages/rich/live.py:260: 
UserWarning: install "ipywidgets" for Jupyter support
  warnings.warn('install "ipywidgets" for Jupyter support')

Sampling 4 chains for 1_000 tune and 5_000 draw iterations (4_000 + 20_000 draws total) took 3 seconds.
There were 2326 divergences after tuning. Increase `target_accept` or reparameterize.
The rhat statistic is larger than 1.01 for some parameters. This indicates problems during sampling. See https://arxiv.org/abs/1903.08008 for details
The effective sample size per chain is smaller than 100 for some parameters.  A higher number is needed for reliable rhat and ess computation. See https://arxiv.org/abs/1903.08008 for details

basic_model = pm.Model()

temperatures = [0, 1, 5,  7,  6,  2 ] # x
n_clients    = [5, 6, 15, 17, 18, 12] # y

with basic_model:

    # Let's do the same with PyMC
    alpha = pm.Normal("alpha", 0, 5)
    beta = pm.Normal("beta", 2, 5)
    sigma = pm.Exponential("sigma", scale=0.2)

    # Expected value of outcome
    mu = alpha + beta * temperatures

    # Likelihood (sampling distribution) of observations
    Y_obs = pm.Normal("Y_obs", mu=mu, sigma=sigma, observed=n_clients)

    trace = pm.sample(5000)

az.plot_trace(trace, figsize=(10, 10));

Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (4 chains in 4 jobs)
NUTS: [alpha, beta, sigma]

/private/tmp/claude-501/-Users-rodolphecambier-Projects-rcambier-github-io--claude-worktrees-blog-post-review-8a57f
4/0010ddf8-6e73-46b1-8c34-53b653ade511/scratchpad/bayes-venv/lib/python3.12/site-packages/rich/live.py:260: 
UserWarning: install "ipywidgets" for Jupyter support
  warnings.warn('install "ipywidgets" for Jupyter support')

Sampling 4 chains for 1_000 tune and 5_000 draw iterations (4_000 + 20_000 draws total) took 1 seconds.

Re-exploring the initial problem, but differently

In the first problem of this notebook, we analyzed the probability of 2 events, “Rain” and “NoRain” based on some additional observerd data.

Let’s reproduce our results with PyMC, where we model the initial events as “Rain” and “NoRain”

basic_model = pm.Model()
with basic_model:

    p = pm.Bernoulli("Rain", 0.5) # A prio on the Rain event
    p_yes = p * 2/3 + (1-p) * 1/3
    Y_obs = pm.Bernoulli("Y_obs", p=p_yes, observed=1) # The chances of observing "Friend saying it rains"

    trace = pm.sample(5000)

az.plot_trace(trace, figsize=(6,3));

Multiprocess sampling (4 chains in 4 jobs)
BinaryGibbsMetropolis: [Rain]

/private/tmp/claude-501/-Users-rodolphecambier-Projects-rcambier-github-io--claude-worktrees-blog-post-review-8a57f
4/0010ddf8-6e73-46b1-8c34-53b653ade511/scratchpad/bayes-venv/lib/python3.12/site-packages/rich/live.py:260: 
UserWarning: install "ipywidgets" for Jupyter support
  warnings.warn('install "ipywidgets" for Jupyter support')

Sampling 4 chains for 1_000 tune and 5_000 draw iterations (4_000 + 20_000 draws total) took 0 seconds.

Another way to tackle the same problem is to consider the events to be all possible values of “p”, the probability of a binomial distribution that would represent the chances of raining.

linspace = np.linspace(0, 1, 30)
pdf = uniform.pdf(x=linspace)
pdf = pdf/pdf.sum()

plt.title('prior')
plt.stem(linspace, pdf)
plt.show()

probabilities = pd.DataFrame({
    'events': linspace,
    'prior_knowledge': pdf,
});

probabilities

# We observed 1 friend saying "rain"
likelihood = pd.DataFrame({
    'events': linspace,
    'likelihood': [
        bernoulli.pmf(k=1, p=1/3*(1-probability) + 2/3*probability) # Here the system is the link between rain and the friend lying about it
        for probability in linspace],
});

likelihood

plt.title('likelihood')
plt.stem(likelihood['events'], likelihood['likelihood'])
plt.show()

bayes = pd.concat([probabilities, likelihood.drop(columns=['events'])], axis=1)
bayes

bayes['posterior_unnormalized'] = bayes['prior_knowledge'] * bayes['likelihood']
bayes

bayes['posterior'] = bayes['posterior_unnormalized'] / bayes['posterior_unnormalized'].sum()
bayes

plt.title('posterior')
plt.stem(linspace, bayes['posterior'])
plt.show()

Now the same but with PyMC

basic_model = pm.Model()
with basic_model:
    p = pm.Uniform("p", 0, 1) # A flat prior between 0 and 1 for the probability of rain
    rain = pm.Bernoulli("rain", p=p) # we plug it in here
    p_yes = rain * 2/3 + (1-rain) * 1/3
    Y_obs = pm.Bernoulli("Y_obs", p=p_yes, observed=1) # The chances of observing "Friend saying it rains"

    trace = pm.sample(5000)

az.plot_trace(trace, figsize=(6,3));

Multiprocess sampling (4 chains in 4 jobs)
CompoundStep
>NUTS: [p]
>BinaryGibbsMetropolis: [rain]

/private/tmp/claude-501/-Users-rodolphecambier-Projects-rcambier-github-io--claude-worktrees-blog-post-review-8a57f
4/0010ddf8-6e73-46b1-8c34-53b653ade511/scratchpad/bayes-venv/lib/python3.12/site-packages/rich/live.py:260: 
UserWarning: install "ipywidgets" for Jupyter support
  warnings.warn('install "ipywidgets" for Jupyter support')

Sampling 4 chains for 1_000 tune and 5_000 draw iterations (4_000 + 20_000 draws total) took 1 seconds.

1. Loading a dataset and fitting a model

X, y = load_breast_cancer(return_X_y=True, as_frame=True)

lr = LogisticRegression(max_iter=100, verbose=0).fit(X.values, y.values)

/usr/local/lib/python3.8/dist-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  n_iter_i = _check_optimize_result(

2. Computing feature importance

coalitions = []
for i in [0,1, len(X.columns)-1, len(X.columns)]: 
  num_ones = i 
  num_zeroes = len(X.columns) - i
  coalitions += list(multiset_permutations([0] * num_zeroes + [1] * num_ones))

pd.DataFrame(coalitions).style.applymap(lambda v: 'background-color:red' if v == 1 else None)

# We will gather all feature impacts 
feature_impacts = {col:[] for col in X.columns}

# The mean values are used for when a feature needs to be "removed"
mean_values = X.mean().values

# For each coalition we have prepared (1s and 0s)
for coalition in tqdm(pd.DataFrame(coalitions).astype(bool).values, total=len(coalitions)): 

  # For each sample in the dataset
  for sample in X.values: 

    # In the sample, replace all the '0s' of the coalition
    # with the mean of that feature
    sample_masked = sample.copy()
    sample_masked[~coalition] = mean_values[~coalition] 

    # For each feature
    for feature_idx, feature_name in enumerate(X.columns):

      # Compute the score without the feature
      sample_without_feature = sample_masked.copy()
      sample_without_feature[feature_idx] = mean_values[feature_idx]
      score_without_feature = lr.predict_proba([sample_without_feature])[0][1]

      # Compute the score with the feature
      sample_with_feature = sample_masked.copy()
      sample_with_feature[feature_idx] = sample[feature_idx]
      score_with_feature = lr.predict_proba([sample_with_feature])[0][1]

      # Store the feature impact
      feature_impacts[feature_name].append(score_with_feature - score_without_feature)

pd.DataFrame(feature_impacts, columns=X.columns).head()

pd.DataFrame(feature_impacts).abs().mean().sort_values().iloc[-10:].plot.barh(figsize=(10, 6))

The method described in the SHAP paper

The SHAP paper gives a faster method in order to compute accurate SHAP values with a subset of coalitions.

They create a linear regression for each sample, trying to match all the coalitions to the scores, in order to compute the SHAP values in one shot.

feature_impacts = []
M = len(X.columns)

# Create a big set of coalitions
coalitions_array = np.random.choice([0, 1], size=(1000, len(X.columns)))

for sample_idx in tqdm(range(len(X)), total=len(X)):
  # Apply all coalitions to sample 0 
  sample_with_missing_replaced = (coalitions_array * X.iloc[sample_idx].values) + ((1 - coalitions_array) * X.mean().values)
  # Get model predictions for all these possibilities
  predictions = lr.predict_proba(sample_with_missing_replaced)[:,0]
  # Find a linear model that matches the coalitioned features to the predictions
  linreg = LinearRegression().fit(coalitions_array, predictions)
  feature_impacts.append(linreg.coef_)

pd.DataFrame(feature_impacts, columns=X.columns).abs().mean().sort_values(ascending=True).iloc[-10:].plot.barh(figsize=(10, 6))

Comparing to SHAP

The calculation above should get us close to the SHAP values

%%capture
!pip install shap

import shap
expl = shap.KernelExplainer(lambda x: lr.predict_proba(x)[:, 1], X.mean())
shap_values = expl.shap_values(X)

shap.plots.bar(shap.Explanation(shap_values, np.repeat(expl.expected_value, len(shap_values)), feature_names=X.columns))

It is highlighting the same set of top features as our manual method, with similar feature importance values.

We would get better results if we considered more coalitions, but the problem becomes exponentially harder.

Sources

Some good links: - https://christophm.github.io/interpretable-ml-book/shap.html#kernelshap - https://www.telesens.co/2020/09/17/kernel-shap/

Using the jobs feature

This is using the foreground/background job feature of the terminal. Here is a more detailed tutorial.

To summarize the few command I use: - CTRL + Z to put a job in the background - jobs to have a look at all the jobs - fg to put the last bakgrounded process back to the foreground - fg %2 to put another job back to the foreground - kill %2 to kill job number 2

For multiprocessing:

from multiprocessing.pool import Pool
from tqdm.auto import tqdm 
import time

def work_function(arg): 
    time.sleep(arg)
    return arg

def iterable():
    for i in range(0,20):
        time.sleep(i/10)
        yield i

with Pool(10) as p: 
    results = list(tqdm(p.imap(work_function, iterable(), chunksize=1)))
    print(results)
    print("done")

For multithreading:

from multiprocessing.pool import ThreadPool
from tqdm.auto import tqdm 
import time

def work_function(arg): 
    time.sleep(arg)
    return arg

def iterable():
    for i in range(0,20):
        time.sleep(i/10)
        yield i

with ThreadPool(10) as p: 
    results = list(tqdm(p.imap(work_function, iterable(), chunksize=1)))
    print(results)
    print("done")

Why not use concurrent.futures ?

Because this option does not allow for the imap multiprocessing. This means that all the iterable will be consumed before being sent to the workers. This could be fine, but sometimes, if the iterable takes time to compute or is a generator itself, you don’t want to consume it fully before starting the concurrent processing.

Try the code below. Notice that the loading bar starts appearing once the iterable has been consumed, which means it already reached the 13/20 iteration.

from concurrent.futures import ProcessPoolExecutor
from tqdm.auto import tqdm 
import time

def work_function(arg): 
    time.sleep(arg)
    return arg

def iterable():
    for i in range(0,20):
        time.sleep(i/10)
        yield i

with ProcessPoolExecutor(10) as p: 
    results = list(tqdm(p.map(work_function, iterable(), chunksize=1)))
    print(results)
    print("done")

What about tqdm process_map ?

tqdm.contrib.concurrent.process_map is essentially the same as the concurrent.futures behind the scenes, and will exhibit the same behavior.

How to share data accross workers

When you use thread workers, the data will simply be accessible to every thread directly. You can share it as a variable or as a global object.

some_global_data = pd.read_parquet("huge_file.parquet")

def work_function(arg): 
    # `some_global_data` and `arg` are coming straight form the memory
    # shared accross threads
    s = some_global_data[some_global_data.val == some_data].sum()
    return s

with ThreadPoolExecutor(10) as p: 
    results = list(tqdm(p.map(work_function, iterable())))

However, when you use process workers, the data shared to workers as arguments is most often pickled and shared as a string. This could quickly be an issue. For 2 reasons: - Pickling a big object and sending it to every worker can be very expensive - This will copy the object many times, which could harm the available memory

The global data is shared directly from the memory, but it is copied to each worker. Which is also harming the memory.

So what can we do ? Not a lot, there are no great mechanics (that I am aware of) to share data accross process workers in Python. It’s basically a work in progress: https://lukasz.langa.pl/5d044f91-49c1-4170-aed1-62b6763e6ad0/

There is still a trick you can use, but with varying degrees of success. It’s to simply share the object globally accross the workers.

When Python starts a process, it’s going to fork the main process. This means that all data will be copied to the child processes. But it’s going to do a copy on write. This means that the underlying data will still be read from the main process (even from a child process) and will only be copied when it changes. This means that you can share the data accross all the processes withtout any memory increases.

Unfortunately, this assumption does not hold very long in Python. Since Python modifies object for reference counting (and other) reasons behind the scenes, the object will soon be copied to the child process even if you don’t explicity modify it yourself.

Still, I’ve noticed that this often works and saves me when processing a huge object with many processes. I would then do something like this:

big_object = pd.read_parquet("big_file.parquet")

def work_function(arg): 
    # big object is in the global scope, and might not be copied to the
    # child process for a while. 
    s = big_object[big_object.val == arg].sum()
    return s

with ProcessPoolExecutor(10) as p: 
    results = list(tqdm(p.map(work_function, iterable())))

The problem:

You’re about to get on a plane to Seattle. You want to know if you should bring an umbrella. You call 3 random friends of yours who live there and ask each independently if it’s raining. Each of your friends has a 2/3 chance of telling you the truth and a 1/3 chance of messing with you by lying. All 3 friends tell you that “Yes” it is raining. What is the probability that it’s actually raining in Seattle?

A first way to solve it

You can solve the problem by applying the rule of bayes.

We are looking for the probability that it rains, given that we had 3 yes.

Through the bayes formula, this can be reformulated as [the probability that we have 3 yes, given that it rains] * [the probability that it rains] / [the probability of having 3 yes]

We want to get p(rain | 3xYes)

p(rain | 3xYes) = (p(3xYes | rain ) * p(rain)) / P(3xYes)        # Bayes formula

p(rain | 3xYes) = (((2/3)^3 ) * p(rain)) / P(3xYes)              # If it rains, there is (2/3)^3 chances of having 3xYes        

p(rain | 3xYes) = ((8/27) * p(rain)) / P(3xYes)

p(rain | 3xYes) = ((8/27) * p(rain)) / (P(3xYes | rain) p(rain) + P(3xYes | not_rain) p (not_rain))      # The chances of having 3xYes are the chances 
                                                                                                         # of having it when it rains, added to chances 
                                                                                                         # of having it when it does not rain

p(rain | 3xYes) = ((8/27) * p(rain)) / (8/27) p(rain) + (1/27) p (not_rain))

p(rain | 3xYes) = ((8) * p(rain)) / (8 p(rain) + ( 1- p (rain) ))

p(rain | 3xYes) = ((8) * p(rain)) / (7 p(rain) + ( 1 ))

The same problem as code

You can use the python library PyMC3 to model such interactions. Here for example, I start with a rain prior distribution with a probability of 0.5. I then construct the probability of hearing a “yes” as a transformation of that rain distribution. This gives me a Bernouilly distribution of the event “hearing a yes”.

This is the event that is observed 3 times in the riddle. This is represented by the observed = [1,1,1].

We can then solve that model with pm.sample, and sample the posterior distribution with pm.sample_posterior_predictive. This gives us an expected value for the rain of 0.8878 (this can vary as this is a random process). This is very close to what we computed above !

with pm.Model() as model: 
  # model the problem
  rain = pm.Bernoulli("rain", p=0.5) 
  p_yes = rain * 2/3 + (1-rain) * 1/3 
  likelihood = pm.Bernoulli("likelihood", p=p_yes, observed=[1,1,1]) 

  # solve it
  trace = pm.sample(5_000, chains=2, )
  samples = pm.sample_posterior_predictive(trace, var_names=["rain"], samples=10_000)
  
pm.traceplot(trace)

print("Posterior sampling of rain: ", samples['rain'].mean())

FutureWarning: In v4.0, pm.sample will return an `arviz.InferenceData` object instead of a `MultiTrace` by default. You can pass return_inferencedata=True or return_inferencedata=False to be safe and silence this warning.
  trace = pm.sample(5_000, chains=2, )
Multiprocess sampling (2 chains in 4 jobs)
BinaryGibbsMetropolis: [rain]

Sampling 2 chains for 1_000 tune and 5_000 draw iterations (2_000 + 10_000 draws total) took 11 seconds.

DeprecationWarning: The function `traceplot` from PyMC3 is just an alias for `plot_trace` from ArviZ. Please switch to `pymc3.plot_trace` or `arviz.plot_trace`.
  pm.traceplot(trace)
/Users/rodolphe_cambier/miniconda3/lib/python3.8/site-packages/arviz/data/io_pymc3.py:96: FutureWarning: Using `from_pymc3` without the model will be deprecated in a future release. Not using the model will return less accurate and less useful results. Make sure you use the model argument or call from_pymc3 within a model context.
  warnings.warn(

Posterior sampling of rain:  0.8953

Quantifying uncertainty

However, that solution seems incomplete to me. When we say we have a prior of p(rain) = 0.5, what is our confidence in that prior ? If we are absolutely certain, then new information should not influence us. If 0.5 was just a random guess, than we should only use new information.

For that reason, I think it is more complete to state the problem using distributions to represent our belief.

In order to do that, I will replace the 0.5 value in the rain prior by another distribution. This distribution, that I call p_rain, will represent my belief in the different potential values of p in the Bernouilli distribution.

I can now use a Uniform distribution to represent that I don’t know anything. I can use a more informative prior to represent some knowledge. I can even use the output of another Bayesian model as the input for this one !

If I use the Uniform prior, we can see that it is similar to using the p=0.5 from aboe in the Bernouilli distribution, and leads to an expected value of 0.889 for the rain event.

But what is very nice now, is that we have a visualisiation of our belief! In the p_rain graph that you see below, you can check for every value of the probability of rain what our belief is.

import pymc3 as pm
with pm.Model() as model: 
    # model the problem
    p_rain = pm.Uniform("p_rain", 0, 1) # this will now represent our belief
    rain = pm.Bernoulli("rain", p=p_rain) # we plug it in here
    p_yes = rain * 2/3 + (1-rain) * 1/3
    likelihood = pm.Bernoulli("likelihood", p=p_yes, observed=[1,1,1])

    # solve it
    trace = pm.sample(5_000, chains=2)
    samples = pm.sample_posterior_predictive(trace, var_names=["rain"], samples=10_000)

pm.traceplot(trace)

print("Posterior sampling of rain: ", samples['rain'].mean())

FutureWarning: In v4.0, pm.sample will return an `arviz.InferenceData` object instead of a `MultiTrace` by default. You can pass return_inferencedata=True or return_inferencedata=False to be safe and silence this warning.
  trace = pm.sample(5_000, chains=2)
Multiprocess sampling (2 chains in 4 jobs)
CompoundStep
>NUTS: [p_rain]
>BinaryGibbsMetropolis: [rain]

Sampling 2 chains for 1_000 tune and 5_000 draw iterations (2_000 + 10_000 draws total) took 14 seconds.

DeprecationWarning: The function `traceplot` from PyMC3 is just an alias for `plot_trace` from ArviZ. Please switch to `pymc3.plot_trace` or `arviz.plot_trace`.
  pm.traceplot(trace)
/Users/rodolphe_cambier/miniconda3/lib/python3.8/site-packages/arviz/data/io_pymc3.py:96: FutureWarning: Using `from_pymc3` without the model will be deprecated in a future release. Not using the model will return less accurate and less useful results. Make sure you use the model argument or call from_pymc3 within a model context.
  warnings.warn(

Posterior sampling of rain:  0.8882

Using an informative prior

We can also use a prior where we include some a-priori knowledge. For example, if we know that it rained 150 days last year, we could use a beta distribution with alpha=150 and beta=215 to represent that knowledge. (I don’t explain here why such a distribution would encode that knowledge. It could for example be the posterior of a previous modelling effort.)

The chances of rain are now 0.8445. This is less than before. Notice also how our p_rain distribution represents a different belief. Way more centered around 0.41.

with pm.Model() as model: 
  # model the problem
  p_rain = pm.Beta("p_rain", 150, 215) # This encodes our beliefs of how much it rains in Seattle
  rain = pm.Bernoulli("rain", p=p_rain) 
  p_yes = rain * 2/3 + (1-rain) * 1/3 
  likelihood = pm.Bernoulli("likelihood", p=p_yes, observed=[1,1,1]) 

  # solve it
  trace = pm.sample(5_000, chains=2)
  samples = pm.sample_posterior_predictive(trace, var_names=["rain"], samples=10_000)
  print("Posterior sampling of rain: ", samples['rain'].mean())
pm.traceplot(trace)

FutureWarning: In v4.0, pm.sample will return an `arviz.InferenceData` object instead of a `MultiTrace` by default. You can pass return_inferencedata=True or return_inferencedata=False to be safe and silence this warning.
  trace = pm.sample(5_000, chains=2)
Multiprocess sampling (2 chains in 4 jobs)
CompoundStep
>NUTS: [p_rain]
>BinaryGibbsMetropolis: [rain]

Sampling 2 chains for 1_000 tune and 5_000 draw iterations (2_000 + 10_000 draws total) took 11 seconds.

Posterior sampling of rain:  0.849

DeprecationWarning: The function `traceplot` from PyMC3 is just an alias for `plot_trace` from ArviZ. Please switch to `pymc3.plot_trace` or `arviz.plot_trace`.
  pm.traceplot(trace)
/Users/rodolphe_cambier/miniconda3/lib/python3.8/site-packages/arviz/data/io_pymc3.py:96: FutureWarning: Using `from_pymc3` without the model will be deprecated in a future release. Not using the model will return less accurate and less useful results. Make sure you use the model argument or call from_pymc3 within a model context.
  warnings.warn(

array([[<AxesSubplot:title={'center':'rain'}>,
        <AxesSubplot:title={'center':'rain'}>],
       [<AxesSubplot:title={'center':'p_rain'}>,
        <AxesSubplot:title={'center':'p_rain'}>]], dtype=object)