Master thesis project in applying model metric uncertainty in Amsterdam

The Wholesale Banking Advanced Analytics team is a large team of data scientists, data engineers, software developers and many more, that are focused on bringing data, machine learning and statistical modeling into the products that we build for our clients or internal users. The data scientists in WBAA furthermore have a strong desire to keep up with and be part of the latest developments in the fields of AI, tooling and statistics. Which they do by working closely together with master’s students on a variety of topics to solve academic yet practical problems.

In a binary classification setting we often associate different costs with different

types of errors. For example, in a medical setting you don’t want to wrongly tell a

patient that he or she has a disease when they don’t. But you rather make this error

than say that they don’t have a disease when they do. In other words, you want to

maximize the specificity, or recall on the negative class, for an almost sure sensitivity,

or recall on the positive class. Such a utility function can be optimized for any

model that outputs a probability, using the classification threshold. However, in

order to pick the optimal classification threshold for out-of-sample data we need

to consider the uncertainty on the metrics given the training-test run(s).

This brings the problem: how to determine the optimal classification threshold that

maximizes a utility function over a pair of classification metrics considering

their simultaneous uncertainty. A secondary problem would be the separation of

the data and model driven uncertainty, i.e. how can we estimate the moments of

the model and data driven error distributions.

Are you a master’s student looking for a thesis project and are you interested in solving this problem.

Do you furthermore

• Have solid experience with Python?
• Have solid skills in statistics and applied probability?
• Get at least six months to do your thesis project?
• Aim to go for a publication?
• Bring good vibes to your fellow data scientists?

Then we offer a master thesis project, a compensation of 600 euros per month, close supervision, and a tight community of data scientists to interact with and learn from.