Introduction

Genshin Impact (henceforth Genshin) is a RPG (role-playing game) set in a fantastical universe. The player controls a party of 4 adventurers/characters (who they can swap out) and complete quests. Each character can be armed with weapons, and each has a different rank (1 star = common to 5 star = super rare), design aesthetics, and abilities.

Players open up ‘loot boxes’ to obtain characters to play as and weapons to use and progress the story. There are many loot boxes, and these all contain a ‘featured prize’ – a unique 5-star prize that is unique to a particular loot box – which players are often trying to get. The probability of getting characters and weapons are given below (please see this for a full list of rules, but they are not necessary for our limited challenge):

In order to protect users from overspending, the video game company (Mihoyo) implemented a protection mechanism where:

If the player doesn’t get a 5-star after opening 90 of the same boxes, the game will gift the player a 5-star character. (coded as pityCount under wishes.csv)
If the player doesn’t get a 4-star after opening 10 of the same boxes, the game will gift the player a 4-star character. (coded as rarePity under wishes.csv)

However, rumors are spreading that there are even more mechanisms in place in this lottery. Your goal as a data scientist is to identify if that is really the case, and in the process, find out other anomalies in the data which points to non-disclosed features of the lottery system.

Problem Statement

Methodology

1. Cleaned data to allow for accurate graphing

2. Created visuals for subsets of the data to search for obvious trends

3. Applied linear regression on these graphs

4. Created neural network to predict probability distribution of obtaining a rare or legendary item

5. Used interpretability analysis on the results of the neural networks to find how individual factors affected probability

Graphs

For the purpose of this experiment, we define pity as the number of boxes left to open before a rare/legendary item is guaranteed

Distribution of Pity Counts

As demonstrated by the plot above, the distribution of pity counts in relation to when legendary and rare items were obtained does not comply to the expected geometric distribution, thus we concluded there must be other factors at play. In the plots below, we attempt to elucidate this by visualizing comparisons between several factors.

Composite Boxes openefore before Legendary obtained and Rare obtained

The above two plots further reinforce the notion that the pity mechanism is not merely geometric; they peak just before the maximum pity value, which goes against the expected peak at the right end (since it represents the cumulative of the right tailed geometric distribution).

Boxes opened before Legendary obtained and Rare obtained (excluding guarantees - i.e. when the pity reached 0)

After removing the guarantees, we would expect the distribution to be geometric, since the probability is 0.006 for legendary and 0.051 for rare every time the box is opened. However, the fact that there is still a right peak clearly demonstrates there is a more complex mechanism.

The number of loot boxes that were open before a 4 or 5 star prize was recieved

In the above two plots, we attempted to compare the factors, grouping and bannerID, to the pity count through visualization methods but the differences are too subtle to be discerned with solely a visual analysis.

Neural Networks

We turned to statistical models to delve deeper into how these facts influence the pity mechanism

Although the plots above showed slight differences in the distributions when certain factors were changed, it's difficult to make substantial conclusions based on these visualizations alone. Thus, we turned to statistical models to more further explore this trend.

The first step was creating two datasets, which each take in the 5 factors supplied in the pull.csv data and output a probability distribution of the pity values at which an item can be expected. For the rare (four star) dataset, the output represents the probability a rare item will be obtained at that pity value. For the legendary (five star) dataset, the output is similar, but binned into groups of 10 to span the ful 0 to 90 possible pity values.

We first tried a simple linear regression model, which performed fairly well, predicting with 30% accuracy on the legendary dataset and 12% accuracy on the rare. This was significantly better than the base probability, but nevertheless we sought to further improve by turning to the more complex deep learning.

Due to the time constraint, we briefly explored some variations of architectures to optimize performance. These are some of the Neural Networks we tested.

After a brief hyper parameter search, we were able to achieve 47% accuracy on the legendary dataset and 36% accuracy on the rare dataset. However, this still raises the question of why can these models predict so well if the process is truly random? To answer this question, we performed a number of inteprretability analysis that aim to understand what the models learned and how they make predictions. The results of some are shown below; we identified the most important factors and examined how the model's predictions change with those factors.

Legendary dataset model

As demonstrated by the significant drop in accuracy when banner ID was removed from the model's inputs, we identified this to be critically important in the pity system. Thus, we further explored how banner ID impacts looting probabilities in the experiment below, where we visualized predicted distributions for several different banner IDs.

As illustrated above, the predicted probability distribution is shifted slightly in each banner ID, further validating that the banner ID, and thus the type of loot box, plays a role in the dispersion of rare and legendary loot; it is not solely determined by pity.

Conclusion

Our analysis shows that there is a hidden pity system that works behind the visible pity system. Once a player reaches 10 pity points, they are far more likely to obtain a legendary item. Before then, obtaining one of these items is nearly impossible.

Aside from this “soft pity,” Banner ID (type of loot box) and Wish ID (player ID based on individual player factors) are the most important factors in determining probability of obtaining a rare or legendary item. Wish ID is a function that is based on hidden factors, and plays a role in determining loot box chances. Other games with a loot box profit model have given players who spend more money better chances on loot boxes. This is an example of a factor that could play into Wish ID.

Members

Rohan
Karim 💅
Shayl
Justin