The lettering was found by looking for 'crackle' texture on papyrus segments from the CT scans which obviously were in the shape of Greek letters, and annotating those as training data. Unfortunately such crackle texture isn't visible, at least by eye, on most of the papyrus. Probably it's only that visible where the ink was very thick. You can easily see the difference in texture in this electron microscope image [1] (far higher resolution than the CT scans) but especially on the very edge of the inked area (the narrow strip in the left image; I think the whole right image is inked) where the ink was pushed to. I'm surprised the crackle was discovered only after the Kaggle Ink Detection contest. Looking at the CT-scanned fragments with infrared ground truths, which were used in the Kaggle contest, Casey Handmer wrote [2]:
> The ongoing apparent failure of deep-learning based ink detection based on the fragments indicated to me that direct inspection of the actual data would be more fruitful, as it has been here.
> ...
> I found similar “cracked mud” and “flake” textures corresponding to known character ink, but only for perhaps 10% of the known characters. It’s been a long day, I can probably find more on closer inspection, but that does make one wonder about automated ink detection and what that is seeing.
These new images are much better than I hoped for, but still only in one small area, so I'm still pessimistic about more than an odd sentence being readable.
I actually participated in the challenge for a little while and this was the approach I took before I dropped out to do a few other things.
However, what I did was a bit different -- instead of looking for a crackle, I surmised that that 'crackling' effect actually is just of course slices of the data over different rifts in the parchment, and that the data of the ink lay on the manifold of that crackling and bending.
It would not be as clear to the human eye for all of the letters, I think, as there are many, many, many layers in the scanned image, and you can only start to see a pattern emerge over time as you cycle through the images.
I was working on code that minimized an optimization function that was basically the total variance loss if I recall correctly, where it just interpolated each pixel column up and down bilinearly to 'align' the blocks of the image so that the crackle texture was flattened.
From there I planned on using a rather optimized convolutional network on the 'flattened' image, which can I think be done rather efficiently as if you look at a cross section of the scroll you can see where it's like a tree in that the pinching and such seems to be somewhat locally consistent, so you might be able to get away with some interpolation.
I should probably share the code if this is of interest to anyone, since I'm not pursuing the competition at the moment.
Also, this is why I did not buy into 3D convolutions for this, at least. Ink that has been laid and dried should follow a semi-predictable pattern that a 2D convolution can detect, I do not know if a 3D convolution really brings us anything, as the invariances we desire can be structured up front more easily.
If there is interest in the code, let me know and I can do a little digging, otherwise, it is a fun challenge, for sure.
I'd studied the problem to work on it, but didn't get as far as you. I agree that intuitively (not backed up by experiment) I expect that preprocessing to further flatten the segments and other hand-crafted features based on desired invariances should work well. It seems that a lot of people really have just used 2D convolutions, applied to just one or a few surface layers and then combined. So I'd also be interested in your code.
> I surmised that that 'crackling' effect actually is just of course slices of the data over different rifts in the parchment, and that the data of the ink lay on the manifold of that crackling and bending.
I ended up cleaning it up to share it and got sucked back into the project. >:'(((( Spent 2 hours today in RawTherapee trying to improve scan contrasts :'((((
I hope to post a perhaps more cleaned up version of at least the dataset code, the aligmnent code is extremely messy and in the discord server (licensed under 'free to steal as many ideas as you want from' basically).
Unless you've, like, named all the variables with ethnic slurs, you're not going to get in trouble because you've just copied and pasted the same subroutine twelve times instead of refactoring it into something clean.
> The ongoing apparent failure of deep-learning based ink detection based on the fragments indicated to me that direct inspection of the actual data would be more fruitful, as it has been here.
> ...
> I found similar “cracked mud” and “flake” textures corresponding to known character ink, but only for perhaps 10% of the known characters. It’s been a long day, I can probably find more on closer inspection, but that does make one wonder about automated ink detection and what that is seeing.
These new images are much better than I hoped for, but still only in one small area, so I'm still pessimistic about more than an odd sentence being readable.
[1] https://scrollprize.org/img/tutorials/sem.png
[2] https://caseyhandmer.wordpress.com/2023/08/05/reading-ancien...