The Eigenschlumpkin Herder

Android Backups: A Yak Shave

Sun, 14 Oct 2018 00:00:00 +0000

The other day I finally got around to taking a backup of an ancient Nexus 7. A Stack Exchange answer by camalot lists a few methods for turning it into plain files, so I go ahead and try the one that doesn’t require me to download anything…

% { printf "\x1f\x8b\x08\x00\x00\x00\x00\x00"; tail -c +25 ../clipper2.adb; } | tar -xvz
...
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now

Uh-oh.

The device

Loose plugs cause bugs

The first few tries, I didn’t even get the “Backup finished” toast notification due to a loose USB cable. Turns out syslog messages like USB disconnect, address 58 indicate bad things when they’re timestamped suspiciously near the backup file’s mtime. Who knew?

adb logcat

But having solved that, did I still end up with a truncated file due to some badness on the device? Apparently misbehaving apps can cause it. Looking through the output of adb logcat -s BackupManagerService, at first things are pretty reassuring:

09-17 23:28:16.568   656  1510 V BackupManagerService: Requesting full backup: apks=false obb=false shared=true all=true system=true pkgs=[Ljava.lang.String;@7c66c2f
...
09-17 23:38:48.689   656  3792 D BackupManagerService: Full backup pass complete.

Googling finds users having problems with shared storage. Hm, that log message wouldn’t have been captured by -s BackupManagerService, so I try again with just the shared storage and indeed, there it is:

% adb backup -shared -f /home/pteromys/clipper5.adb
% adb logcat
...
09-18 00:08:16.363 21132 21143 I FullBackup: Unrecognized domain shared/0

That looks beyond my ability to fix. Fortunately I’m not seeing any other obvious problems in the log, so I just rerun the backup with -noshared, then pull the shared storage separately as plain files:

% adb backup -all -noshared -f /home/pteromys/clipper6.adb
% adb pull /sdcard /home/pteromys/clipper-backup-sdcard

When your tools think your devices are too old

As I’m closing out the accumulated windows this catches my eye elsewhere in the logs:

09-17 23:42:25.840  8500  8500 D AndroidRuntime: Calling main entry com.android.commands.bu.Backup
09-17 23:42:25.841  8500  8500 D bu      : Beginning: help
09-17 23:42:25.842  8500  8500 E bu      : Invalid operation 'help'
09-17 23:42:25.842  8500  8500 D bu      : Finished.
09-17 23:42:25.842  8500  8500 D AndroidRuntime: Shutting down VM

Riiiiight. So this was sort of a lie:

% adb help
...
backup/restore:
   to show usage run "adb shell bu help"
...

I’m guessing it’s because Nexus 7s stopped getting updates long before bu got its help message but I keep getting new releases of adb.

Anyway, mission still accomplished, right? Close remaining windows, unplug the USB cable, go to sleep, and finish this another day.

The file

The zlib file format according to pyflate

Would’ve been too easy if it had ended there, wouldn’t it? I get the same gzip message on clipper6.adb the next day. Maybe the zlib file format will have some clues? Calmarius’s guide to zlib + deflate explains that it’s (modulo header and footer) just a concatenated stream of blocks. Awesome, except…

blocks aren’t necessarily byte-aligned,
their boundaries are marked by an end-of-block symbol instead of listed up front so you have to scan the file to find them, and
it’s Huffman encoded so you can’t find the end-of-block symbol until you’ve decoded all the codewords before it.

Still, if you can do it, you can test a hypothesis: maybe gzip expects more because adb/bu didn’t bother to set the “this is the last block” bit on the last block. Neither gzip nor python’s zlib module divulges this kind of information about their internal state; but pyflate will, and in more detail than your most prolific friends on Facebook.

Maybe hack out the parts that write the decompressed data to a file, though; all we’re interested in is what it does to the end. And of course it’ll be slower than gzip, so go find a snack or something. And eventually…

The pyflate output

% python2 ~/.local/lib/python2/site-packages/pyflate.py =(echo -n '\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03'; tail -c +27 /home/pteromys/clipper6.adb)
...
new block at (580253004, 0)
last bit 0x0
deflate-blocktype 0 stored beginning at (580253004, 0)
raw block data at (580253004, 3)
new block at (580253009, 0)
last bit 0x1
deflate-blocktype 1 static huff beginning at (580253009, 0)
raw block data at (580253009, 3)
raw data at (580253009, 3) bits 3
found symbol 0xa3 of len 8 mapping to 0x11d
reading 0 extra bits for len
found symbol 0x0 of len 5 mapping to 0x0
reading 0 extra bits for dist
dictionary lookup: length 258 copy -1 num bits 13 data '\x00...\x00'
found symbol 0xa3 of len 8 mapping to 0x11d
reading 0 extra bits for len
found symbol 0x0 of len 5 mapping to 0x0
reading 0 extra bits for dist
dictionary lookup: length 258 copy -1 num bits 13 data '\x00...\x00'
found symbol 0xa3 of len 8 mapping to 0x11d
reading 0 extra bits for len
found symbol 0x0 of len 5 mapping to 0x0
reading 0 extra bits for dist
dictionary lookup: length 258 copy -1 num bits 13 data '\x00...\x00'
found symbol 0x23 of len 8 mapping to 0x11c
reading 5 extra bits for len
found symbol 0x0 of len 5 mapping to 0x0
reading 0 extra bits for dist
dictionary lookup: length 250 copy -1 num bits 18 data '\x00...\x00'
found symbol 0x0 of len 7 mapping to 0x100
eos 0 count 0 bits 7
end of Huffman block encountered
this was the last block, time to leave (580253017, 3)
Traceback (most recent call last):
  File "/home/pteromys/.local/lib/python2/site-packages/pyflate.py", line 678, in <module>
    _main()
  File "/home/pteromys/.local/lib/python2/site-packages/pyflate.py", line 648, in _main
    out = gzip_main(field)
  File "/home/pteromys/.local/lib/python2/site-packages/pyflate.py", line 629, in gzip_main
    final_length = b.readbits(32)
  File "/home/pteromys/.local/lib/python2/site-packages/pyflate.py", line 71, in readbits
    self.needbits(n)
  File "/home/pteromys/.local/lib/python2/site-packages/pyflate.py", line 37, in needbits
    self._more()
  File "/home/pteromys/.local/lib/python2/site-packages/pyflate.py", line 62, in _more
    c = self._read(1)
  File "/home/pteromys/.local/lib/python2/site-packages/pyflate.py", line 32, in _read
    raise "Length Error"
TypeError: exceptions must be old-style classes or derived from BaseException, not str

The breakdown

Here’s what we see in those last two blocks:

An empty uncompressed block, which ends with its length marker 0000ffff.
The next block starts with a 1 bit, so it was marked as the last block after all. So much for that hypothesis.
- Its contents are 258 + 258 + 258 + 250 = 1024 zeros, which reassuringly is how a tarball is supposed to end.
The end-of-block symbol gets read… then what is it trying to read after that?

The (zero-indexed) byte in which the empty block starts is 580253004 + 26 (bytes we chopped off with tail) - 10 (bytes in the fake header we added) + 2 (bytes that pyflate.py skips before starting its count), which comes out to 580253022. The file contents at this byte and forward are (recall tail is 1-indexed):

% tail -c +580253023 clipper6.adb | xxd
00000000: 0000 00ff ff1b 05a3 6014 8c5c 0000 df82  ........`..\....
00000010: f504                                     ..

Decompressing that final block by hand is an educational experience, but let’s skip that and pretend I trusted pyflate all along like I should have.

The last message before the exception tells us that the end-of-block symbol ends in the fifth-to-last byte, which leaves 4 bytes behind. The gzip footer is supposed to be 4 bytes of checksum and 4 bytes of uncompressed data size; maybe the size is all we’re missing? That would be very consistent with the final_length = b.readbits(32) in the traceback above…

% { printf "\x1f\x8b\x08\x00\x00\x00\x00\x00"; tail -c +25 ../clipper6.adb; } | zcat | wc -c
gzip: stdin: unexpected end of file
3734081536

% python -c 'print("%x" % 3734081536)'
de919000

% { printf "\x1f\x8b\x08\x00\x00\x00\x00\x00"; tail -c +25 ../clipper6.adb; printf "\x00\x90\x91\xde"; } | zcat > /dev/null
gzip: stdin: invalid compressed data--crc error

Ok, I can live with that for now. Let’s just try one more time with a fake size, to check that we really got the size right…

% { printf "\x1f\x8b\x08\x00\x00\x00\x00\x00"; tail -c +25 ../clipper6.adb; printf "\x00\x90\x91\xdf"; } | zcat > /dev/null
gzip: stdin: invalid compressed data--crc error
gzip: stdin: invalid compressed data--length error

Yay.

git-annex DIY visual cribsheet

Fri, 14 Apr 2017 00:00:00 +0000

git-annex lets git track files and delegates the actual storage to some other backend (plain files or a “special remote”). For your photo collection, you probably just want the git-annex assistant. But if not, here’s a cribsheet for everyday operations…

The cribsheet

Some assembly required.

…and now you see why it’s do-it-yourself. So…

Copy this onto some scratch paper¹ and skim through the git-annex walkthrough for the command that each arrow represents (a forked arrow counts as one arrow). Use the thick orange arrows for your files and the dotted purple arrows for the tracking info. (And that purple dot next to “remotes” represents a copy of the long purple arrow as a fork.)

Don’t be afraid to set up a test repo and remote somewhere in /tmp. Later, when it’s time to create a repo for a real-world use case, here are the quick hits…

Setup time analysis paralysis for a real repo

git config --global annex.genmetadata true
- This automatically records the creation month of each file annexed.
- For later reading: uses of metadata and autogenerating more metadata.
For each remote: any encryption? (none, hybrid, or other?)
- If yes, maybe you also want chunking.
Which hash to use: annex.backend (set in .gitattributes).
- The default SHA256E (SHA-2 of content and file extension) is probably fine.

Footnotes

I’d suggest you do this on a sticky note on your monitor, but sticky notes are kind of small. Anyway, it’s probably better if you lose it next week and have to do this exercise a second time—after that, you’ll have it memorized for sure. ↩

Seven½ volumes of n-balls

Wed, 01 Jun 2016 00:00:00 +0000

In a pale imitation of the Victor Guillemin style, here are seven ways to calculate the volume of an n-ball, in seven not-so-easy but somewhat spoiler-ful exercises. What’s your favorite? And can you shed any light on negative dimensions or the generating function in the last method?

Contents
Elementary inductions
Multivariable calculus and n-spheres
Without integration
Generating functions
Special values, just for fun

Elementary inductions

If you weren’t satisfied with being confined to dimension 3 when you learned the formula \[ |D^3| = \frac{4}{3} \pi \] for the volume of the unit 3-ball, then I suspect you eventually figured out that you could:

Method 1: Integrate over a 2-disc and induct from dimensions 0 and 1. (Walkthrough on Wikipedia)

If you were particularly bored in your calculus class, maybe you challenged yourself to use only one base case:

Method 2: Starting from a 1-disc instead, finish off the resulting integral with a trigonometric substitution and a reduction formula. (Walkthrough by Raulston)

Multivariable calculus and n-spheres

There’s a variant of Method 2 for the analogue of surface area, in which you calculate \[ |S^{n+1}| = \int_{x=-1}^1 |S^n| (1-x^2)^{\frac{1}{2}n} \, ds \] where \(ds = (1-x^2)^{-1/2} \, dx\) is the element of arc length for \(y = \sqrt{1-x^2}\). This is halfway to a purely spherical-coordinates method, which a little multivariable calculus lets you access:

Method 3: Carefully write down, in spherical coordinates, the element of \(n\)-volume in the unit \(n\)-sphere (the boundary \(S^n\) of the unit \((n+1)\)-ball). Then integrate to find \(|S^n|\). (Walkthrough on Wikipedia, using Beta functions instead of a reduction formula).

Or if you were bored in a probability or stats class, maybe you pulled some shenanigans with Gaussian integrals and the radius coordinate:

Method 4: Using spherical symmetry, write \(\left(\int_{-\infty}^{\infty} e^{-x^2} \, dx\right)^n\) as a product of \(|S^{n-1}|\) and a 1-variable integral that can be written as a value of the Gamma function. Then compute its value for all \(n\) by using polar coordinates in the case \(n=2\). (Walkthrough on Wikipedia)

A high-power variant of this method is to compute the Laplace transform of \(f(y) = |D^n| y^{n/2} = \int_{|x|^2 \leq y} dx\) and is detailed by Lasserre (paywalled).

Without integration

Wikipedia’s a great source for these walkthroughs, and it even includes mention of unit \(L^p\)-balls (with sources!). But let’s go off the trail a little bit. Our first stop goes way back—to Archimedes’s use of Cavalieri’s principle to avoid explicitly using calculus:

\(D^1 \times S^1\) is a cylindrical surface, the cone on which is a solid cylinder with two cones cut out.

Method 5: Calculate that over each point of \(D^n\), the 2-dimensional cross-sections of \(D^{n+2}\) and the cone on \(D^n \times S^1\) (with vertex at the origin) have the same area. Then note that \(|S^1 \times X| = 2\pi |X|\) and that passing to a height-1 cone divides volume by \(n+2\) (i.e. the dimension of the ambient space). (Walkthrough by Makarov, paywalled)

As with Method 2, there’s an \(S^n\) variant that deserves mentioning. (To recover the same recurrence as above, use the fact that \(D^{n+2}\) is the cone on \(S^{n+1}\) to relate their volumes.)

Method 5½: Calculate that projecting to the equatorial disc \(f: S^n \to D^n\) multiplies n-volume by a factor of \((1-|f(x)|^2)^{-1/2}\). Since the \(S^1\)-shaped strip of \(S^{n+1}\) over \(f(x) \in D^n\) has radius \((1-|f(x)|^2)^{1/2}\), conclude that \(|S^{n+1}| = 2\pi|D^n|\).

If you’re willing to think only in even dimensions (i.e. \(D^{2n}\)), here’s a super-cute method exploiting complex numbers that I first heard of from my advisor:

Method 6: Let \((D^2)^n\) denote the product of unit discs in \(\mathbb{C}^n\) (so its volume is \(\pi^n\)), and let \(E = \{ 0 < |z_1| < \cdots < |z_n| < 1\} \subseteq (D^2)^n\) be a fundamental domain under the action of permuting the coordinates. Check that \[ \begin{align*} f: z &\mapsto w \in \mathbb{C}^n \\ w_1 &= z_1 \\ w_i &= z_i \sqrt{1 - \left| \frac{z_{i-1}}{z_i} \right|^2} \end{align*} \] defines a volume-preserving diffeomorphism from \(E\) to a full-measure subset of \(D^{2n}\). (Walkthrough by Hijab, paywalled)

Generating functions

Method 5½ produces the recurrence relation \(|S^{n+1}| = 2\pi |D^n|\). Generating functions are a tool for solving recurrence relations; it’s overkill in this case, but you can do it:

Method 7: Let \( b(r) = \sum_{n=0}^\infty |D^n| r^n \). Show that \(b’(r) = \sum_{n=0}^\infty |S^n| r^n\), and use the recurrence above to conclude \(b’ = 2 + 2\pi r b\). This is a linear ODE; solve it to get \[b(r) = e^{\pi r^2} \left(1 + 2\int_0^r e^{-\pi t^2}\, dt\right),\] and expand it as a power series to recover the coefficients \(|D^n|\). (Walkthrough by Badger, paywalled)

Cook remarks that there ought to be some way to explain the occurrence of the Gaussian in the formula for \(b(r)\); you can get part of the way there with the following variant.

Method 7½: The Laplace transform of \(f(x) = e^{-x^2}\) is \[ \mathcal{L}\{f\}(y) = \int_0^\infty e^{-x^2} e^{-xy} \, dx . \] Expand \(e^{-xy}\) as a power series, and use the techniques of Method 4 to write the \(y^n\) coefficient of \(\mathcal{L}\{f\}(y)\) in terms of \(|S^n|\). (Yes, you can swap the integral and the sum since the sum of integrals converges.)

The wrinkle here is that \(|S^n|\) ends up in the denominators of the coefficients, while a substitution in the formula for \(b(r)\) shows that \[ b(r) = \mathcal{L}\{x \mapsto e^{-\pi x}\}(-2\pi r) . \] Alas, I don’t see how to get more directly from this Laplace transform to \(b(r)\); but on the bright side, equating the coefficients in the two variants provides a roundabout way of proving the Gamma function’s duplication formula at integers, \[ \Gamma(n)\Gamma\left(n+\frac{1}{2}\right) = 2^{1-2n} \sqrt{\pi} \, \Gamma(2n) . \]

Special values, just for fun

Trivial coefficients occur with \(|D^0| = 1\), \(|D^2| = \pi\), and \(|S^5| = \pi^3\), which makes me think the Raspberry Pi Foundation missed a minor branding opportunity earlier this year. Or if you’re a tauist, you also get \(|S^1| = \tau\). For something approaching a tauist spiritual experience, go back and replace all occurrences of \(\pi\) with \(\frac{1}{2}|S^1|\).

Maybe you noticed that the Gamma function is still defined at negative half-integers, which goads you into asking whether \[ |S^{-2}| = \frac{2 \pi^{\frac{-2+1}{2}}}{\Gamma\left(\frac{-2+1}{2}\right)} = -\frac{1}{\pi} \] has any meaning. Despite having stumbled over negative-dimensional spheres in a quest to convince myself that (-2)-categories are a thing, I’m leaning toward: probably not—at least, not in any sense that looks obviously to me like a moral analogue of volume. But if you’ve got ideas, I’m listening.

Simplicial whatever quickstart guide

Wed, 25 May 2016 00:00:00 +0000

Simplicial sets are useful for… okay, I really have no idea what. I just got sucked into reading about them. Of many available references, I enthusiastically recommend Friedman’s notes; the quality of exposition is totally unparalleled. The following are merely my notes on the differences between simplicial complexes, delta complexes, and simplicial sets.

Complexes

A 3-D simplex.

So you want to jam simplices together to make shapes. Maybe you’re building a 3-D model out of triangles for a video game. Because we attempt to understand things via their sub-things, the following assumptions are convenient, and thus shared by all the complexes we’re about to discuss:

Each simplex comes with all its sub-simplices (itself, faces, vertices, etc), and
The intersection of simplices is always a union of their sub-simplices.

Simplicial complexes

To recover the definition of a simplicial complex, just require:

Additional assumption for simplicial complexes: Each simplex can be specified uniquely by the set of its vertices.

This is a union of simplices in the nicest way possible; and video game model formats such as MD2 actually specify their simplices as lists of vertices.

Delta complexes

Sometimes you want to do more than play video games. It being May, maybe you wanted to pick some flowers.

A flower, not colored in.

This is still made of simplices (a point and some (curved) line segments); but a set of vertices no longer determines the simplex. We attempt a generalization:

Delta complex data model: For each simplex, list not just the 0-subsimplices, but all the subsimplices.

Then the above flower can be specified with the following information:

a point \(P\);
five line segments \(a\), \(b\), \(c\), \(d\), and \(e\); and
the left endpoint of \(a\) is \(P\) and so is the right endpoint;
…and similarly for the other four line segments.

Organizing the delta complex data model

Data needed to describe the flower above

A set of 0-simplices (points)
A set of 1-simplices (line segments)
A “left endpoint function” taking 1-simplices to 0-simplices
A “right endpoint function”, similarly

In higher dimensions

From the 2-simplices we’d need 3 edge maps and 3 vertex maps, along with some compatibility conditions like how some vertices of some edges are vertices of some other edges—in other words, the subsimplex functions have to compose the same way they do in a typical simplex. If we call that a “reference simplex” and encode it as a category \(\widehat{\Delta}^{\mathrm{op}}\) that looks like this:

The reference simplex.

…then we can define a delta complex to be a functor from \(\widehat{\Delta}^{\mathrm{op}}\) to the category of sets.

Recovering the usual definition of \(\widehat{\Delta}\)

A subsimplex of the reference simplex is determined by the set of its vertices; so the arrows in \(\widehat{\Delta}^{\mathrm{op}}\) can be expressed by reversing inclusions of sets of vertices. Then it’s just a matter of realizing that we can make there be exactly one arrow per face inclusion by ordering the vertices and requiring the set inclusions to be increasing functions.

Simplicial sets

Making a degenerate 3-simplex.

To go from Delta complexes to simplicial sets is to allow “degenerate simplices”—simplices that have secretly collapsed down to a lower dimension. This is encoded as a category \(\Delta\), obtained from \(\widehat{\Delta}\) by replacing “increasing” with “nondecreasing”. Significant consequences of this minor-looking change include:

Every simplex now comes with all its possible degeneracies. So that nice triangle mesh from your favorite video game becomes full of squashed 3-simplices and 4-simplices and 5-simplices and…
On the bright side, those degenerate simplices make it possible to define products of simplicial sets in a way that looks too good to be true. A good picture of why it works is in Chapter 5 of Friedman’s notes; but I have to leave it to you to come up with a pithy colloquial explanation that you personally find satisfying.

Exercise: As simplicial sets, how does the flower in Figure 2 differ from a point? (That is, what property of the degeneracy function from 0-simplices to 1-simplices distinguishes them?)

What’s still missing

Geometry textbooks are full of wacky instructions like “glue this face to that face with an \(x\)-degree twist”; but there’s no freedom to do so on the faces of a simplex—each face comes with an ordering of its vertices, which determines the only way they can be glued together. Even degeneracies don’t help, owing to the “nondecreasing” condition.

Really, we shouldn’t expect simplicial sets to allow arbitrary reordering when gluing faces together. That would allow gluing a face of a face to itself with arbitrary reordering—which amounts to taking a quotient by some subgroup of permutations of vertices. If you need that kind of flexibility, it’s probably time to invoke good ol’ barycentric subdivision.

A survey of underlines in HTML

Mon, 09 Nov 2015 00:00:00 +0000

So I was preparing my résumé the other day, because it’s job season. Something sounded elegant about having it in a texty format like LaTeX or HTML. Since the LaTeX community contains a few too many prescriptivists for my taste, HTML it was. Soon, having learned fascinating nuggets like how reluctant browsers are to do subpixel drawing, I began to regret not using this as an excuse to learn Scribus.

But as much as I would love to tell you about my regrets, I meant instead to write a post about drawing pretty underlines. Check out the way Epiphany’s underline skips the descender on the “p”:

Chrome 46 (left) vs. Epiphany 3.18 (right)

Native support in browsers

It turns out that a text-decoration-skip:ink has been in a Candidate Recommendation since August 2013. There’s no support yet in ~~Chrome or~~¹ Gecko; but in November 2013, only seven months after Blink forked from WebKit, support landed in WebKit. The following January, it got made the default—in time for the release of iOS 8 and Safari 8 that fall.

I haven’t been able to find the discussion of why this was the right thing to do. In fact, Comment 11 in that last link noted that the change deviates from the specification, provoking a so-far-unfinished effort to follow the spec more closely. But given the enthusiasm with which the change was embraced by observers of iOS 8, it may be here to stay.

Non-native implementations and line thickness

Back in early 2014, descender-skipping was a part of Marcin Wichary’s tremendous effort on underlines at Medium that included work to control the thickness of underlines. (Despite a noticeable need for thin underlines in light font-weights, support for controlling line thickness is nonexistent in the spec and in browsers. And if you searched for such on Google, this may also be an opportunity to lament how quickly HTML answers become dated on Stack Overflow—like this one.)

Though the skipping part was dropped from Wichary’s final version, the technique of using a gradient background and text-shadow has inspired a number of other implementations, including at least two libraries you could use:

Adam Schwartz’s SmartUnderline covers descender-skipping and line thickness with no effort on your part.
Wenting Zhang’s Canvas-powered UnderlineJS is carefully thought out, down to the sizes of the holes around descenders. And the live demo makes sounds!

Obstacles to adoption

First, skipping descenders might not actually be the right move; going against the grain of iOS, Ilya Birman argues for thin, uninterrupted link underlines on semantic grounds. A thread on HN indicates a number of commenters share this viewpoint (and also points out possible prior art by Roman Komarov).

Meanwhile, I’m a little bothered by how much closer the “o” is to the “m” than the “t” in Epiphany’s rendering of “motions”. Maybe that’s just because I’m used to looking at Chrome; but until I learn more about kerning, I guess I’m sticking with using Chrome to render my résumé to PDF.

Still, I’ll keep the above in mind in case I ever need to underline mathematical formulas with subscripts. Somehow my handwritten notes are full of such, while the concept has never even occurred to me for typeset math. (There’s a good chance it’s a bad idea.)

A footnote on LaTeX and prescriptivism

Okay, it’s not really the prescriptivism that I mind so much as the total lack of interest in justifying prescriptions. For example: despite widespread knowledge that footnotes in tables are considered bad style, (1) nobody seems to know why, and (2) there’s a profusion of implementations. Do these seem causally related to you?

Admittedly, the OP was probably unwise to put two questions in one post, since accepting an answer for #2 made #1 all but invisible. And on the bright side, it could have been less civil. (Then again, the postfix thread has a happy ending, which the SE thread can’t quite claim.)

Updates

Chrome support was added on November 19, 2016. ↩

What is a quasicrystal?

Fri, 30 Oct 2015 00:00:00 +0000

This post was originally a talk given in Pizza Seminar on October 15, 2014. Most of the mathematical content is from Wikipedia and this paper by Niizeki.

Abstract

The crystallographic restriction theorem says a finite-order automorphism of \(\mathbb{Z}^3\) (in chemistry language, a symmetry of a crystal) has order 1, 2, 3, 4, or 6. But \(\mathbb{Z}^3\) is small potatoes; so let’s go to \(\mathbb{Z}^n\), do some Galois theory in cyclotomic fields, and get us some icosahedral symmetry! (Icosahedrite is a real thing, say serious scientists.) Pretty pictures will be provided, including demonstrations of self-similarity. Self-similarity is always cool, right?

Abstract
Contents
The crystallographic restriction theorem
X-ray crystallography: peeking inside real crystals
Slicing higher-dimensional lattices
Self-similarity
1. A number-theoretic model of quasicrystals
2. A mathematical description of zooming
3-D quasicrystals
1. The n-dimensional viewpoint
2. The search for 3-D quasicrystals
Footnotes

The crystallographic restriction theorem

A crystal is a regular grid (like \(\mathbb{Z}^3\)) of atoms. There is a remarkable theorem about what symmetries a crystal can admit:

Theorem (crystallographic restriction theorem)

If \(A \in \operatorname{Aut} \mathbb{Z}^3\) has finite order, then the order is 1, 2, 3, 4, or 6.

Proof

Properties of \(A\) imply properties of its eigenvalues:

\(A\)	eigenvalues
finite order	on the unit circle \(S^1\)
real	in complex conjugate pairs (so one is \(\pm 1\))
integer	their sum, \(\operatorname{tr} A\), is an integer

So if \(e^{i\theta}\) is an eigenvalue of \(A\), then \[ \operatorname{tr} A = e^{i\theta} + e^{-i\theta} \pm 1 = 2 \cos \theta \pm 1. \] Therefore \(\cos \theta \in \frac{1}{2} \mathbb{Z}\). The eigenvalues \(e^{i\theta}\) satisfying this are of order 1, 2, 3, 4, and 6.

Eigenvalues on the unit circle with half-integer real part

X-ray crystallography: peeking inside real crystals

Setup

Here’s how scientists figure out the internal structure of a crystal. To start, shine an x-ray on it and record the scattered radiation on film. For our purposes, an x-ray is a function \(f\) satisfying \[ f(x+?) = e^{i k \cdot x} f(?) \] for some vector \(k\) that depends on the ray direction and wavelength. This lets us calculate the factors by which \(f\) changes from source \(x_1\) to atom \(x\) and from atom to film \(x_2\).

X-rays bouncing off an atom

The total change in \(f\) is by a factor of \( e^{i[(k_1 - k_2) \cdot x - k_1 \cdot x_1 + k_2 \cdot x_2]} . \) Integrating this over all atom positions \(x\) in the crystal yields the total radiation bounced in the direction of \(k_2\). So…

Upshot: The photo records (part of) the Fourier transform of the distribution of atoms.¹

A troublesome photo

Since rotational symmetries of a lattice are inherited by its Fourier transform, photos like the one below would seem to violate the crystallographic restriction theorem.

X-ray photo with order 10 symmetry
by Materialscientist on Wikimedia Commons (CC-BY-SA-3.0)

Demo: Making waves

To get a picture of what kind of crystal we’re looking at, let’s approximate the inverse transform by adding waves at \(n\) regularly spaced angles. Try it for 5, 8, 12, and 7.

Demo
Sums of waves

You can animate these too; and Jason Davies has a WebGL adaptation of the animation.

Slicing higher-dimensional lattices

The problem: locating atoms

The last demo showed you the graph of \(f(x) = \displaystyle\sum_{i=1}^n \cos (k_i \cdot x) \) in \(\mathbb{R}^2\) for evenly spaced \(k_i\) on the unit circle. This is all very nice and smooth, but we want to actually place atoms—discrete points—in a grid. Let’s put them in the hotspots—i.e. where \(f\) hits some threshold.

To turn this into a discrete process, use the following trick: realize the waves \(\cos (k_i \cdot x)\) as 2-D slices of independent waves in \(\mathbb{R}^n\)—then hotspots are where the slice passes near lattice points.

The explicit recipe

Let \(f\) be the restriction of the function \[ \mathbb{R}^n \ni x \mapsto \sum_{i=1}^n \cos (e_i \cdot x) \] to a 2-plane on which the basis vectors \(e_i\) project to \(k_i\). To choose the 2-plane, let \(A \in \operatorname{GL}_n \mathbb{Z}\) act by cycling the basis vectors \(e_i\). Diagonalizing over \(\mathbb{C}\) gets you the matrix \[ \begin{pmatrix} \zeta & & & & \\ & \zeta^{-1} & & & \\ & & \zeta^2 & & \\ & & & \zeta^{-2} & \\ & & & & \ddots \end{pmatrix} \] where \(\zeta = e^{2 \pi i / n}\). Over \(\mathbb{R}\), the eigenvalues pair into 2×2 blocks, corresponding to 2-planes on which \(A\) acts as rotation by a multiple of \(\frac{2\pi}{n}\). Pick any one of them.

Upshot

The 2-plane is a 2-dimensional subrepresentation \(V\) of \(\langle A \rangle \curvearrowright \mathbb{R}^n\).
Atoms (hotspots) are points of \(V\) that lie kinda near points of \(\mathbb{Z}^n\).

Demo: Atoms in 2-D quasicrystals

With this in hand, a computer can be convinced² to find the atoms for you, drawing larger dots for better hotspots. Try it for several values of \(n\); I suggest 5, 7, 8, 6, 2, 12, and 11.

Demo
2-D quasicrystals

Remark: An alternative method is to slice fundamental domains (\(n\)-cubes) with the 2-plane, which yields pictures like the Penrose tiling below. Jason Davies has an applet you can play with, based on an older one by Greg Egan.

A Penrose Tiling
by Inductiveload on Wikimedia Commons (PD)

Self-similarity

At this point, we could go home—we have a description of quasicrystals from which we can produce both interactive computer renderings and theorems (like quasicrystals not actually having translational symmetry, due to \(V\) containing no integer points besides 0).

Of course, there is more—the “quasi”-ness of quasicrystals allows a sort of self-similarity that honest crystals can’t have. To see it in action, go back to that atoms demo and try zooming in or out for a while.

To explain this self-similarity—the apparent indistinguishability of different zoom levels—we will need a bit of number theory.

A number-theoretic model of quasicrystals

Theorem (crystallographic restriction in arbitrary dimensions): \(\operatorname{GL}_m \mathbb{Z} \) contains a matrix \(A\) which has \(\zeta_n = e^{2\pi i/n}\) as an eigenvalue (so it rotates some 2-plane by \(\frac{2 \pi}{n}\)) if and only if \(m \geq \varphi(n)\), where \(\varphi\) is Euler’s totient function.

Proof

The Galois conjugates of \(\zeta_n\) must also be eigenvalues of \(A\).

Since \(\zeta_n^n = 1\), they’re \(n\)th roots of 1.
Since \(\zeta_n\) generates all \(n\)th roots of 1, they’re primitive—i.e. \(\zeta_n^k\) where \(k\) and \(n\) are coprime.

The number of such \(k\) is \(\varphi(n)\). To prove that \(\varphi(n)\) works, observe that \[ \mathbb{Z}[\zeta_n] \hookrightarrow \mathbb{Q}(\zeta_n) \cong_{\mathbb{Q}\text{-vect}} \mathbb{Q}^{\varphi(n)} \subset \mathbb{R}^{\varphi(n)} \] as a discrete subgroup, since the field norms of the leftmost terms are in \(\mathbb{Z}\).

Remark: If \(p\) and \(q\) are large enough primes, you can produce an integer matrix \(A\) of order \(pq\) and size \(\varphi(p) + \varphi(q) < \varphi(pq)\), but it won’t rotate any 2-plane by \(\frac{2\pi}{pq}\).

A mathematical description of zooming

Here’s how we implement³ zooming: scale the 2-plane \(V\) by some factor and the unseen \(m-2\) dimensions by some other factor to preserve \(m\)-dimensional volume. That way the total visual density of dots remains about the same. Here’s how we interpret zooming and self-similarity:

Zooming acts by scalars on \(V\) and the other irreducible sub-representations of \(\langle A \rangle \curvearrowright \mathbb{R}^{\varphi(n)}\), so zooming is a 1-parameter subgroup \(\gamma\) in \(\operatorname{SAut}_A \mathbb{R}^{\varphi(n)} \) (volume preserving automorphisms).
Visually, transforming by anything in \(\operatorname{Aut} \mathbb{Z}[\zeta_n]\) isn’t detectable.
Self-similarity means that the image of \(\gamma\) in \(\operatorname{SAut}_A \mathbb{R}^{\varphi(n)} / \operatorname{Aut} \mathbb{Z}[\zeta_n]\) accumulates on itself (or is periodic).

The existence of self-similarity rests on one key fact from number theory.

Theorem (Dirichlet’s Unit Theorem)

Suppose a number field has \(r\) embeddings in \(\mathbb{R}\) and \(c\) embeddings in \(\mathbb{C}\). Then its group of units has rank \[ r + \frac{c}{2} - 1 . \]

Corollary

\(\operatorname{SAut}_A \mathbb{R}^{\varphi(n)} / \operatorname{Aut} \mathbb{Z}[\zeta_n]\) is compact.

Proof

Since automorphisms act by scalars on all irreducible subrepresentations,

\[\begin{align*} \operatorname{Aut}_A \mathbb{R}^{\varphi(n)} &\cong (\mathbb{C}^\times)^{\frac{\varphi(n)}{2}} \cong (\mathbb{R} \times S^1)^{\frac{\varphi(n)}{2}} \\ \operatorname{SAut}_A \mathbb{R}^{\varphi(n)} &\cong \mathbb{R}^{\frac{\varphi(n)}{2} - 1} \times (S^1)^{\frac{\varphi(n)}{2}} . \end{align*}\]

Since \(\mathbb{Z}[\zeta_n] = \mathcal{O}_{\mathbb{Q}(\zeta_n)}\), Dirichlet’s Unit Theorem implies that \(\operatorname{Aut} \mathbb{Z}[\zeta_n]\) contains a copy of \(\mathbb{Z}^{\frac{\varphi(n)}{2} - 1}\). Then \(\operatorname{Aut} \mathbb{Z}[\zeta_n]\), being discrete (it’s conjugate to a subgroup of \(\operatorname{SL}(\varphi(n), \mathbb{Z})\)), is cocompact.

Self-similarity now follows, having been reduced to the statement that a 1-parameter subgroup of a torus is recurrent.

Remark: The image of \(\gamma\) is periodic when (and only when) \(\varphi(n) = 2\)—so when \(n\) is 5, 8, 10, or 12. This makes the self-similarity exact—i.e. different zoom levels can look identical (not just suspiciously similar)—and allows one to build games like Richard Schwartz’s Lucy and Lily based on facts like \[ (2 + \sqrt{5})(2 - \sqrt{5}) = -1. \]
Remark: Michael Rule’s collection of quasicrystals graphics includes an alternative description of zooming that sidesteps the number theory, instead using Shepard Tones to produce self-similarity.

3-D quasicrystals

The n-dimensional viewpoint

A quasicrystal with irreducible symmetry \(G\) is an irreducible-over-\(\mathbb{R}\) subrepresentation \(V\) of an irreducible-over-\(\mathbb{Z}\) representation \(G \hookrightarrow \operatorname{SL}(m,\mathbb{Z})\). The atoms are points of \(\mathbb{Z}^m\) near \(V\), and the Galois conjugates of \(V\) (the other subrepresentations over \(\mathbb{R}\)) allow the possibility for self-similarity.

The search for 3-D quasicrystals

To look for 3-dimensional quasicrystals, one first finds a list of the seven irreducible finite subgroups of \(\operatorname{SL}(3,\mathbb{R})\). Most of these preserve an ordinary 3-D lattice—i.e. the representation can be defined over \(\mathbb{Z}\). The two exceptions are the two icosahedral groups—the alternating group \(A_5\) and the product \(A_5 \times \{\pm 1\}\)—so up to finite index subgroups, there’s only one irreducible 3-D quasicrystal, which you can explore in the following demo.

Demo
3-D quasicrystals

The standard representation of \(A_5\) has one Galois conjugate, which can be found in a list of representations of \(A_5\). The two representations can be described explicitly as follows: in the standard representation, \(A_5\) acts by rotation on the icosahedron with vertices \(\pm(\sqrt{5}, 0)\) and \(\pm(1, 2\zeta_5^k)\) in \(\mathbb{R} \times \mathbb{C}\). To obtain the other representation, merely conjugate the action on the set of vertices by the permutation induced by \(\zeta_5 \mapsto \zeta_5^2\).

Oh, and—yes, they’re real.

A Ho-Mg-Zn quasicrystal over a millimeter grid
by AMES Lab., US DoE via Wikimedia Commons (PD)

Footnotes

The photo loses phase information, the recovery of which is a nontrivial problem. ↩
For efficiency reasons, the demo actually uses the number-theoretic model from the section after. ↩
Self-similarity is what makes zooming possible (i.e. not unbearably slow) in the atoms demo—the atom finder can get away with exploring a 2-dimensional slice instead of the entire \(m\)-dimensional space. ↩

Sol geometry and the tangent bundle of a hyperbola

Tue, 21 Jul 2015 00:00:00 +0000

If you try to make rigid motions in some dimension act as transformations of a lower-dimensional space, you might hope that it respects some structure in the lower-dimensional space. Unfortunately this is guaranteed for neither distances nor angles. Here are some pictures—one of which is interactive (!)—for a particularly pretty example using Thurston’s Sol geometry.

\( \newcommand{\R}{\mathbb{R}} \newcommand{\H}{\mathbb{H}} \newcommand{\SO}{\operatorname{SO}} \newcommand{\Sol}{\operatorname{Sol}} \)

Setup

Definition: \(\Sol^3\) is the Lie group \(\R^2 \rtimes \SO(1,1)^0\).

Visualize \(\Sol^3\) as a line’s worth of copies of \(\R^2\), where moving up or down in the stack stretches them in one direction and shrinks them by the same factor in the other.

\(\Sol^3\) is a stack of planes. (Interactive)

We are interested in the action of \(\Sol^3\) on the tangent bundle \(T\H^1\) of one branch of a hyperbola.

Description of the action

Let \(\mathcal{F}\) be the 1-D foliation spanned by \((\cosh z, \sinh z, 0)\) at \((x,y,z)\). Under the metric \(dx^2 - dy^2\) on a plane \(\{(*,*,z)\}\), the line tangent to \(x^2 - y^2 = 1\) at \((\cosh z, \sinh z, 0)\) is orthogonal to \(\mathcal{F}\). This identifies \(\Sol^3/\mathcal{F}\) with the tangent bundle \(T\H^1\) of one branch of \(x^2 - y^2 = 1\).

A tangent line to a hyperbola

Since \(\mathcal{F}\) is left-invariant (being spanned by a left-invariant vector field), the left action of \(\Sol^3\) on itself descends to \(T\H^1\). Letting \((p,q)\) denote a tangent velocity of \(p\) at \(q \in \H^1\), you can check that this action is given by \[ \begin{align*} (0,0,z)(p,q) &= (p,q+z) \\ (x,y,0)(p,q) &= (p+y \cosh q - x \sinh q, q) . \end{align*} \]

A diffeomorphism and the image of the q-axis

No preservation of distances or angles

Were we hoping to preserve either in \(T\H^1\), we’d be out of luck—the stabilizer of \((0,0)\) consists of maps \[ (p,q) \overset{(x,0,0)}{\mapsto} (p - x \sinh q, q) , \] whose action on the tangent space \(T_{(0,0)}(T\H^1)\) is by matrices \[ \begin{pmatrix} 1 & * \\ & 1 \end{pmatrix}. \]

Diffy queues

You can check using the formula that \(\Sol^3\) acts faithfully on \(T\H^1\). In fact, here’s a nice description of \(\Sol^3\) as its image in \(\operatorname{Diff} T\H^1\).

Proposition: The image of \(\Sol^3\) in \(\operatorname{Diff} T\H^1\) consists of the diffeomorphisms \(f = (f_p, f_q)\) that satisfy \[ \begin{align*} \frac{\partial f_p}{\partial p} &= 1 & \frac{\partial^2 f_p}{\partial q^2} &= f_p \\ \frac{\partial f_q}{\partial p} &= 0 & \frac{\partial f_q}{\partial q} &= 1 . \end{align*} \]

Recovering the group

To put it another way, the image consists of those diffeomorphisms expressible as \(P \circ Q_z\) where

\(Q_z\) is translation by \((0,z)\), and
\(P\) adds to \(p\) some linear combination of \(\cosh q\) and \(\sinh q\) (or of \(e^q\) and \(e^{-q}\)).

To see that this is \(\Sol^3\), parametrize the space \(V\) of images of the \(q\)-axis as \[ V = \{ \text{graph of } (q \mapsto xe^q + ye^{-q}) \mid x,y \in \R \} . \] Then \(Q_z\), sending \(q\) to \(q+z\), acts on \(V\) by \( Q_z (x,y) = (xe^z, ye^{-z}) \), so \[\Sol^3 = V \rtimes \{Q_z \mid z \in \R\} . \]

Remark: This is an alternative parametrization of \(\Sol^3\) under which the action on \(T\H^1\) becomes \[ (x,y,z)(p,q) = (p + ye^{q+z} - xe^{-q-z}, q+z) . \]

window.requestAnimationFrame troubles

Thu, 09 Oct 2014 00:00:00 +0000

Some old posts about window.requestAnimationFrame (like this one) depend on the callback taking, as an argument, the time the next frame will be displayed. Unforunately, as some have noticed, they’re kind of wrong: it’s the time the frame started processing. Here’s what I gathered when I tried to figure out why…

The current state of affairs seems to be able to differ from what MDN and the spec say only in Webkit on Cocoa (OS X and iOS), but testing on an iPhone and a Macbook gives results in line with the spec.

By May 2011, it was noticed that the spec didn’t actually define the callback’s argument unambiguously.

In November, discussion arrived at the following consensus:

The working group feels the time parameter should be the time of the next frame. As developers will typically want to construct the next frame to be painted, this seems like the intuitive option. James or Cameron should also follow up with CSS animations and other animation specs and ensure this design is consistent with any similar concepts they may have.

In May, the relevant section of the spec was updated to match what the actual behavior had been all along in IE, Chrome, and Firefox.

(In March, plans to make the time parameter a DOMHighResTimestamp had moved forward, but this didn’t actually have any effect on the semantic meaning of the callback parameter.)

I have to wonder whether Paul Irish’s message approving local high-resolution timestamps was also intended to approve of this change, from “next frame display time” to “started processing”, given some seemingly-mostly-ignored comments from outside developers on the mailing list (1, 2).

The last word on the issue was that the predicted time of refresh would likely be passed as a second parameter. Alas, this could conflict with intentions to depend on element visibility; though this post of Paul Irish suggests that intention’s gone.

Side note: The MDN docs were only updated to the spec this January, even though info on DOMHighResTimestamp was there in August 2013 when the relevant change landed.

Highlights of Simon Tatham's puzzle collection

Thu, 05 Jun 2014 00:00:00 +0000

I recently discovered Simon Tatham’s remarks on writing friendly (i.e. guaranteed solvable) puzzle games. Of note:

a variant of minesweeper that never requires you to guess
a pile of proofs in net.c, along with tactics to generate uniquely solvable boards for a game you might know as KNetWalk.
untangle.c, a reimplementation of Planarity with a different graph generation method (see static char * new_game_desc).

Glue records for Melon Island fixed

Wed, 26 Feb 2014 00:00:00 +0000

Last night I noticed that my website’s had incorrect glue records at *.gtld-servers.net since probably mid-2012—which explains why a number of ISPs weren’t resolving it at all!

Paul Young explains what this means, and SmallClanger remarks in this ServerFault answer that it isn’t supposed to happen with modern software. The issue was probed using Squish and dig; and fixed by changing the nameservers listed at the registrar, forcing the glue records to be rewritten. Correctness was checked afterwards using whatsmydns.net.

The Eigenschlumpkin Herder

Android Backups: A Yak Shave

The device

Loose plugs cause bugs

adb logcat

When your tools think your devices are too old

The file

The zlib file format according to pyflate

The pyflate output

The breakdown

git-annex DIY visual cribsheet

The cribsheet

Setup time analysis paralysis for a real repo

Footnotes

Seven½ volumes of n-balls

Contents

Elementary inductions

Multivariable calculus and n-spheres

Without integration

Generating functions

Special values, just for fun

Simplicial whatever quickstart guide

Complexes

Simplicial complexes

Delta complexes

Organizing the delta complex data model

Data needed to describe the flower above

In higher dimensions

Recovering the usual definition of \(\widehat{\Delta}\)

Simplicial sets

What’s still missing

A survey of underlines in HTML

Native support in browsers

Non-native implementations and line thickness

Obstacles to adoption

A footnote on LaTeX and prescriptivism

Updates

What is a quasicrystal?

Abstract

Contents

The crystallographic restriction theorem

X-ray crystallography: peeking inside real crystals

Setup

A troublesome photo

Demo: Making waves

Slicing higher-dimensional lattices

The problem: locating atoms

The explicit recipe

Demo: Atoms in 2-D quasicrystals

Self-similarity

A number-theoretic model of quasicrystals

A mathematical description of zooming

3-D quasicrystals

The n-dimensional viewpoint

The search for 3-D quasicrystals

Footnotes

Sol geometry and the tangent bundle of a hyperbola

Setup

Description of the action

No preservation of distances or angles

Diffy queues

Recovering the group

window.requestAnimationFrame troubles

Highlights of Simon Tatham's puzzle collection

Glue records for Melon Island fixed