CROSSWORDS LA 2012

If you want to help promote Crosswords LA (and why wouldn’t you? All proceeds go to a very worthy cause) please consider adding the Paypal button to your own site. Here’s the code:

<form action="https://www.paypal.com/cgi-bin/webscr" method="post">
	<input type="hidden" name="cmd" value="_s-xclick">
	<input type="hidden" name="hosted_button_id" value="VBXJ8LVZ3BCEG">
	<input type="image" src="https://www.paypalobjects.com/en_US/i/btn/btn_buynowCC_LG.gif" border="0" name="submit" alt="PayPal - The safer, easier way to pay online!">
	<img alt="" border="0" src="https://www.paypalobjects.com/en_US/i/scr/pixel.gif" width="1" height="1">
</form>

You may want to add the “check your spam folder” caveat. Thanks so much, and enjoy the puzzles!

As I mentioned in the comments of the last post, FillBot Jr. has successfully solved a crossword! Now admittedly, it was low-hanging fruit — a Monday Newsday by Gail Grabowski — but I’m happy to have that result as a sort of “proof of concept.” So far the algorithm is ultra-simple: it looks for the answer it has the most confidence in and fills it in, ignoring any potential problems it may have down the road.

Buoyed by this result, I decided to tackle the following day’s puzzle. And it did great — filling in the entire puzzle except for one blank: the crossing of:
NO_E [Not any] and
MI_E [Not yours]
Wait, really? That’s what tripped up my bot? What gives? This is especially infuriating because I have both of those EXACT CLUES in my database.

Well, as you may recall, I am leaning heavily on MySQL’s full-text search which, for a given clue, quickly scours my database finding similar clues and even gives them a numerical value according to how well the clue matches. Well, it turns out that this has a big limitation in the form of stopwords — words that aren’t indexed by MySQL. And, you guessed it — “not”, “any”, and “yours” are on that list. I’m not really sure how to get around this; like I said, I am relying very heavily on this capability of MySQL. Maybe once I add some logic for “checking the crossings” this problem will be mitigated.

In any case, I’m happy with the results so far. Once I add some cross-checking logic, I think I might just be left with optimization improvements. And if it turns out to be decent, I’d love to add it to Crossword Nexus to allow users to upload .puz files to see how the bot handles them.

Thoughts?

Here’s the basics of it: MySQL has a terrific full-text match based on the tf*idf algorithm which I’m using to find good fits for clue/entry pairs. For instance, if I have a four-letter word for “Fried Cajun side” you can run the MySQL query

SELECT Entry,
MATCH (
Clue
)
AGAINST (
'Fried Cajun Side'
) AS Score
FROM `Clues`
WHERE MATCH (
Clue
)
AGAINST (
'Fried Cajun Side'
)
AND Entry LIKE '____'
LIMIT 0 , 10

and get the following results:

Entry 	Score
SLAW 	15.20439338684082
OKRA 	14.877264976501465
OKRA 	9.801589965820312
OKRA 	9.801589965820312
OKRA 	9.801589965820312
OKRA 	9.801589965820312
OKRA 	9.801589965820312
OKRA 	9.801589965820312
ROUX 	9.69262981414795
OKRA 	9.69262981414795

(OKRA appears several times because it has matched with several different clues.) Of course, if you knew even one of the letters, this process would immediately give you the correct answer (OKRA).

At each step of the algorithm, I’m taking these results and adding the “Score” to the difference between the top entry and the second-ranked entry. The entry with the highest sum gets added to the grid and the process is restarted. For now I’m not doing any backtracking; I just want to see how this algorithm will do simply marching ahead and filling in entries.

Ready? Let’s step through with it on Brendan’s grid and see how it does.

Step 1: 49 ACROSS: Electrical resistance symbol — OMEGA
We’re off to a good start. This clue was an exact match for one I had in my database, and was a much better match than the second-best possibility (which turned out to be SAILS, for whatever reason).

Step 2: 10 DOWN: 1961 Charlton Heston epic — ELCID
Step 3: 39 DOWN: Renault vehicle marketed to the U.S. — LECAR
Step 4: 33 DOWN: North Carolina town or college — ELON
These are all stone-cold gimmes for anyone who does a lot of crosswords, so it’s nice to see the algorithm doing well on them too. Nice to see it reject DUKE for 33-Down; the database’s clue of “North Carolina college town” was a great match.

Step 5: 57 ACROSS: ___ Center (home of the Nets and Devils) — IZOD
Step 6: 1 ACROSS: Model airplane material — BALSA
I was shocked that the algorithm got the IZOD Center so quickly, I guess because I didn’t know this little bit of trivia. And now that I’ve learned it, it’s already out of date, since those teams have moved to the Prudential Center. Oh well.

Step 7: 6 DOWN: Immune system stuff — SHREDBLOODCELLS
Well, we had to mess up eventually. A human would never ever have entered this into the grid, but the bot sees the clue in its database that reads [Attack the immune system?] and thinks it’s got a good match. Maybe I need some logic for when the clue in the database has a question mark? This seems especially important for long entries. (And incidentally, I don’t have WHITEBLOODCELLS in my database, which is shocking, considering it’s a totally legit 15-letter answer. I’ll have to data-mine Brendan’s puzzle after I’m done here to add it.)

Step 8: 59 ACROSS: Pummels — ROUTS
Ah, shoot, we’ve messed up again even though this one doesn’t look quite so bad. The intended answer was PELTS. There’s only one five-letter entry in my database with the word “pummels” in it, so it gets the green light here.

Step 9: 35 DOWN: Clothing joint? — SEAM
Step 10: 8 DOWN: Miso soup enhancer — MSG
Step 11: 60 ACROSS: Choosing word — EENY
Yeah, it’s doing fine on these, but let me show the grid at this point:

B A L S A . . S _ M . _ E _ _
_ _ _ _ _ . _ H _ S . _ L _ _
_ _ _ _ _ . _ R _ G _ _ C _ _
_ _ _ _ _ _ _ E _ . _ _ I _ _
. . . _ _ _ _ D _ _ _ _ D _ _
_ _ _ _ _ _ _ B . _ _ _ . . .
_ _ _ _ . . _ L _ _ _ _ _ _ E
_ _ _ . . S _ O _ _ . . _ _ L
_ _ _ _ L E _ O _ . . _ _ _ O
. . . _ E A . D _ _ _ _ _ _ N
_ _ _ _ C M _ C _ _ _ _ . . .
O M E G A . _ E _ _ _ _ _ _ _
_ _ _ _ R _ _ L _ . _ _ _ _ _
I Z O D . _ _ L _ . R O U T S
E E N Y . _ _ S . . _ _ _ _ _

Anything jump out at you? Something sure jumps out at me: we’re hardly crossing any words at all. We absolutely HAVE to use the information we’ve got in terms of the crossing words or we’ll never get anywhere. The algorithm isn’t doing that, and definitely needs a push in that direction.

Okay, look at the above grid one more time. What answer can you put in next, even without looking at the clues? I see an obvious one: _M_ZE has to be AMAZE, right? Let’s see what the bot fills in next.

Step 12: 27 ACROSS: Chicago-based insurance company that sponsors Manchester United’s uniforms — RIO
Okay … that’s wrong, first of all, and why didn’t you fill in AMAZE?

So … this needs work. Any thoughts on how to make this better?

What would the algorithm look like from this point? Here are my initial thoughts.

I think it would need to be a recursive/iterative algorithm. At each step we have a partially filled-in grid. The algorithm would:

• Go to each position in the grid and apply the above process to get possibilities for that position.  If no possibilities are returned, simply select all possibilities that fit and assign them a low score.
• Take the highest-scoring match and place it in the grid.
• Repeat with the new grid.

If we get to a point where there are no possibilities we have to backtrack.  That subroutine would go to (I guess) the most recently filled-in answer, remove it and go to the next possibility.  Meanwhile, once the grid is completed we stop, unless there’s some way to “check” the grid.

Is there a better way to do this, or some tweaks to the above I can add?  I kind of want to do this to see how well it would perform.

DECLARE @myserver sysname;
SET @myserver = (
        SELECT
        	s.name
        FROM
        	sys.servers s
        WHERE
        	s.server_id = 0
    );

SELECT
	@myserver as 'Server'
,   d.name AS 'Database'
,   fbp.permission_name AS Permission
FROM
	(
	    SELECT
	    	'DATABASE' AS mytype
	    ,   *
	    FROM
	    	sys.databases
	) d
       JOIN sys.fn_builtin_permissions(null) fbp
            ON  d.mytype = fbp.class_desc
WHERE
	Has_perms_by_name(quotename(d.name) , 'database' , fbp.permission_name) = 1
ORDER BY
	d.name

First observation: I don’t like the distribution of scores. I actually made a histogram of the distribution which you can see here:

Click to embiggen

Looks okay, right? Except … the vast majority of the useful entries are clustered in the 97-100 range. That’s way too tight a range. Also, no one will care at all about pretty much anything scored 50 or below. That’s way too loose a range for the junk. So next time I need to tailor the histogram to make the bars on the right smaller and the bars on the left bigger. Of course, this will test my sorting algorithm — if it ranks things wrong there will be a bigger gap.

Second: Some things that we don’t want get too high a ranking. I mentioned in the last post that Wikipedia seems to have a bias toward geographical sites. But it also has a bias toward things Americans tend not to care about … notably, soccer. Romario, Ruud van Nistelrooy and Guus Hiddink all rate 100 on the site, and there’s no way they will ever appear in an American crossword.

Is there a fix to this? Not really … unless we cheat a little bit. One bit of information I didn’t use when making the current rankings is the list of categories assigned to each article. So I can simply add some logic in the code that says something like “If the article has a category that starts “Cities”, multiply the page length by 0.8, and if it has a category that ends “footballers”, multiply the page length by 0.5.” The intent is to bring these articles to a length that an American would assign to them if he were making the page. Will it hurt some legitimate articles? Of course. But it will help weed out some of the chaff as well.

Thoughts? Have you been playing with it any?

P.S. My son is doing very well. His cancer is in remission but he is at a high risk of infection, so we are being very careful. And right now he is in my arms and GO TO SLEEP WHY AREN’T YOU GOING TO SLEEP?!?

P.P.S. Have fun at the ACPT everyone!

P.P.P.S. “Weed out the chaff” is a terrible mixed metaphor.

For the most part, this works pretty well. But let’s ask the site for the best results of the form ??E?L??.

Before we go on, try to think of some good crossword entries that would fit this pattern, preferably ones with Wikipedia pages.

Let’s see … there’s Kremlin and gremlin, Sheila E., The Blob, shellac … quite a few options. What are the top 10 results returned from Crossword Nexus?

Clenleu
Breilly
Kremlin
Fresles
Creully
Treclun
Bresles
Rieulay
Clesles
Treslon

That’s … extremely ugly. Wait, what the heck are those things even? Communes in France, are you kidding me? Is something messed up?

No, nothing’s messed up. Take a look at Clenleu‘s page. See all those links at the bottom (click on “Show” on the “Communes of the Pas-de-Calais department” tab)? Yeah, all of those pages in turn link back to Clenleu. In fact, about 900 pages in all link to Clenleu, which kills the results.

Luckily, there’s an easy way around this. If we only count links in the article text itself, we can skip all of those useless links. If we do that, Clenleu only has (drum roll please) 3 inlinks. So that takes care of that. Here are the new top 10 results with this method:

Kremlin 396
Siedlce 239
Heerlen 226
Shellac 160
Sheila E. 139
Peebles 137
Ixelles 133
Breclav 88
Feedlot 82
Preslav 72

Ahhhh, better. Still no “gremlin” or “The Blob” but those finished 12th and 13th respectively. So that’s it, right? We’ve got our new ranking metric?

Well, hold on, why stop there? There might be other good possibilities, too. Let’s look at length of article, number of languages an article is translated into, and recency of last edit. Maybe these have some value too.

Length of article:

MHealth 55963
The Play 21931
Tien len 21263
Gremlin 17843
The Kliq 17477
Shellac 16519
Heerlen 16194
Sheila E. 14344
Eden Log 13981
Ixelles 13020

Uh, MHealth? I guess it’s a thing. People like to write about it, at least. “The Play” didn’t mean anything to me until I noticed it was referring to the “The band is on the field!” play. This is pretty good, too.

Next up, number of translations:

Kremlin 38
Apelles 34
Heerlen 31
Ixelles 29
Sterlet 27
Shellac 26
Siedlce 25
Buellas 25
Preslav 24
Usellus 23

This … is not so great. I fear that things translated into many languages might be mostly geographical sites.

I have high hopes for recency of last edit. Let’s take a look:

O'Neills 1325721157
Kvevlax 1325693146
Alex Lee 1325693087
The Kliq 1325642247
Sheila E. 1325640534
Buellas 1325616464
The Flow 1325616163
Poe's law 1325612070
Rieulay 1325611112
Uxelles 1325605593

Oh, no, this is the worst one yet. And what is so important about Buellas that it’s translated into so many languages and updated so frequently?

Based on these results, I’m thinking of going with a metric that’s 70% inlinks and 30% article size. But what do you think? Should I exclude the other two completely? I’d like to get a little feedback before going live. Thanks!

UPDATE (3/7/2012 9:31 PM) In case you were curious how the 70/30 split would look, here are the first few:

Sheila E.
Shellac
Heerlen
Siedlce
Ixelles
Gremlin
Apelles
Kremlin
Peebles
Feedlot
The Play
The Bled
EHealth
Preslav
The Blob
Breclav
Abe clan
Wheelie

That’s pretty good. Cities and communes and the like are simply overrated by Wikipedia, so I’m not sure we could ever get rid of things like Heerlen and Siedlce and Ixelles. Thoughts?

This past Wednesday my two year-old son was diagnosed with leukemia. At this age the chances of a cure are surprisingly high and the doctors have been extremely pleased with his progress so far. Still, it is an incredibly scary process which is draining in so many ways.

I have been hesitant to write about this, but I now think it is necessary to do so. It doesn’t hurt to raise a little awareness about cancer. And writing this now will allow me to write more about it in the coming weeks, months and years as his treatment progresses. I’m sure I will have lots to say on the subject.

Many people have asked how they can help. I would ask you to consider giving blood or (especially) platelets. A timely transfusion of platelets early on in the process may have saved my son’s life. If you are eligible and can find the time, please consider donating.

Also: I don’t know what this will do to my schedule quite yet, but I am guessing that a lot of non-essential activities will fall by the wayside. Don’t expect much from me on Twitter or here for a little while. I also expect @FakeWillShortz to slow down his rate of tweeting, though he’ll be taking on some new writers soon.

Thanks for reading.

1. It is now an online interface, accessible at http://crosswordbutler.com
2. It doesn’t blindly get all available crosswords; only the ones from providers who have given their permission
3. Most interestingly (I think), it gives independent crossword constructors a site to host their puzzles and distribute them easily for free or for profit.

Right now the site is essentially a souped-up puzzle pointers page, that in addition to most of the standard links, also links to puzzles like Matt Gaffney’s weekly contest and Andrew Ries’s weekly Rows Garden puzzle. But the fun part will begin when independent constructors start using the site to host their puzzles. Right now there isn’t a one-stop shopping destination for independent puzzlemakers, so constructors have to set up Paypal accounts, websites, e-mail lists, etc. to be able to distribute their work. Crossword Butler will take care of all of that; all a constructor has to do is upload the puzzles to the site (through a simple interface). If the puzzle is paid, I would take a (small) cut; if the puzzle is free I’ll happily host it free of charge.

Feel free to become a member of the site and poke around. If you have suggestions, comments, questions, leave a comment below!

Thanks for reading. This is my grand experiment this year — hope it works.

The hardest thing,” says Billy [Beane], “is there is a certain pride, or lack of pride, required to do this right. You take a guy high no one else likes and it makes you uncomfortable. But I mean, really, who gives a f**k where guys are taken? Remember Zito? Everyone said we were nuts to take Zito with the ninth pick of the draft. And we knew everyone was going to say that. One f**king month later it’s clear we kicked everyone’s ass.

…

A lot of people in the room have forgotten that the scouting department hadn’t wanted to take Barry Zito because Barry Zito threw an 88-mph fastball. They preferred a flamethrower named Ben Sheets. “Billy [Beane] made us take Zito,” Bogie [a scout] later confesses.

Well, now that both guys’ careers are essentially over, we can ask the question: who had the better career, Zito or Sheets? Did Billy really kick everyone’s ass with that pick? Or were the scouts right?

I have chosen to measure the careers by fWAR (Fangraphs Wins Above Replacement) because I think that’s probably the best way. (If you want to do it some other way, be my guest, but bear with me for this part.) Now, before I reveal who has accumulated more WAR over his career, what do you think the answer will be, based on what you know about both careers?

Personally, my guess was Zito. I didn’t follow either guy especially closely during their careers, but I recall that while Sheets probably had the higher peak, Zito was much, much more durable, and he wasn’t so bad himself, even winning a Cy Young award along the way. Sure, he’s been awful since joining the Giants, but he’s still been above replacement level. So I would have guessed Zito would have accumulated more WAR, maybe about 15% more.

Well, here’s what Fangraphs says — Zito: 30.8 WAR. Sheets: 31.7.

So, yeah. Sheets’ peak was WAY higher than Zito’s. And it actually gets better — the FANS prediction for Zito this year is 0.9 WAR, which means he and Sheets would be exactly even in WAR at year’s end.

“But wait,” you might say. “It’s not fair to compare their entire careers. The A’s would only have had control over their pick for six years, before he hit free agency. Who accumulated more WAR over those first six years?” Good point, theoretical reader. Let’s take a look: during Zito’s time with the A’s, he accumulated 24.2 WAR. Meanwhile, in Sheets’s first six years with the Brewers* he accumulated 24.5 WAR.

* I’m not sure when he would have first hit free agency — the Brewers bought out at least one year of it with a contract extension. This number might actually be higher.

So to recap: the A’s didn’t exactly “kick everyone’s ass” with this pick. If anything, it was a wash. But there is an amusing postscript to the story — the A’s did end up signing Ben Sheets to a one-year contract for $10 million at the end of his career. He gave them 0.6 WAR for their efforts. Guess they just picked him up at the wrong time.