Friday, May 10, 2013

So after all that...

What would be our ultimate solution for the problem? Given the previous post, we have two cases:
  • m' and n' both have d digits: in this case, if m' is greater than the largest d-digit Y, the result is 0; otherwise, we have to actually generate the d-digit Y values and return the number that are in range.
  • m' has d digits and n' has d' digits, d < d': the result then is the sum of the number of Ys between m' and 10d-1, the sum of the number of k-digit Ys for k from d + 1 to d' - 1, and the number of Ys between 10d'+1 and n'.
We need smarter memoization of the function that maps d to the list of d-digit Ys and the function that maps d to the count of d-digit Ys, that doesn't grind out all the results for all values from 1 to d. (If they were recursive in d, I wouldn't mind.) Time to do some research.

UPDATE: or maybe I have a misconception about lazy evaluation. If the list elements don't depend on one another save for their position, maybe you don't have to evaluate an element just to skip over it with !!.

(Of course, we can easily generate the largest d-digit Y. If d == 1, it's 3; otherwise it's the largest two-2 d-digit Y.)

Thursday, May 09, 2013

Combinatorics

You know, for the Code Jam problem,  you just have to know how many "fair and square" numbers there are in a given interval. That's not necessarily the same thing as having to generate all of them.

Having done (or read) the analysis, we know that the number of "fair and square" numbers between m and n is the number of palindromic numbers between ⌈sqrt(m)⌉ and  ⌊sqrt(n)⌋ (we've shown how to get those values in previous posts)--let's call those values m' and n' respectively--for which the sum of the squares of the digits is less than 10.

Let's just consider d-digit base 10 numbers for d > 1, so we don't have to worry about 0 or 3. If [m'..n'] includes all d-digit base 10 numbers, or, given our theorem, at least the d-digit base 10 numbers from 10...01 to either 20...02 (if d is even) or 20..010...02 (if d is odd), then it has all the d-digit palindromes of the sort we want. (Writing them that way is serious handwaving about how big d is, sorry.) How many is that?

m `choose` 0 = 1
m `choose` n = product [m - n + 1..m] `div` product [1..n]

numTwoTwos d
    | d == 1    = 0
    | even d    = 1
    | otherwise = 2

numOneTwos d
    | d == 1    = 1
    | even d    = 0
    | otherwise = d `div` 2

numNoTwos d
    | d == 1    = 2
    | otherwise = if even d then n else 2 * n
                  where d' = d `div` 2 - 1
                        n = sum [d' `choose` i | i <- [0..min d' 3]]

numYs :: Int -> Int

numYs d = numTwoTwos d + numOneTwos d + numNoTwos d


(OK, so I went ahead and handled the d = 1 case.)

Then you need only actually generate and filter the Ys from m' to the next higher power of ten, and from the largest power of ten <= n' to n'. The others you can count. Sorry, Little John; if you wanted the numbers themselves you should have taken it up with the Code Jam people.

UPDATE: Fixed my errors, and went ahead and dealt with the one-digit case in the various num*Twos functions.

Now, the only question is what to do about the portion of the interval from ⌈sqrt(m)⌉ to ⌊sqrt(n)⌋ that doesn't cover a whole range of values for a given power of ten? More precisely, how to handle it efficiently, without generating Y values any more than we have to?

UPDATE: revised numNoTwos to omit 0 for the one-digit case, since the statement of the problem says it's not dealing with 0.

Wednesday, May 08, 2013

Compare and contrast

...as my English teachers used to say.
"Finally, although the subject is not a pleasant one, I must mention PL/1, a programming language for which the defining documentation is of a frightening size and complexity. Using PL/1 must be like flying a plane with 7000 buttons, switches and handles to manipulate in the cockpit. I absolutely fail to see how we can keep our growing programs firmly within our intellectual grip when by its sheer baroqueness the programming language —our basic tool, mind you!— already escapes our intellectual control. And if I have to describe the influence PL/1 can have on its users, the closest metaphor that comes to my mind is that of a drug. I remember from a symposium on higher level programming language a lecture given in defense of PL/1 by a man who described himself as one of its devoted users. But within a one-hour lecture in praise of PL/1. he managed to ask for the addition of about fifty new “features”, little supposing that the main source of his problems could very well be that it contained already far too many “features”. The speaker displayed all the depressing symptoms of addiction, reduced as he was to the state of mental stagnation in which he could only ask for more, more, more... When FORTRAN has been called an infantile disorder, full PL/1, with its growth characteristics of a dangerous tumor, could turn out to be a fatal disease." --Edsger W. Dijkstra, from "The Humble Programmer" (ACM Turing Award Lecture 1972)
"This is the first Part of n, or lets [sic] say many entries in this blog. In total I hope to be able to cover most papers in 3-4 blog posts, giving the reader an overview over the suggestions and changes for C++ at the coming C++ Committee Meeting in April. In total there are 98 Papers, so I will skip some, but try to get as much covered as possible. I'll skip papers with Meeting Minutes for sure, and try to focus on those focusing on C++11 or C++14 features. As the papers are ordered by there [sic] Number (N3522 being the first), I'll go top down, so every blog post will contain different papers from different fields of C++ Standardization. As N352-24 are Reports about Active Issues, Defects and Closed Issues I'll skip them for now, also I will not read through Meeting Minutes and such." --Jens Weller, from "A look at C++14: Papers Part I"
Plus ça change...

Tuesday, May 07, 2013

Good company

Just saw something on Hacker News pointing to a tweet from John Carmack:

"I want to do a moderate sized Haskell project, and not fall to bad habits. Is there a good forum for getting code review/criticism/coaching?"

From a later tweet from Mr. Carmack: "The Haskell code I started on is a port of the original Wolf 3D...."

I'm sure I'm not at his level, but it's reassuring to be in good company... as I feel I am in my opinion of C++.

UPDATE: here's an HN link. This comment is particularly interesting; I'll have to keep an eye open for new ghc releases.

More attempts at optimization

I set up for profiling, and I made two changes. First, I put back the memoization of powers of ten. Second, it occurred to me that most of the time I invoke oddDigitPal, I'm invoking it twice with just the middle digit changing. So, I went for

oddDigitsPals :: Integer -> Int -> [Integer] -> [Integer]

oddDigitsPals topHalf nDigits middles =
    let noMiddle = topHalf * tenToThe (nDigits + 1) + backwards topHalf
    in [noMiddle + m * tenToThe nDigits | m <- middles]


I also changed over to Int (32-bit integers) for numbers of digits. I'm not worried at this point about going for fair and square numbers of over two billion digits. (Little John, you're on your own after that, OK?)

It did make some difference, based on profiling output. Most notably, it took about 14% off bytes allocated. Time was less impressive; with the profiling pulled, it took execution time down from not quite 1.9 seconds to not quite 1.7 seconds.

A C solution that I downloaded, compiled, and ran took more like 0.4 seconds. I'm bummed by that factor of slightly over four, but OTOH:
  • I'm no expert at Haskell optimization.
  • I'd much rather try to understand the Haskell I've written than the C or C++ solutions I've seen so far.

Monday, May 06, 2013

Just goes to show that Knuth was right

One other thing occurred to me as a possible optimization: I'm raising 10 to some integer power a lot. Why not try

powersOfTen = [10 ^ i | i <- [0..]]

tenToThe :: Int -> Integer

tenToThe n = powersOfTen !! n

which would memoize those pesky exponentiations? (!! lets you retrieve elements from lists sort of as if they were arrays, with "subscripts" starting at zero.)

It was easy enough to try out, but the results were disappointing. Even on my Eee 900A, with a 32-bit processor that you'd think would get the most benefit, the variations in time output from one run to the next were large enough that I can't say with certainty that it made any difference at all. Time output for the first large data set:

real    0m1.093s
user    0m1.068s
sys     0m0.016s

For the second large data set:

real    0m5.531s
user    0m5.472s
sys     0m0.044s

These are with the program compiled--I still haven't done the file opening code.

Sunday, May 05, 2013

One thing that doesn't carry immediately over

One thing that you can do with a compiled Haskell program that doesn't lend itself to ghci is I/O redirection. That's why I have yet to run the large data sets against the code running under the interpreter--I'll have to modify main to take a file name and use it as input source.

I will do it, though. I'm highly motivated, because it's related to what caused me to start all this in the first place.

Since I don't have Code Jam time limits...

...I should improve the style.

Lisp is a wonderful language. It's the first (largely) applicative language. Having a simple form that all language constructs follow means you can write seriously powerful tools to manipulate programs without wasting your time on convoluted parsing.

Compare that with the horrors of parsing C++, which strictly speaking is impossible! Thanks to templates, you have to solve the halting problem to successfully parse C++. Even ignoring that, there are ambiguous constructs, with a rule that says which way to decide in the presence of ambiguity. I suspect it all goes back to Stroustrup's eschewing formal language-based tools when writing that first ad hoc preprocessor for what was then "C with Classes". (BTW, Perl has the same problem. Perhaps there's something about kitchen sink languages.)

All that said, Haskell style isn't Lisp style. Any serious Haskell programmer would look at my palindrome program and say I lean far too heavily on if ... then ... else and I don't take advantage of Haskell's pattern matching and guards when defining functions.

Guards!

Take the backwards function. At first it was

backwards n = backwards' n 0
    where backwards' n accum = if n < 10 then n + accum
                                         else backwards' (n `div` 10) (10 * (n `mod` 10 + accum))


and that's perfectly valid Haskell, but we can do better. How about this?

backwards n =
    let backwards' m accum
          | m < 10    = accum + m
          | otherwise = backwards' (m `div` 10) (10 * (m `mod` 10 + accum))
    in backwards' n 0


Here we are using guards, and while, thanks to the current layout, there is still wraparound, the code now honors the golden 80-column punched card that some things, most notably standards for source code layout, still bow down to in the computer world. (UPDATE: I waved a dead chicken at the layout, and it now fits.) Things to note:
  • there's no = immediately after the list of formal parameters of backwards' the way there was in the first version. The = are after the guards.
  • it may just be that I'm not familiar enough with Haskell syntax, but I had to turn things around and use a let rather than a where for the definition of the helper function backwards'
  • speaking of style, did you notice I changed the name of the first parameter of backwards'? I decided I'd better do that, the same  way I'd avoid gratuitously redeclaring a variable in an inner block with the same name as one in an outer block.

Patterns

That's not the only way to avoid if then else, though. Remember choose?

nDigits `choose` nOnes =
    if      nOnes   == 0     then [0]
    else if nDigits == 1     then [1]
    else if nDigits == nOnes then [sum [10 ^ n | n <- [0..nDigits - 1]]]
    else                          concat [[0 + 10 * x | x <- (nDigits - 1) `choose` nOnes],
                                          [1 + 10 * x | x <- (nDigits - 1) `choose` (nOnes - 1)]]


Let's take advantage of pattern matching as well as guards:

_ `choose` 0 = [0]
1 `choose` _ = [1]
m `choose` n
    | m == n    = [sum [10 ^ i | i <- [0..m - 1]]]
    | otherwise = concat [[0 + 10 * x | x <- (m - 1) `choose` n],
                          [1 + 10 * x | x <- (m - 1) `choose` (n - 1)]]


For this, you should know that _ matches anything. An identifier matches anything, too, but you can refer to it on the other side of the =. Use _ when, given the other parameters and/or the previously checked patterns, the value of the function does not depend on the value that _ matches. The patterns are applied in order, by the way, so here 1 `choose` 0 will match  _ `choose` 0 and not 1 `choose` _.

Yes, I must admit that I chose shorter parameter names. While in the case of this test, I am passing in a number of digits, that is distracting from the function, so here I would say that my initial choice was wrong. It's not just a matter of making lines fit in 80 columns.

DRY

DRY here is an acronym, and it applies no matter what your programming language. DRY = Don't Repeat Yourself. I said, Don't Repeat Yourself.

I very much repeat myself in the original program when it comes to the formation of a palindrome from its upper half (or in the case of an odd number of digits, the upper half and the middle digit). That needs its own function, or rather functions.

In theory those are all the parameters needed; you can calculate the number of digits in the upper half. Since we have that value, though, I'm inclined to pass it in.

Also speaking of DRY, I should use these functions to generate the palindromes with two 2s rather than making a special case of them. Say what you mean, as clearly as possible.

[We pause here for some testing and checking of run times for the input file with values up to 10100.]

It's hard to say whether it makes a difference; just running the same program repeatedly shows variance in the time output. Here's one towards the low end:

real    0m1.822s
user    0m1.804s
sys     0m0.012s


while here's one towards the high end:

real    0m1.905s
user    0m1.872s
sys     0m0.028s


IMHO the result is worth the minimal, if it even exists, increase in run time.

Here's the code after the smoke cleared:

import Prelude
import Data.List

backwards :: Integer -> Integer

backwards n =
    let backwards' n accum
          | n < 10    = accum + n
          | otherwise = backwards' (n `div` 10) (10 * (n `mod` 10 + accum))
    in backwards' n 0

evenDigitsPal :: Integer -> Integer -> Integer

evenDigitsPal topHalf nDigits = topHalf * 10 ^ nDigits + backwards topHalf

oddDigitsPal :: Integer -> Integer -> Integer -> Integer

oddDigitsPal topHalf nDigits middle = (topHalf * 10 + middle) * 10 ^ nDigits + backwards topHalf

choose :: Integer -> Integer-> [Integer]

_ `choose` 0 = [0]
1 `choose` _ = [1]
m `choose` n
    | m == n    = [sum [10 ^ i | i <- [0..m - 1]]]
    | otherwise = concat [[0 + 10 * x | x <- (m - 1) `choose` n],
                          [1 + 10 * x | x <- (m - 1) `choose` (n - 1)]]

chooseRange :: Integer -> [Integer] -> [Integer]

m `chooseRange` ns = concat [m `choose` n |  n <- ns, n <= m]

twoTwos :: Integer -> [Integer]

twoTwos n
    | even n    = [evenDigitsPal msd (n `div` 2)]
    | otherwise = [oddDigitsPal msd (n `div` 2) d | d <- [0,1]]
    where msd = 2 * 10 ^ (n `div` 2 - 1)

oneTwo :: Integer -> [Integer]

oneTwo n
    | even n    = []
    | otherwise = let halfN = n `div` 2
                      msd = 10 ^ (halfN - 1)
                  in [oddDigitsPal (msd + x) halfN 2 |
                        x <- (halfN - 1) `
chooseRange` [0..1]]

noTwos :: Integer -> [Integer]

noTwos n
   | even n    = [evenDigitsPal (msd + x) halfN | x <- choices]
   | otherwise = concat [[oddDigitsPal (msd + x) halfN 0,
                          oddDigitsPal (msd + x) halfN 1] | x <- choices]
   where halfN   = n `div` 2
         msd     = 10 ^ (halfN - 1)
         choices = (halfN - 1) `chooseRange` [0..3]

possibles :: Integer -> [Integer]

possibles 1 = [0..3]
possibles 2 = [11, 22]
possibles n = sort (concat [noTwos n, oneTwo n, twoTwos n])

ys = concat [possibles nDigits | nDigits <- [1..]]

xs = [y * y | y <- ys]

palSquares :: Integer -> Integer -> [Integer]

palSquares m n = takeWhile (<= n) (dropWhile (< m) xs)

-- jagd's main, which I believe is set up to read stdin

solve i num = do
        [a, b] <- getLine >>= return . map read . words
        putStrLn $ "Case #" ++ show i ++ ": " ++ (show $ length (palSquares a b))
        if i < num
           then solve (i+1) num
           else return ()

main = do
       num <- getLine >>= return.read
       solve 1 num


UPDATE: Aside from something in a later message, the only things I can think of to do are
  • adding concise comments that suffice to describe what the code does and how to the extent that the code itself is unclear
  • changing some function names. possibles sort of made sense when we were range checking and/or palindrome checking right there, but the range checking is moved out and given the math, they're not possible, they're certain. Also, choose isn't quite right. We're not counting combinations, as the phrase "m choose n" is meant in standard math nomenclature--we're generating them. I'm not sure what to use in their place, though.

Riddler Classic, May 23, 2020—Holy Mackerel!

Another one using Peter Norvig's word list . It turns out that the word "mackerel" has a curious property: there is exactly ...