When reviewing, how to assign the number acorns?

For a long time, I've been thinking about getting my energy together and reviewing a bunch of books here. I am prone to analysis paralysis and have spent four months or so just thinking in the background about how to reasonably assign scores for the grid. I suspect most people just go with their gut and figure that's good enough. Sometimes I even agree. (A day or three ago, I noticed EFN has a slot on the seed-supplier grid and since they're basically my favorite source, it felt fine just giving them a 10 without much analysis.) How do you folks do it?

But books seem more serious to me. And also, a long time ago, I got am M.Ed. emphasizing assessment and psychometry, so I have some exposure to actual research on grading practices and their validity and reliability. A quick example is that teachers show high reliability and validity when sorting the work of students into five buckets (e.g. A / B / C / D / F) but are much worse at sorting that work into 12 buckets (e.g. A / A- / B+ / B / B- / C+ / C / C- / D+ / D / D- / F) and literally terrible when using a 1-100 score system. This ten acorns scheme, especially since 0 can be assigned, and even more since we aren't restricted to integers, more closely matches those latter two, less reliable less valid systems. The traditional approach in education would be to build and use a grading rubric that instructs the grader on how to evaluate the work without bias.

So then, I've been thinking about the things a book should do and came up with a series of questions I could use to interrogate how good a book is:

Does the book present a coherent thesis that is then backed up by the rest of the work?

Is the subject of the book more important than average?

Do I feel that the information in this book is broadly under-appreciated?

Did this book particularly electrify me when I read it?

Is the book easy to read and maintain my interest throughout?

Was the author’s voice appealing?

Is there another book that covers the same topic, but does so better?

If assertions were made that should be backed up with citations, were they?

Did the book make good use of organizational techniques: sections, headers, sidebars, etc?

Did the book have the right amount of diagrams, photos, charts, etc?

Did this book convince me to change my practices?

Did this book teach me to do something new?

Did this book include anything that was broadly incorrect?

How relevant is the book to venue where the rating is taking place?

But I'm finding the translation from answers to those questions to a 0-10 score a little evasive. Maybe it's good enough to ask those questions (and others -- can you think of good additions to that list?), ruminate on the answers, and then come up with a score while taking those factors into account. But what I really wanted was a more explicit algorithm like: assign a score of 0-3 to each question, sum the squares of the scores and truncate the square-root of the sum at 10 when greater than 10 ...y'know, or something.

I'll keep thinking about this, but if anyone has any bright ideas, I'd love to read them.

My approach is similar to the one I used as an accreditation surveyor.   I assume the book is perfect before I open it.  It is up to the author to show me it isn’t.  Persistent small glitches result in an acorn loss.  Of course a major glitch can as well.   Things that come to mind that grab my attention are a misleading title, lack of continuity, errors in grammar and spelling, statements that could result in injury or illness, obvious errors in fact, confusing statements, not covering something that should have been covered, etc.

While I tend to think in terms of the book beginning perfect and me taking away points, positive mitigating factors might be introducing a process or information that is new to me, especially informative photos and drawings (I especially appreciate it when the author realizes a photo is not quite clear enough and includes a detailed drawing as well), multiple alternative methods, etc.

Understanding writing structure at all levels in the book: Tell me what you are going to say, say it, tell me what you said.

Gosh...it took me ten months to get back to this. I guess that's fine -- perennial discussions and all. (John's comment above about working backward from 10 caused me a several month detour while figuring out if that's the way I wanted to go. :-) )

I have finalized(ish) an instrument that I'm using to rate books. I am assigning a 1-5 score for each of 15 categories, with the putative assumption that most books will get 3s in most categories. I'm weighting extreme values, particularly positive ones so that they tip things more than neutral values, and weighting each rating differently based on how important they are to me. (So, for example, the degree to which I felt electrified by reading the book is three times as important as whether I actively learned a new skill) combining the massaged ratings in that way, I build a composite score. The book can theoretically have a score from -172.4 to 344.8. The tricky part was figuring out how to translate that into acorns. To do that, I rated 25 books and played with the numbers to determine what felt right for assignments. I had 3 of the 25 books that I wanted to give 10 acorns to. I had several that I thought deserved only 5. So I broke up the composite rating space around and between the book-ratings I had, and asked Google Sheets to generate a polynomial trend-line. If you're nerdy enough to want the details, you can poke around at the formulae in my assessment instrument here.

I might yet fiddle with weights and calculations, but getting this squared away has been a big step toward being in a position to rate books here.

ETA: The three books I wanted to have 10 acorns ended up getting: 10, 9.9, and 9.7. There's a huge area of the configuration space of this model that results in a score of 10, but I think not many books are really going to attain that high a slot. Probably that means that my personal rating system is going to bring the averages down because it looks like a lot of 10s are awarded and not very many low numbers. (I could build a cogent argument for why that means y'all are misusing the rating system, but that would bore everyone and change no one's mind.)

I have only reviewed books that I really liked so giving the books a score was fairly easy.

I give this book 10 out of 10 acorns.

Maybe a 9 or 8 to be accurate I would have to go back and look.

I have tried to approach a system that I liked that could explain a bit more in depth the score I was giving instead of the ole gut check.

What I run into is I have WAY too many variables that I have found to be 'important' in what I consider to be a good book. I'm working on trying to wittle it down to about... five or six? I'd be extremely pleased to get it down to a score structure of four variables.

I really like what you are putting together and its organization. I'm going to pull some inspiration from it so thank you.

Anne, yeah -- if everyone is only reviewing books they especially like, that neatly accounts for the "too high" scores without anyone doing anything wrong.

Tim, I get what you mean about whittling down the number of factors. Believe it or not, while I was ranking my 25 sample books shown above, I eliminated three whole columns from the instrument because I decided they weren't adding anything. But the value to me in keeping more than just a few is that a book can shine in this way or that way, having entirely different strengths, and I want them both to have the option of being pretty good when reduced to the acorn scale.