We use a programmatic assessment of Linux skills in our hiring process. I'm not thrilled with such assessments, but they do give a good foundation to talk over in the tech interview.
There's this one question that _everyone_ gets wrong, but only on a technicality. They figure out how to find the answer, but they always copy-paste the full contents of the proc 'file' which contains all the possible settings with marks around the currently selected one. The grader want's just the selected value.
One of my jobs I taught the interviewer new things about bash/linux (he was the lead java dev, I was interviewing for sysadmin.)
I've also had interviewees teach me new things about bash/linux.
I've had a couple of automated type assessments but never been super happy with the inflexible nature of the results they want.
@dcbaok The inflexible nature is why I
1) use it to establish a baseline of knowledge and the beginning of a conversation; never as a screen that disqualifies someone
2) took the assessment myself and seek to understand the sticking points and why people give the answers they do.
I'm interviewing for near entry-level, and it's a poor use of time to explore "do you understand chmod" in a live interview.