[SATLUG] Rackspace

Brad Knowles brad at shub-internet.org
Thu Nov 12 16:38:42 CST 2015

On Nov 11, 2015, at 1:38 AM, typedeaf at yahoo.com wrote:

> I worked with a guy that published a book on IT security. That guy didn't even know what an RBAC was, couldn't describe the purpose of a single non-general purpose register, or couldn't describe a single common web based attack vector. I worked with a guy that wrote the book on how to tune Oracle RAC for Linux, and that guy didn't know the difference between user and kernel address space mappings (talking 32-bit x86 era) or even what a I/O scheduler was. In my opinion, being the author or editor of a book qualifies you to be an author or editor of a book, and nothing more.

You are correct, that some (many?) book authors are clueless.  But that’s not the case for O’Reilly authors from the mid-90s.  Cricket Liu was (and is) widely regarded by the DNS community as one of the leading experts in the field, and he brought a great deal of depth with him when he was working at Men & Mice.  I did some consulting for them at the time, and they had some top-notch tools, as well as good training materials.  Cricket has since moved on to Infoblox, where I think he has continued to do some pretty good work.

Bryan Costales knew as much or more about Sendmail than just about anyone on the planet, short of Eric Allman himself.  And Eric contributed heavily to the book, but it was still ultimately Bryan’s book.  I came close to working for Eric and Bryan at MercuryMail, but that deal ended up not working out, and Eric later transitioned to Sendmail, Inc.  When I was working at Belgacom Skynet SA/NV, we brought in Nick Christenson from Sendmail, and he and I ended up creating an architecture that I later turned into an invited talk at LISA, where I stood up in front of a room full of hundreds of people to explain what we had done.  I was pleased to find that they extensively used my LISA slides to present their large-scale systems architecture to their customers — there was more than one job interview I walked in to where they showed me the architecture document that described how their systems were set up, and I was able to turn around and show them the latest versions of those same slides — from my talk.

> Are you saying that you have never met a single Linux admin that knows network services? That is crazy. I can not relate.

Sadly, I have met very few.

> You say you were asked absurd questions about DNS. This is the first question I asked about DNS every time, and still ask today. "Name 5 record types and describe what they are used for." 99% of the people who put DNS on their resume could not answer that question. If you have EVER added a new server to your DNS records, chances are extremely high that you added an A record, PTR record, maybe a CNAME record, and updated the serial in the SOA record. That leaves one more record to get the answer correct, and every admin worth his salt has performed a nslookup/dig to find the MX record when working with mailers. I don't think that is an absurd question, but hundreds of people would disagree with me.

No, that isn’t absurd.  It is basic.

I would go into the differences between AAAA and A6 records.  And why you need both TCP and UDP for DNS.  And why you never, ever want to do DNS debugging with “nslookup”, but instead use a tool like “dig”.

Or maybe why does in-addr.arpa exist?  Why is it formatted in that weird way?  What other second-level domains have existed inside of .arpa?

Do you know what a lame delegation is?  Do you know a script to automatically detect lame delegations and notify the appropriate admins?  Do you know of a script to automatically detect other types of typical delegation errors?  Who is the original author of both of these programs?  Who is the current maintainer of these two programs?  When were they first contributed to the BIND source code, and when was the most recent update?

How much of a book do you want me to write?

> I used to say, "On a scale of 1-10, how well do you know Perl? 1 being, what is a pearl and 10 being I am Larry F'ing Wall" Invariably people would answer 7. Never 5. Never 6. Never 10. 7. So I would ask, "What the the three built in data type in perl?" If they were quiet, I would say, "Okay, I mean data types like scalars and arrays. Can you name one more?" 99% of the time, they couldn't say "hash", and when questioned further, they would say they have never used hashes. How could you be a 7:10 in Perl and never have had used a hash? Point is, know what you don't know, and do not BS your skill level.

I’ve used a bit of Perl over the decades, and about the only thing I really recall using much in Perl was hashes.

But I never would have been able to name those three data types.

>> Well, at least until I got into this DevOps thing, because so many places had decided overnight they could cut their staff in half by replacing all their “admins” with “DevOps” guys and make the people involved do twice as much work.

> You realize that statement is equivalent to saying, 'they cut their staff in half by replacing their "developers" with "Agile" guys', right? DevOps is a methodology for SA, not a job title.

You might know that.  But apparently a large amount of people in this world do not.  I can’t tell you the number of people who see “DevOps” in my profile on LinkedIn and who get the concept 100000% wrong.

I worked at AOL before Gene Kim did, and when I started there we effectively actually did DevOps in our group because both the Internet Mail developers and the one and only Internet Mail operations person (me) both reported to Jay Levitt, who was head of Internet Mail Development.  Internet Mail Operations got split out later, and the whole thing ended up devolving into the mess that Gene wrote about in his book “The Phoenix Project".

I was in Belgium before Kris Buytaert thought up the concept.  I knew John Willis during his days at what was then called “Opscode”.

I’ve had discussions with both Gene and Kris as to how did we not cross paths all those many years ago.

> Most jobs hiring, "DevOps Engineers" are actually looking for people with experience in tools that aid in continuous integration and automation, like Chef/Puppet/Ansible. Some companies looking for "DevOps Engineers" are actually looking for SA that specialize in writing or heavily modifying existing automation tools written in HLL like Python, Ruby, Perl, Go, or even Bash. Unfortunately, that has mislead a lot of people into thinking that "DevOps" means a developer that also does operations.

If you’re a “Developer” and you’re part of an interdisciplinary group doing DevOps, then you damn well better be ready to be woken up at os-shit-dark-thirty in order to be able to debug some sort of operational problem with your code.

The days of you working on your code in your own private cave in the corner and then throwing your code over the wall when you’re done — those days are over.

Brad Knowles <brad at shub-internet.org>
LinkedIn Profile: <http://tinyurl.com/y8kpxu>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://www.satlug.org/pipermail/satlug/attachments/20151112/c000846a/signature-0001.bin

More information about the SATLUG mailing list