[chef-dev] Re: Dialect support and loading enhancements

[Brad Knowles < >]

On Sep 20, 2013, at 5:58 PM, Lamont Granquist 
<
 >
 wrote:

> I wrote thousands of lines of (retrospectively) awful perl and cfengine 
> scripts to keep Amazon and Real Networks running, but nothing that would be 
> considered "software".

At AOL, we tried cfengine, and we never managed to get anything useful done 
with it.  Our most academic-oriented guy (Eugene Day) tried long and hard, 
but ultimately even he gave up.  I did my share of bad perl scripting and the 
odd custom sendmail rewrite rule, but most of the time I spent dealing with 
internal political BS and fighting back the latest wave of "Awesomest Ideas 
Ever to Fight Spam" from VPs that were hired the previous week -- this from 
guys who had no idea about how SMTP actually worked.

> At the tender age of 39 I got sick of operations, and couldn't see myself 
> landing another job in ops, and I was impatient with the devops revolution 
> and so I switched to being a full time SDE and learning ruby and new rules 
> like "use composition instead of inheritance" and started noodling on rails 
> and angular websites in my spare time.

I got tired of pure operations more recently -- being treated really horribly 
by UT Austin was enough to get me permanently off that track.

I've always had the greatest respect for guys who've been able to do both 
systems programming and systems administration -- like Eric Allman.  
Unfortunately, those are typically also the sorts of guys who simply don't 
get that these are two related but separate aptitudes.


It's like Dutch versus Flemish.  Ask someone from The Netherlands whether 
Flemish is a separate language or a dialect of Dutch, and they'll answer that 
it's a dialect.  Ask someone from the northern regions of Belgium that 
question, and it's clear that it's a separate but distinct language.

In that case, who is right?

I submit that the people who are native speakers of the language in question 
are the ones that are best positioned to answer that question.

> The shift was midly terrifying at times, but not as radical as I expected.  
>  I think most of the barrier between programmer and software dev is mostly 
> in learning a different *human* language, and it takes immersion in the 
> culture to do it. For me, I don't learn well through theory and reading 
> dissertations on a subject but more from rolling up my sleeves and trying 
> it until it breaks, which is why I had a really hard time becoming a 
> software dev without actually being a software dev.

I learn best by doing, and second-best by watching someone else do.  
Listening to someone describe how to do something is a very distant third.  
Reading about it falls somewhere down in the gutter.


I recently spent several months working in Denver on a Chef gig for a 
government contractor (actually, my employer was a subcontractor), and I was 
as deeply immersed in "programming" as I think it is possible for me to be.

At the end, I felt slightly more comfortable with Ruby than I had before 
doing that job, but the biggest challenge I faced was being forced to become 
an expert in how to deploy the single most mission-critical application that 
company had (on a multi-billion dollar project).  While I was able to call on 
their people for support if I had questions, ultimately when I left I took 
most of what I learned with me and there was little or no transfer of Chef 
skills or knowledge to the customer.

I didn't have a choice -- that's how I was forced to work.

I got better at learning how to get Chef to do the things I had to do, and 
working around certain very weird bugs or features, but I wouldn't say that I 
am a materially better programmer.  I increased my skills in those areas, but 
my talent in that aptitude hasn't materially changed.

> From my perspective, the idea that the Salt example is somehow "easier" is 
> just flatly insane (and I should mention that I'm not speaking for Opscode 
> here, this is just my own opinion).

I actually like the Salt example better.  Less fiddliness with quotes, in the 
majority of the cases.  I wasn't aware of the if/elif/endif syntax, but in my 
mind it is certainly no worse than the Chef case example.

Keep in mind that Salt also has much more sane file organization for states, 
and you only have to create hierarchy within that structure if you want/need 
it, and then you only have to create it to the degree you want/need.

> In fact now you have to explain that you're writing your config language in 
> YAML and then go on to explain that when its not powerful enough they need 
> to use the jinja templating language to get their jobs done.

Take the multiple OS issue out of the picture, and the Salt declarative model 
becomes much simpler.  In this case, you only get into these issues if you 
have to support multiple OSes, and then you've got flexibility in how you 
support them -- I'd be inclined to write separate state files for each 
platform, so that you minimize the amount of the templating you need to do.

> Now they have two things to learn, conceptually, and they have to context 
> switch between all the leading -'s and trailing :'s in YAML markup and 
> putting all the nasty programming language in between whatever template 
> expansion operators {% %} that you are using.

Jinja is the preferred solution, but others are also possible.  With Chef, 
you're largely writing code in what amounts to the Chef DSL, but you can 
always fall through to Ruby if you need to.  With Salt, they give you more 
options than just YAML or Python, but those are two obvious choices.

I'm not convinced that more options in this case is actually that much of an 
improvement for Salt, but in this case I don't see that it's necessarily 
worse than Chef in this regard.

> I think if we just wait awhile people are going to start posting non-trival 
> examples of Salt and Ansible that will make your eyes bleed.

That's certainly possible, but we've certainly got that situation already 
with Chef.

I certainly remember similar arguments being made about VMailer when it first 
came out, but even in the early days it was remarkably capable.  Over the 
years, Wietse has added functionality and the configuration file syntax has 
gotten more complex and harder for humans to parse, but it's still light 
years ahead of Sendmail.

Because sendmail has an embedded programming language, there are still some 
things you can do with it that you probably can't do with postfix, but 
postfix does have a lot of interfaces and APIs and relatively speaking it's 
not that hard to write mini-modules that sit listening on a port or a socket 
and do a small function that you can't otherwise do, and let postfix handle 
all the rest.

> Our community cookbooks are awful in a lot of places precisely because they 
> try to do so much, and I think if you translated those to jinja-YAML that 
> you're going to see the readability deterioriate substantially.  The 
> problem there I think lies in this idea that the community cookbooks are 
> necessarily the objectively right way of doing Chef.

Even the Opscode cookbooks have a significant amount of variability in the 
quality of their code, not to mention all the dozens of different versions of 
each that have been hacked by various people and then re-uploaded to the 
community site.

> Anyway, having changed from SE to SDE and coming from your same background, 
> I'm skeptical of the premise of your argument.  And even given your premise 
> I'm skeptical that the proposed solution actually solves it for anyone in 
> other than the most trivial of cases.

I know there are a lot of moving parts in Chef and it has a very complex code 
base, and it depends on a lot of other components that Opscode doesn't 
necessarily have much control over.  But I'm coming around to the view that a 
radical simplification of Chef is becoming more and more overdue.

Allowing the use of a YAML-based templating language for the simpler 
functions of Chef may not be the best way to achieve that radical 
simplification.  Maybe instead of just papering over the existing flaws we 
need to start deconstructing the system as a whole and see how things need to 
be completely re-built from the ground up.

But for the moment, I'm throwing in with Noah.

> As a historical aside how did we even wind up in this situation anyway?  
> Back when I was growing up 'system adminstrators' like Wietse Venema went 
> off and wrote tcp_wrappers and postfix, and him and Dan Famer wrote 
> SATAN/SAINT.  Having SAs who were proud of not programming back then would 
> have seemed bizzare to me.

It's not that I'm proud of not having a programming background.  It's that I 
recognize the difference between these two closely related aptitudes, and 
that I have one set of talents and skills, and there are another group of 
people out there who don't seem to be able to comprehend that there actually 
is a difference.


When I tell you that my opinion is that these two things are similar but 
different, having you tell me in response that my opinion is wrong and stupid 
... isn't very conducive to forward progress.

Ultimately, my opinion is my opinion, and it's every bit as valid as yours.  
And both differ by some degree from whatever could be called "objective 
reality".

> Back before then, there wasn't any good package management, and there 
> wasn't even GNU autoconf when I started learning C and Unix.  There was no 
> "./configure; make; make install" we had to hack up Makefiles and at least 
> choose our CFLAGs from all the different Unix variants like 3B2 that we'd 
> never heard of before, and then cobble together our own CFLAGs based on the 
> compiler errors we got because the version of Dynix that we had on the 
> Sequent Symmetry wasn't in the Makefile yet.

We don't have to go back to the "bad old days" in order to solve these 
problems.  Throwing moths into the relays in order to create "bugs in the 
program" isn't going to be productive.

If you do want to do that, then please feel free to break out the toggle 
switches and the tubes, but don't expect me to support you in any respect 
towards that goal.

> And if we go off and extend the dialect to attempt to get parity, which of 
> the outstanding tickets of yours and everyone else's do we bump in order to 
> fix the dialect?  TANSTAAFL...

That's true enough, but we also need to avoid the problem of confusing the 
urgent with the important.  Otherwise, we risk focusing on the alligators and 
not the swamp.

> Noah *almost* made it through to me that there is a barrier to entry 
> between the python community and chef because of the need to use ruby.  I 
> can see that.

That's certainly true.  It does seem a little silly for any Python-based shop 
to be forced to use a rather different programming language in order to 
achieve higher levels of infrastructure automation.  I find it very 
instructive that both Salt and Ansible seem to be aggressively involved in 
supporting OpenStack.  I do wonder how hard they'd have to work to become the 
official incubated infrastructure automation solution for OpenStack, thus 
largely kicking both Chef and Puppet to the curb.

Ansible also seems to be interested in trying to solve some of the 
orchestration problems that Chef explicitly disclaims.  I have yet to see how 
well the Ansible "agent-free" model actually scales, but if they have an 
accelerated communications option that is based on 0mq, then maybe they can 
actually keep up with Chef, Salt, etc....  It does seem to me that you would 
at least want to cache as much of the infrastructure code as possible, 
however.

> I certainly see that barrier from the other side as well. But at the same 
> time I've managed to pick up javascript and angular in my free time because 
> I was interested in that.  I also don't see this as a solution because it 
> would push chef into the situation where if a python dialect of chef was 
> actually successful you'd now need to learn *2* programming languages to 
> interact fully with the community.

I'm not convinced that we need to fully support a separate language like 
Python for interacting with the Chef server.  But having a simpler and less 
Ruby-specific solution is definitely something I am very interested in 
looking at.

Maybe we could just simplify the Chef DSL so that it's more generic, and both 
sides could be less unhappy with the solution?

Meanwhile, I really do want to see what Noah has cooking.

--
Brad Knowles 
<
 >
LinkedIn Profile: <http://tinyurl.com/y8kpxu>