[chef-dev] Re: Dialect support and loading enhancements

[Noah Kantrowitz < >]

Back from vacation and I posted my example dialects and test cookbooks 
https://github.com/coderanger/chef-dialects-testbed

I've also made a few additions to the dialect API from working on these. You 
can now easily install a dependent gem from a dialect class, and you can 
setup cleanup hooks to avoid leaking memory between runs for people not using 
--fork mode.

--Noah

On Sep 24, 2013, at 4:52 PM, Noah Kantrowitz 
<
 >
 wrote:

> I'm working on porting over the two dialects I did for chef-funnel just to 
> make sure the underlying framework is flexible enough and has the needed 
> hook. Just got JS working tonight, and hopefully Python will be easier now 
> that a few kinks are worked out. Once those are up on github I hope I can 
> give people some more concrete examples to look at which might help with 
> the "changing so much at once" overload.
> 
> --Noah
> 
> Adam Jacob 
> <
 >
>  wrote:
> This hasn't fallen off the radar - we should make some decisions here soon.
> 
> Adam
> 
> 
> On Sun, Sep 22, 2013 at 1:58 PM, Avishai Ish-Shalom 
> <
 >
>  wrote:
> What an awesome thread this turned out to be!
> 
> 
> On Sat, Sep 21, 2013 at 5:50 AM, Brad Knowles 
> <
 >
>  wrote:
> On Sep 20, 2013, at 5:58 PM, Lamont Granquist 
> <
 >
>  wrote:
> 
> > I wrote thousands of lines of (retrospectively) awful perl and cfengine 
> > scripts to keep Amazon and Real Networks running, but nothing that would 
> > be considered "software".
> 
> At AOL, we tried cfengine, and we never managed to get anything useful done 
> with it.  Our most academic-oriented guy (Eugene Day) tried long and hard, 
> but ultimately even he gave up.  I did my share of bad perl scripting and 
> the odd custom sendmail rewrite rule, but most of the time I spent dealing 
> with internal political BS and fighting back the latest wave of "Awesomest 
> Ideas Ever to Fight Spam" from VPs that were hired the previous week -- 
> this from guys who had no idea about how SMTP actually worked.
> 
> > At the tender age of 39 I got sick of operations, and couldn't see myself 
> > landing another job in ops, and I was impatient with the devops 
> > revolution and so I switched to being a full time SDE and learning ruby 
> > and new rules like "use composition instead of inheritance" and started 
> > noodling on rails and angular websites in my spare time.
> 
> I got tired of pure operations more recently -- being treated really 
> horribly by UT Austin was enough to get me permanently off that track.
> 
> I've always had the greatest respect for guys who've been able to do both 
> systems programming and systems administration -- like Eric Allman.  
> Unfortunately, those are typically also the sorts of guys who simply don't 
> get that these are two related but separate aptitudes.
> 
> 
> It's like Dutch versus Flemish.  Ask someone from The Netherlands whether 
> Flemish is a separate language or a dialect of Dutch, and they'll answer 
> that it's a dialect.  Ask someone from the northern regions of Belgium that 
> question, and it's clear that it's a separate but distinct language.
> 
> In that case, who is right?
> 
> I submit that the people who are native speakers of the language in 
> question are the ones that are best positioned to answer that question.
> 
> > The shift was midly terrifying at times, but not as radical as I 
> > expected.   I think most of the barrier between programmer and software 
> > dev is mostly in learning a different *human* language, and it takes 
> > immersion in the culture to do it. For me, I don't learn well through 
> > theory and reading dissertations on a subject but more from rolling up my 
> > sleeves and trying it until it breaks, which is why I had a really hard 
> > time becoming a software dev without actually being a software dev.
> 
> I learn best by doing, and second-best by watching someone else do.  
> Listening to someone describe how to do something is a very distant third.  
> Reading about it falls somewhere down in the gutter.
> 
> 
> I recently spent several months working in Denver on a Chef gig for a 
> government contractor (actually, my employer was a subcontractor), and I 
> was as deeply immersed in "programming" as I think it is possible for me to 
> be.
> 
> At the end, I felt slightly more comfortable with Ruby than I had before 
> doing that job, but the biggest challenge I faced was being forced to 
> become an expert in how to deploy the single most mission-critical 
> application that company had (on a multi-billion dollar project).  While I 
> was able to call on their people for support if I had questions, ultimately 
> when I left I took most of what I learned with me and there was little or 
> no transfer of Chef skills or knowledge to the customer.
> 
> I didn't have a choice -- that's how I was forced to work.
> 
> I got better at learning how to get Chef to do the things I had to do, and 
> working around certain very weird bugs or features, but I wouldn't say that 
> I am a materially better programmer.  I increased my skills in those areas, 
> but my talent in that aptitude hasn't materially changed.
> 
> > From my perspective, the idea that the Salt example is somehow "easier" 
> > is just flatly insane (and I should mention that I'm not speaking for 
> > Opscode here, this is just my own opinion).
> 
> I actually like the Salt example better.  Less fiddliness with quotes, in 
> the majority of the cases.  I wasn't aware of the if/elif/endif syntax, but 
> in my mind it is certainly no worse than the Chef case example.
> 
> Keep in mind that Salt also has much more sane file organization for 
> states, and you only have to create hierarchy within that structure if you 
> want/need it, and then you only have to create it to the degree you 
> want/need.
> 
> > In fact now you have to explain that you're writing your config language 
> > in YAML and then go on to explain that when its not powerful enough they 
> > need to use the jinja templating language to get their jobs done.
> 
> Take the multiple OS issue out of the picture, and the Salt declarative 
> model becomes much simpler.  In this case, you only get into these issues 
> if you have to support multiple OSes, and then you've got flexibility in 
> how you support them -- I'd be inclined to write separate state files for 
> each platform, so that you minimize the amount of the templating you need 
> to do.
> 
> > Now they have two things to learn, conceptually, and they have to context 
> > switch between all the leading -'s and trailing :'s in YAML markup and 
> > putting all the nasty programming language in between whatever template 
> > expansion operators {% %} that you are using.
> 
> Jinja is the preferred solution, but others are also possible.  With Chef, 
> you're largely writing code in what amounts to the Chef DSL, but you can 
> always fall through to Ruby if you need to.  With Salt, they give you more 
> options than just YAML or Python, but those are two obvious choices.
> 
> I'm not convinced that more options in this case is actually that much of 
> an improvement for Salt, but in this case I don't see that it's necessarily 
> worse than Chef in this regard.
> 
> > I think if we just wait awhile people are going to start posting 
> > non-trival examples of Salt and Ansible that will make your eyes bleed.
> 
> That's certainly possible, but we've certainly got that situation already 
> with Chef.
> 
> I certainly remember similar arguments being made about VMailer when it 
> first came out, but even in the early days it was remarkably capable.  Over 
> the years, Wietse has added functionality and the configuration file syntax 
> has gotten more complex and harder for humans to parse, but it's still 
> light years ahead of Sendmail.
> 
> Because sendmail has an embedded programming language, there are still some 
> things you can do with it that you probably can't do with postfix, but 
> postfix does have a lot of interfaces and APIs and relatively speaking it's 
> not that hard to write mini-modules that sit listening on a port or a 
> socket and do a small function that you can't otherwise do, and let postfix 
> handle all the rest.
> 
> > Our community cookbooks are awful in a lot of places precisely because 
> > they try to do so much, and I think if you translated those to jinja-YAML 
> > that you're going to see the readability deterioriate substantially.  The 
> > problem there I think lies in this idea that the community cookbooks are 
> > necessarily the objectively right way of doing Chef.
> 
> Even the Opscode cookbooks have a significant amount of variability in the 
> quality of their code, not to mention all the dozens of different versions 
> of each that have been hacked by various people and then re-uploaded to the 
> community site.
> 
> > Anyway, having changed from SE to SDE and coming from your same 
> > background, I'm skeptical of the premise of your argument.  And even 
> > given your premise I'm skeptical that the proposed solution actually 
> > solves it for anyone in other than the most trivial of cases.
> 
> I know there are a lot of moving parts in Chef and it has a very complex 
> code base, and it depends on a lot of other components that Opscode doesn't 
> necessarily have much control over.  But I'm coming around to the view that 
> a radical simplification of Chef is becoming more and more overdue.
> 
> Allowing the use of a YAML-based templating language for the simpler 
> functions of Chef may not be the best way to achieve that radical 
> simplification.  Maybe instead of just papering over the existing flaws we 
> need to start deconstructing the system as a whole and see how things need 
> to be completely re-built from the ground up.
> 
> But for the moment, I'm throwing in with Noah.
> 
> > As a historical aside how did we even wind up in this situation anyway?  
> > Back when I was growing up 'system adminstrators' like Wietse Venema went 
> > off and wrote tcp_wrappers and postfix, and him and Dan Famer wrote 
> > SATAN/SAINT.  Having SAs who were proud of not programming back then 
> > would have seemed bizzare to me.
> 
> It's not that I'm proud of not having a programming background.  It's that 
> I recognize the difference between these two closely related aptitudes, and 
> that I have one set of talents and skills, and there are another group of 
> people out there who don't seem to be able to comprehend that there 
> actually is a difference.
> 
> 
> When I tell you that my opinion is that these two things are similar but 
> different, having you tell me in response that my opinion is wrong and 
> stupid ... isn't very conducive to forward progress.
> 
> Ultimately, my opinion is my opinion, and it's every bit as valid as yours. 
>  And both differ by some degree from whatever could be called "objective 
> reality".
> 
> > Back before then, there wasn't any good package management, and there 
> > wasn't even GNU autoconf when I started learning C and Unix.  There was 
> > no "./configure; make; make install" we had to hack up Makefiles and at 
> > least choose our CFLAGs from all the different Unix variants like 3B2 
> > that we'd never heard of before, and then cobble together our own CFLAGs 
> > based on the compiler errors we got because the version of Dynix that we 
> > had on the Sequent Symmetry wasn't in the Makefile yet.
> 
> We don't have to go back to the "bad old days" in order to solve these 
> problems.  Throwing moths into the relays in order to create "bugs in the 
> program" isn't going to be productive.
> 
> If you do want to do that, then please feel free to break out the toggle 
> switches and the tubes, but don't expect me to support you in any respect 
> towards that goal.
> 
> > And if we go off and extend the dialect to attempt to get parity, which 
> > of the outstanding tickets of yours and everyone else's do we bump in 
> > order to fix the dialect?  TANSTAAFL...
> 
> That's true enough, but we also need to avoid the problem of confusing the 
> urgent with the important.  Otherwise, we risk focusing on the alligators 
> and not the swamp.
> 
> > Noah *almost* made it through to me that there is a barrier to entry 
> > between the python community and chef because of the need to use ruby.  I 
> > can see that.
> 
> That's certainly true.  It does seem a little silly for any Python-based 
> shop to be forced to use a rather different programming language in order 
> to achieve higher levels of infrastructure automation.  I find it very 
> instructive that both Salt and Ansible seem to be aggressively involved in 
> supporting OpenStack.  I do wonder how hard they'd have to work to become 
> the official incubated infrastructure automation solution for OpenStack, 
> thus largely kicking both Chef and Puppet to the curb.
> 
> Ansible also seems to be interested in trying to solve some of the 
> orchestration problems that Chef explicitly disclaims.  I have yet to see 
> how well the Ansible "agent-free" model actually scales, but if they have 
> an accelerated communications option that is based on 0mq, then maybe they 
> can actually keep up with Chef, Salt, etc....  It does seem to me that you 
> would at least want to cache as much of the infrastructure code as 
> possible, however.
> 
> > I certainly see that barrier from the other side as well. But at the same 
> > time I've managed to pick up javascript and angular in my free time 
> > because I was interested in that.  I also don't see this as a solution 
> > because it would push chef into the situation where if a python dialect 
> > of chef was actually successful you'd now need to learn *2* programming 
> > languages to interact fully with the community.
> 
> I'm not convinced that we need to fully support a separate language like 
> Python for interacting with the Chef server.  But having a simpler and less 
> Ruby-specific solution is definitely something I am very interested in 
> looking at.
> 
> Maybe we could just simplify the Chef DSL so that it's more generic, and 
> both sides could be less unhappy with the solution?
> 
> Meanwhile, I really do want to see what Noah has cooking.
> 
> --
> Brad Knowles 
> <
 >
> LinkedIn Profile: <http://tinyurl.com/y8kpxu>
> 
> 
> 
> 
> 
> -- 
> Opscode, Inc.
> Adam Jacob, Chief Dev Officer
> T: (206) 619-7151 E: 
>

Attachment: signature.asc
Description: Message signed with OpenPGP using GPGMail