[chef] Re: An deep topic

["Julian C. Dunn" < >]

Thank you for posting this, John.

I think we have to step back a bit to see how much Chef as a tool & as
an ecosystem has evolved, even in the 1.5 years I've worked at
ChefInc. At ChefConf 2013 we taught a course on infrastructure testing
with Vagrant, minitest-handler, and an early version of Test Kitchen.
At ChefConf 2014 we had to completely blow that course up and start
over, because the tools & practices had changed so much.

Despite all the noise & messiness of the implementation details, the
journey we are on is to treat our infrastructure code as true
software, with the same discipline as software engineers would. This
has downstream effects on our tooling, our artifact management, our
testing practices, and so on. Not all the practices are there & the
tools are definitely not all there yet. Some, like ChefSpec and Test
Kitchen, are more mature & stable than others, but even these are
still fairly new & have many missing features that I would love to
see.

The point is that anything that adds to the goal of treating
infrastructure code seriously and as true software (call it "TDD" if
you wish) is where we want to go. The ingestion of Berkshelf into
Chef-DK makes the tooling easier.

Do I believe that we ingested too much? Yes, the #1 pain point people
had with Chef was client-side dependency resolution. We grabbed
Berkshelf, the first thing that was convenient & met the need. But we
also got a lot of things that are kind of superfluous to the goal. The
"Berkshelf Way", "Environment Cookbooks", and friends all have little
to do with that, and will not be used by most people. (Bring on the
flames?)

Let me try and answer a few of your other questions as best I can.

On Mon, Jul 7, 2014 at 11:05 PM, John E. Vincent (lusis)
<
 >
 wrote:

> 1) Is Berkshelf becoming a core chef dependency?
> I heard no but let's be clear when ALL of the tooling coming out of Chef
> proper has a dependency on Berkshelf, that's no different. Also ChefDK
> doesn't solve that problem (especially since it's not actually released
> yet). Related is "The Berkshelf Way"/model the prescribed  workflow for chef
> repos (note that I'm not saying cookbooks here for a reason).

I think the answer is "for now", at least in the DK, but it's to solve
a couple of point problems as I mentioned above. That's not to say
that Berkshelf just won't get picked apart and have certain pieces
ingested into Chef-DK down the road.

I can't see Berkshelf becoming a Chef Client dep though.

> 2) What problem is Berkshelf trying to solve?
> This is part of my frustration and other folks as well. The running joke is
> that Berkshelf solved a problem I didn't have. And it's true. Again I can
> totally accept that some people had a problem and found that Berkshelf
> solved it but is that reason enough to bake it into EVERYTHING?

Client-side dependency resolution is what it solves, in other words,
the need to get into dependency downloading hell ("knife cookbook
download xyz", oh noes, "xyz" depends on these other 3 cookbooks that
depend on 2 other ones that depend on...)

> FWIW I didn't have a problem with Berkshelf originally. One: I didn't use it
> and didn't HAVE to use it. Two: It didn't have this entirely new service it
> depended on.
>
> Now I can't avoid it and it requires a separate internet service (though now
> apparently rolled into the supermarket) just to be used.

Berkshelf 3 originally had a separate API service that would assist
with resolution, but you're right, the new community site does that
now in the /universe endpoint. You also never actually *had* to use
the server-side solver.

> But this ties into bigger workflow questions. From almost every corner of
> the Chef proper universe (rough paraphrases):
>
> "Berkshelf won"

... for client side dependency resolution. I would venture a guess
that more people are using it versus Librarian because it has much
stronger integration with the other tools (Test Kitchen chief among
them)

I think there are a couple takeaways for all of us here.

For ChefCo:

1. We have to do more of our software engineering in the open and
involve more community members proactively in both design &
implementation. That's something we're addressing. (see previous notes
from Adam)
2. It's easy for us at ChefCo to get wrapped up in the day-to-day and
forget that, as you said, that people use Chef as a means to an end.
We need to be more cognizant of how people are using the toolchain,
*why* they are asking for the things they are asking for, and
addressing their pain points.

For the community:

1. When we open the gates to more direct contributions, we'll actually
need people to step up, while not diminishing the quality of the
product.
2. Bad news: I'm not sure we'll ever arrive at a place where there is
an "official ChefCo way of doing things". There may be an easier
glidepath than others (e.g. "consider installing ChefDK rather than
fighting with 15000 gem dependencies and waiting for gecode to compile
& overheat your CPU") but nothing is mandatory. Don't mistake the
easier glidepath for other paths being second-class citizens.
3. If you see something wrong, please raise it as an opinion or a
question. Don't be afraid of offending people. It's just software,
after all.

To that last point: Having been a part of the Chef community for >2
years now, I find that one downside of our friendliness and the
"hugops" mentality is that we often do not surface contentious issues,
for fear of being hurt or hurting other people.

I think this leads to a thought vacuum. While I'm obviously not
advocating that Chefs start screaming at each other, I think we need
to find a way to have more disagreement -- nay, even conflict -- to
make a better product. So thank you John for raising this.

- Julian